Product of two reflections lying in a parabolic subgroup of a Coxeter group

$\DeclareMathOperator\Im{Im}\DeclareMathOperator\Fix{Fix}$As suggested by Sam, I am posting this as an answer.

As mentioned in the comments, the above question can be solved using a result of Baumeister–Dyer–Stump–Wegener from A note on the transitive Hurwitz action on decompositions of parabolic Coxeter elements, which says the following: let $t_i$, $i=1,\dotsc, k$ be reflections in an arbitrary Coxeter group $(W,S)$. Assume that $W'$ is a (not necessarily standard) parabolic subgroup of $W$, and assume that $t_1 t_2\dotsm t_k$ is a reduced reflection factorization of an element $w\in W'$. Then $t_i\in W'$ for all $i$. Applied to the case $k=2$ and $W'=W_I$ this answers the question.

I also think that there is a more elementary way to see this (for $k=2$). Assume that $w\mathrel{:=}t_1 t_2\in W_I$. Let $V$ be the geometric representation of $W$ and let $\beta_1$ and $\beta_2$ be the roots corresponding to $t_1$, respectively $t_2$. Then $$\Im(t_1 t_2 - 1)\subseteq \mathbb{R} \beta_1 \oplus \mathbb{R} \beta_2.$$ Now since $w\in W_I$, denoting $\alpha_i$, $i\in I$ the simple roots attached to the simple generators in $I$ we also have $$\Im(t_1 t_2 - 1)=\Im(w - 1)\subseteq \bigoplus_{i\in I} \mathbb{R} \alpha_i.$$

Claim : $\mathbb{R} \beta_1 \oplus \mathbb{R} \beta_2\subseteq \bigoplus_{i\in I} \mathbb{R}\alpha_i$ (which concludes the proof, as $\beta_i$ are then linear combinations of the simple roots in $I$, hence $t_i\in W_I$).

To see this, by the above inclusions, it is enough to see that $M(w)\mathrel{:=}\Im(w-1)=(w-1)(V)$ has dimension $2$. The point is that, unfortunately, this is not the case in general: this is related to what has already been said in the comments but for instance, this fails in type $\widetilde{A}_1$, taking $w=st$, where $\{s,t\}=S$, then the line spanned by $\alpha_s+\alpha_t$ is fixed by $w$, hence $\dim(M(w))=1$. As noted in the comments, as $t_1$ and $t_2$ are distinct, the roots $\beta_1$ and $\beta_2$ are not proportional, and therefore $\Fix(t_1t_2)=\Fix(t_1)\cap\Fix(t_2)$. But these two hyperplanes may be equal, as in the example.

But I think that on can cheat as follows by taking a bigger Coxeter group $\widetilde{W}$ (with geometric representation $V'$) in which $W$ is a standard parabolic subgroup, and such that $(w-1)(V')$ has dimension two. For such an enlarged geometric representation we still have the inclusions $(w-1)(V') \subseteq \mathbb{R}\beta_1\oplus\mathbb{R} \beta_2$ and $(w-1)(V')\subseteq \bigoplus_{i\in I}\mathbb{R} \alpha_i$.

For instance, one can take $V'$ as follows: first, let $x\in W$ such that $x t_1 x^{-1}=s\in S$, and let $w'\mathrel{:=}x w x^{-1}= s t_2'$, with $t_2'=xt_2 x^{-1}$. Consider the Coxeter group $\widetilde{W}$ with one more simple generator $s'$ than $W$, such that $ss'=s's$ and $m_{s't}=\infty$ for any $t\in S\setminus\{s\}$. Let $V'$ be its geometric representation (note that $\dim(V')=\dim(V)+1$). Then $s'$ commutes with $s$ but cannot commute with $t_2'$ (as $s\neq t_2'$), hence $\alpha_{s'}\in\Fix(s)$ but $\alpha_{s'}\notin\Fix(t_2')$, and therefore $\Fix(s)\neq \Fix(t_2')$. It follows that the intersection $\Fix(s)\cap \Fix(t_2')=\Fix(st_2')=\ker(w'-1)$ has dimension $\dim(V')-2$, and hence, that $(w'-1)(V')$ has dimension $2$. Conjugating back by $x^{-1}$ we get that $(w-1)(V')$ has dimension $2$, which concludes the proof.


$\DeclareMathOperator\Fix{Fix}\DeclareMathOperator\Stab{Stab}$I want to share a possible solution that I found just developing the idea contained in Nathan's deleted answer. His argument just needed a representation $V$ with the following two properties:

(i) $\Fix(t_1t_2)=\Fix(t_1)\cap \Fix(t_2)$, if $t_1$, $t_2$ are distinct reflections in $W$.

(ii) $\exists v_0\in V$ such that $\Stab(v_0)=W_I$

With such a representation in hand one can easily conclude as follows (assuming $t_1\neq t_2$): $t_1t_2\in W_I \Rightarrow v_0\in \Fix(t_1t_2)=\Fix(t_1)\cap \Fix(t_2) \Rightarrow t_1,t_2 \in W_I$.

As discussed in previous comments, the geometric representation of $W$ satisfies (i) but fails (ii), while its dual satisfies (ii) but fails (i). However, a certain enlargement of the geometric representation (having the dual of it as a quotient) does the job:

Claim: Let $V$ be a finite dimensional real vector space with given linearly independent vectors $\{e_s\}_{s\in S}\subseteq V$ and given linearly independent linear forms $\{e_s^\vee\}_{s\in S}\subseteq V^*$ such that $\langle e_t,e_s^\vee\rangle=-2\cos(\frac{\pi}{m_{st}})$ $\forall s,t\in S$. Assume also that $V$ has the smallest possible dimension to satisfy these requests. Then $s\cdot v\mathrel{:=}v-\langle v,e_s^\vee\rangle e_s$ turns $V$ into a representation of $W$ that satisfies (i) and (ii).

The fact that such a vector space $V$ always exists is straightforward, while the fact that the above formula actually defines a representation of $W$ can be found in Soergel's article Kazhdan–Lusztig polynomials and indecomposable bimodules over polynomial rings (Proposition 2.1) (actually the original article is in German, the linked one is just an English preprint). The same Proposition also shows that this representation is reflection faithful: this in particular means that $\Fix(w)$ is a hyperplane if and only if $w$ is a reflection in $W$ and that distinct reflections fix different hyperplanes, which implies at once (i).

For (ii), we use the fact (contained in the proof of the above mentioned Proposition) that, if $K\mathrel{:=}\{v\in V \mathrel\vert \langle v, e_s^\vee\rangle=0\ \forall s\in S\}$ (so $K$ is a subrepresentation of $V$, where $W$ acts trivially), then $V/K$ is isomorphic to the dual of the geometric representation of $W$, and so it contains a vector with stabilizer equal to $W_I$. As a consequece, for any $v_0\in V$ lying in the preimage of such a vector one has $\Stab(v_0)\subseteq W_I$. To show the other inclusion, let $s \in I$: one has then $s\cdot v_0\in v_0+K$, but by definition one also has $s\cdot v_0=v_0-\langle v_0, e_s^\vee\rangle e_s$. Since $e_s\notin K$ ($\langle e_s,e_s^\vee\rangle=2$) it has to be $\langle v_0,e_s^\vee\rangle=0$, i.e. $s\in \Stab(v_0)$, as desired.