About stabilizer in group action

Take the action of $S_3$ on itself by conjugation and $x=(1\ 2)$. Then the stabilizer of $x$ is $\{e, x\}$ and this is not a normal subgroup of $S_3$.

If this where true all subgroups would be normal. Take a subgroup $H$ of $G$. Consider the left regular action of $G$ on the set of left cosets of $G$ by $H$. The stabilizer of $H$ is clearly $H$. Using the above we conclude $H$ is normal.

Consider the symmtric group on $X$. Conjugation here corresponds to 'renaming' the elements of $X$. Suppose as usual $X=\{1,2,\dots,n\}$. So for example conjugating the cycle $(1234)$ by the transposition $(24)$ gives the cycle $(1432)$, because we've renamed '4' to be '2' and vice versa.

So a normal subgroup of $S_X$ is one which has the same elements in it when you do any 'renaming' of the symbols $1,\dots,n$ throughout its elements. You can see then that a stabilizer of a point here is the last thing we'd expect to be normal. The subgroup $G_4$ consisting of elements that fix $4$, say, contains some elements that fix $2$ and some that don't, so it won't consist of the same elements if '4' and '2' swap roles. (It will conjugate to $G_2$, of course.)