When can an $\mathfrak{S}_n$-equivariant map be extended to an $\textrm{O}(n)$-equivariant map?

If I have understood the problem correctly, the map $\Phi$ deterimines $\varphi = \Phi \circ d$, so the question amounts to classifying possible compositions $\Phi \circ d$, where $d$ is the "diagonal" map, and $\Phi$ is $O(n)$ equivariant. Clearly the image of $\varphi$ must be contained in the image of $\Phi$ which is isomorphic to a quotient of $\mathrm{Sym}_2(\mathbb{R}^n)$.

Let's begin by noting that as a representation of $\mathfrak{S}_n$, we have the decomposition $\mathbb{R}^n = \mathbf{1}_{\mathfrak{S}_n} \oplus U$, where $\mathbf{1}_{\mathfrak{S}_n}$ is the trivial representation (spanned by the "all ones" vector), and $U$ is the (irreducible) standard representation (consisting of "mean zero" vectors).

Similarly, as a representation of $O(n)$, the representation $\mathrm{Sym}_2(\mathbb{R}^n)$ (which we are interpreting as symmetric $n \times n$ matrices under conjugation) decomposes as $\mathbf{1}_{O(n)} \oplus W$, where $\mathbf{1}_{O(n)}$ is the trivial representation (spanned by the identity matrix), and $W$ is an irreducible representation which consists of symmetric matrices of trace zero.

It is not difficult to see that the map $d: \mathbf{1}_{\mathfrak{S}_n} \oplus U \to \mathbf{1}_{O(n)} \oplus W$ is of the form $f \oplus g$, with $f : \mathbf{1}_{\mathfrak{S}_n} \to \mathbf{1}_{O(n)}$ (it sends the all-ones vector to the identity matrix) and $g: U \to W$.

Now, $\Phi$ must map $\mathbf{1}_{O(n)}$ to an invariant vector (possibly zero), and it must map $W$ to either a copy of $W$ or zero. So, we must give an "intrinsic" description of the image of $g$, i.e. traceless diagonal matrices, inside $W$.

Note that $O(n)$ contains not only $\mathfrak{S}_n$, but the larger hyperoctahedral group $\mathfrak{H}_n = C_2 \wr \mathfrak{S}_n = C_2^n \rtimes \mathfrak{S}_n$, via signed permutation matrices (permutation matrices, but the nonzero entries can be $\pm 1$). The hyperoctahedral group contains a subgroup $C_2^n$ consisting of diagonal matrices with entries $\pm 1$. It is not difficult to check that a matrix that commutes with every element of $C_2^n$ (viewed as a diagonal matrix) must itself be diagonal, and conversely it is clear that every diagonal matrix commutes with $C_2^n$. Phrasing this in terms of the module structure (commuting with = fixed under conjugation by), the diagonal matrices are fixed by the action of $C_2^n$.

We are now in a position to give the characterisation. A map $\varphi: \mathbb{R}^n \to V$ may be written in the form $\Phi \circ d$ if and only if all of the following conditions are satisfied.

    1. The $O(n)$-module generated by the image of $\varphi$ is a quotient of $\mathrm{Sym}_2(\mathbb{R}^n)$.
    2. The image of the "all ones" vector is $O(n)$ invariant (possibly zero).
    3. The image of $\varphi$ is fixed pointwise by $C_2^n$ (and therefore has a $\mathfrak{H}_n$ action that factors through $\mathfrak{S}_n$).

We've demonstrated that these conditions are necessary, so let's check that they are also sufficient. Note that by (2), given $\varphi$ as above, the restriction $\mathbf{1}_{\mathfrak{S}_n} \to V$ determines the restriction $\Phi: \mathbf{1}_{O(n)} \to V$. So it suffices to construct the restriction $\Phi: W \to V$. By (1), we are guaranteed to find such a map whose image agrees with the $O(n)$-module generated by $\varphi(U)$; we check that that (3) implies that $\varphi(U)$ coincides with the image of the (traceless) diagonal matrices in $W$ (because these form an irreducible representation of $\mathfrak{S}_n$, rescaling $\Phi$ is all that is needed to guarantee pointwise agreement of $\varphi$ and $\Phi \circ d$).

As a representation of $\mathfrak{H}_n$, $\mathrm{Sym}_2(\mathbb{R}^n)$ can be decomposed as follows. Recognise that

$$ \mathbb{R}^n = \mathrm{Ind}_{\mathfrak{H}_1 \otimes \mathfrak{H}_{n-1}}^{\mathfrak{H}_n}(\varepsilon \boxtimes \mathbf{1}_{\mathfrak{H}_{n-1}}), $$

where $\varepsilon$ is the sign character of $\mathfrak{H}_1 = C_2$. Now, using a little Mackey theory (to write down the tensor square, and then extract they symmetric part), we see that

$$ \mathrm{Sym}_2(\mathbb{R}^n) = \mathrm{Ind}_{\mathfrak{H}_1 \otimes \mathfrak{H}_{n-1}}^{\mathfrak{H}_n}(\mathrm{Sym}_2(\varepsilon) \boxtimes \mathbf{1}_{\mathfrak{H}_{n-1}}) \oplus \mathrm{Ind}_{\mathfrak{H}_2 \otimes \mathfrak{H}_{n-2}}^{\mathfrak{H}_n}(\varepsilon \otimes \varepsilon \boxtimes \mathbf{1}_{\mathfrak{H}_{n-1}}) $$

where $\varepsilon \otimes \varepsilon = \varepsilon \otimes \varepsilon \otimes \mathbf{1}_{\mathfrak{S}_2}$ is a one-dimensional representation of $\mathfrak{H}_2$ where a signed matrix acts by $(-1)^m$, where $m$ is the number of $-1$'s in the signed matrix. Enthusiasts of wreath products will recognise both summands as being irreducible representations of the hyperoctahedral group. The key point here is that $\mathrm{Sym}_2(\varepsilon) = \mathbf{1}_{C_2}$, so the first summand simply becomes

$$ \mathrm{Ind}_{\mathfrak{H}_1 \otimes \mathfrak{H}_{n-1}}^{\mathfrak{H}_n}(\mathbf{1}_{\mathfrak{H}_1} \boxtimes \mathbf{1}_{\mathfrak{H}_{n-1}}), $$

which is simply $\mathbb{R}^n$, viewed as a $\mathfrak{H}_n$ module that factors through the action of $\mathfrak{S}_n$. In particular, this summand corresponds to diagonal matrices in $\mathrm{Sym}_2(\mathbb{R}^n)$. (The other summand does not factor through the $\mathfrak{S}_n$ action.)

One final comment: If $n \geq 4$, as a representation of $\mathfrak{S}_n$,

$$ \mathrm{Sym}_2(\mathbb{R}^n) = S^{(n)} \oplus S^{(n)} \oplus S^{(n-1,1)} \oplus S^{(n-1,1)} \oplus S^{(n-2,2)} $$

where $S^\lambda$ is a Specht module (irreducible representation) indexed by $\lambda$. The key point here is that $U = S^{(n-1,1)}$ appears with multiplicity 2. This means that trace zero symmetric matrices are not the unique subspace of $\mathrm{Sym}_2(\mathbb{R}^n)$ isomorphic to $U$. This means that condition (3) is not automatic. In particular, Nate's suggestion of understanding the interaction with the Casimir element will help pin down condition (1) (it helps identify the ambient $O(n)$ representation), but some extra information will be required to detect condition (3).


Not an answer, just too long for a comment.

All representations in sight can be decomposed as direct sums of irreducible representations. By the Schur lemma, mapping between irreducibles can be either zero, or multiple of identity. In this way the problem can be broken down into subproblems. (I am not sure it helps much, but at least you can see immediately, once you have the decomposition, whether there is any $O(n)$ map at all.)

The symmetric matrices decompose as $O(n)$-modules into traceless matrices and multiples of identity matrix. I.e. $$A \mapsto (A - \frac{1}{n}(\mathrm{Tr}\, A)\; \mathrm{Id}) \oplus \frac{1}{n}(\mathrm{Tr}\, A ) \; \mathrm{Id}.$$

If $V$ is a tensor representation (i.e. subrepresentation of $\bigotimes^k\mathbb{R}^n$) then the decomposition into $O(n)$-modules works in a similar way. One substracts all possible traces over all possible pairs of indices and then decomposes resulting modules according to their $\mathfrak{S}_n$-symmetries in indices. (See Goodmann, Wallach for reference.)

Unfortunately, I am not much familiar with representations of finite groups so I don't know what the decomposition of the symmetric matrices looks like.