The coefficient of a specific monomial in the expansion of the following polynomial

This is not an answer but a conjecture based on some trials. $$a_{n,k} = \begin{cases} 1, & n=1 \\ 0, & n\gt 1, k\text{ odd} \\ (-1)^{t\binom n2}\displaystyle\frac{(tn)!}{(t!)^n}, & k=2t. \end{cases} $$ Note that $\frac{(tn)!}{(t!)^n}$ is the number of ways of partitioning a set of size $tn$ into $n$ sets of size $t$.

Oh, now I see that this is a special case of Macdonald's constant-term conjecture, see this article, which has references.


Let's call your polynomial $\Delta$. Then $\Delta$ is anti-symmetric in the $x_i$ (if you switch $x_i$ to $x_j$ then $\Delta$ turns into $-\Delta$). If $k$ is odd, then $\Delta^k$ also has the property, so $a_{n,k}=0$ whenever $k$ is odd.

I will now show that $(-1)^{\binom{n}{2} m} a_{n,2m}>0$. Write $k=2m$. Let $T$ be the torus $\theta_1 + \theta_2 + \cdots + \theta_n=0$ inside $(\mathbb{R}/2 \pi \mathbb{Z})^n$, equipped with the Haar measure $d V$ with volume one.

Since $\Delta^{k}$ is homogenous of degree $k \binom{n}{2}$, it cannot contain monomials of the form $(x_1 x_2 \cdots x_n)^e$ for any $e$ other than $k \tfrac{n-1}{2}$. Therefore, we can extract $a_{n,k}$ by a contour integral over $T$:

$$a_{n,k} = \int_T \Delta(e^{i \theta_1}, \ldots, e^{i \theta_n})^{2m} dV$$ $$=\int_T \prod_{a<b} \left( e^{i \theta_a} - e^{i \theta_b} \right)^{2m} dV$$ $$=\int_T \prod_{a<b} \left( e^{i2 \theta_a} - 2 e^{i (\theta_a+\theta_b)} + e^{i 2 \theta_b} \right)^{m} dV$$ $$=\int_T e^{i (n-1) \sum \theta_a} \prod_{a<b} \left( e^{i (\theta_a- \theta_b)} - 2 + e^{i (\theta_b-\theta_a)} \right)^{m} dV$$ $$=(-2)^{\binom{n}{2} m} \int_T \prod_{a<b} \left( 1 - \cos (\theta_a - \theta_b) \right)^{m} dV.$$ Going from the second-to-last line to the last line, I used that $\sum \theta_a=0$ on $T$.

The integrand is nonnegative (and positive except on a set of measure zero) so the final integral is positive.