How to find the coefficients of an iteration of a polynomial function?
Edit: See below for a much better way to derive the main result.
Represent the coefficients of a polynomial $p$ as a map $c_p$ from integers to real numbers, where $c_p(j)$ is the $j^{th}$ coefficient of $p$. Equivalently, $c_p$ is just the vector of coefficients of $p$. Multiplying two polynomials $p$ and $q$ corresponds to taking a discrete convolution of the corresponding coefficient vectors: $$ c_{pq}(i) = \sum_j c_p(i-j) c_q(j) = (c_p*c_q)(i) $$
By the convolution theorem, the Fourier transform of a convolution is a product of the individual Fourier transforms. Hence, the Fourier transform of the coefficient vector of $p^{k}$ (Caution! This is a regular power of $p$, not the same as $p^{(k)}$) is simply $\mathcal{F}[c_p^{*k}]=\mathcal{F}[c_p]^k$, where $\mathcal{F}$ is a discrete Fourier transform and $c_p^{*k}$ means $c_p$ convolved with itself $k$ times.
Now from the relation $p^{(k)}(x)=\sum_j c_p(j) \left(p^{(k-1)}(x)\right)^j$ we find that the coefficient vector $c_{p^{(k)}}$ satisfies $$c_{p^{(k)}}=\sum_j c_p(j) c_{p^{(k-1)}}^{*j}$$ hence the Fourier transform satisfies $$\mathcal{F}[c_{p^{(k)}}]=\sum_j c_p(j) \mathcal{F}[c_{p^{(k-1)}}^{*j}]=\sum_j c_p(j) \mathcal{F}[c_{p^{(k-1)}}]^j=p\left(\mathcal{F}[c_{p^{(k-1)}}]\right)$$ and inductively $$\mathcal{F}[c_{p^{(k)}}]=p^{(k-1)}\left(\mathcal{F}[c_{p}]\right)$$
Hence $$c_{p^{(k)}}=\mathcal{F}^{-1}\left[ p^{(k-1)}\left(\mathcal{F}[c_{p}]\right) \right]$$
Using Fast Fourier Transforms, this is a very numerically efficient way to compute the coefficients you seek, as opposed to the brute force way of multiplying all the polynomials or convolving the coefficients directly. (There may also be cases where the DFT can be computed analytically, so that this formula leads to closed form expressions for the coefficients, but this will not be the case in general.)
EDIT: A better derivation.
I recently found a much better way to derive the main result above. As before, let $c_{p}(j)$ denote the $j^{th}$ coefficient of polynomial $p$. By taking a (conventionally normalized) DFT we find $$\mathcal{F}[c_p](k) = \sum_j c_p(j) \left(e^{-2\pi i k/N}\right)^j = p\left(e^{-2\pi i k/N}\right)$$ where $N$ is the size of the DFT, and can be any integer larger than the degree of the polynomial $p$. Hence, we have this expression for the coefficients of $p$: $$c_p = \mathcal{F}^{-1}\left[ p\left(e^{-2\pi i k /N}\right) \right]$$ In particular, this works for iterated polynomials $p^{(n)}$: $$\boxed{c_{p^{(n)}} = \mathcal{F}^{-1}\left[ p^{(n)}\left(e^{-2\pi i k /N}\right) \right]}$$
A few remarks:
- A small variation on this works for analytic functions too. You simply replace the DFT with a DTFT (i.e. let the size $N$ of the DFT $\rightarrow \infty$). Then the expression for $c_p$ becomes $\mathcal{F}^{-1}\left[ p\left(e^{-2\pi i k}\right) \right]$, with $k\in [0,1]$ real valued.
- The preceding expression can be viewed as Cauchy's integral formula. Explicitly, $$\begin{align*} c_p(j) & = \mathcal{F}^{-1}\left[ p\left(e^{-2\pi i k}\right) \right] \\ & = \int_0^1 e^{2\pi i k j} p\left(e^{-2\pi i k}\right) dk \\ & = \frac{1}{2\pi i}\oint \frac{p(z)}{z^{j+1}} dz \\ & = \left. j! \frac{d^j p}{dz^j}\right|_{z=0} \end{align*} $$ where the third line follows from a change of variables and the last line is from Cauchy's derivative formula.
- The boxed equation above can similarly be viewed as a discrete version of Cauchy's formula.
I implemented this in Mathematica to make sure it works, and sure enough it did. Something to watch out for if you try to implement this numerically is the normalization on your Fourier transforms. We've assumed particular normalizations of the Fourier transform above, and if you use a different one (as is the default in Mathematica, it turns out), you need to multiply by a constant factor on every Fourier and inverse Fourier transform. In Mathematica, that factor is $\sqrt{N}$, where $N$ is the length of vector you use to store the coefficients. Define your own transform which is the default times $\sqrt{N}$, and your own inverse which is the default divided by $\sqrt{N}$, and everything will work out. Also note that Mathematica's Fourier function is actually an inverse Fourier transform per standard conventions, and similarly InverseFourier is a Fourier transform.