Why is this allowed? ("Fourier's Trick"; finding the coefficients in a Fourier Series)

Your suspicion about inner product is entirely correct. The trigonometic polynomials $\{\sin(nx), \cos(mx)\}$ (possibly translated and scaled) are known to form an orthogonal system with respect to the scalar product given by $\langle f, g\rangle:=\int fg$, and if you choose the function space correctly (usually one uses a space called $L^2$) and use properly chosen norming factors then one can show they actally form a complete orthonormal system $e_k$, which simply means you can express any function in that space as $$ f = \sum_k\langle f, e_k \rangle e_k$$ (where convergence is to be understood in that space wrt the norm derived from the scalar product). The coefficients in that sum are what you are looking at. You may know that kind of representation from the finite dimensional case.

I deliberately did not specify the constants which make the orthogonal system orthonormal, nor an interval as domain of definition -- by translation and scaling you can do something like that on any bounded interval in $\mathbb{R}$ There is also a complex version of this, in which case one would use the scalar product $\int f\bar{g}$, and $\{e^{ikx}\}$ as orthogonal system.

If you want to look up the details, then most introductions to real analysis will have a section on that topic. Rudin's books, for example do explain this.

It might help to break this up into smaller steps.

\begin{align*} &V_0(y) = \sum_{n=1}^{\infty} C_n \sin \left(\frac{n \pi y}{a} \right) \\ \implies & V_0(y) \sin \left(\frac{n' \pi y}{a} \right) = \sum_{n=1}^{\infty} C_n \sin \left(\frac{n \pi y}{a} \right)\sin \left(\frac{n' \pi y}{a} \right) \\ \implies & \int_0^a V_0(y) \sin \left(\frac{n' \pi y}{a} \right) \, dy = \sum_{n=1}^{\infty} C_n \int_0^a \sin \left(\frac{n \pi y}{a} \right)\sin \left(\frac{n' \pi y}{a} \right) \, dy = \frac{a}{2} C_{n'}. \end{align*}

We now solve for $C_{n'}$ to obtain \begin{equation*} C_{n'} = \frac{2}{a} \int_0^a V_0(y) \sin \left(\frac{n' \pi y}{a} \right) \, dy. \end{equation*}