How to effectively calculate $\int_0^\pi \sin^3x \sin(nx)dx$, $n\in \mathbb{N}$

You're doing fine. To finish, you can use the following Hint : $\sin(p)\sin(q)=\frac{\cos(p-q)-\cos(p+q)}{2}$.

HINT:

Through integration by parts, the integral $\sf{\int\sin mx\sin nx\,dx}$ is equivalent to $$\sf{-\frac1n\sin mx\cos nx+\frac mn\int \cos mx\cos nx\,dx}$$ and use integration by parts again for $\sf{\int\cos mx\cos nx\,dx}$ to get $$\sf{\frac1n\cos mx\sin nx-\frac mn\int\sin mx\sin nx\,dx}$$ and equate the two common integrals to get that $$\sf{\int\sin mx\sin nx\,dx=-\frac1n\sin mx\cos nx+\frac m{n^2}\cos mx\sin nx-\frac{m^2}{n^2}\int\sin mx\sin nx\,dx.}$$

Maybe it's a more advanced tool. I'm expose anyway my solution (for people who are interested). Remark that $x\mapsto \sin(x)\sin(nx)$ is even for all $n\in\mathbb N^*$, therefore, $$\int_0^{\pi}\sin^3(x)\sin(nx)\,\mathrm d x=\frac{1}{2}\int_{-\pi}^\pi\sin^3(x)\sin(nx)\,\mathrm d x.$$

As the OP remarked, $$\sin^3(x)=\frac{3}{4}\sin(x)-\frac{1}{4}\sin(3x).$$ Therefore, using Fourier series, we immediately obtain

$$\int_{-\pi}^\pi\sin^3(x)\sin(nx)\,\mathrm d x=\begin{cases}\frac{3\pi}{4}&n=1\\ -\frac{\pi}{4}&n=3\\ 0&\text{otherwise}\end{cases}.$$