Unitary irreducible representations of the little group $SO(3)$

This answer is based on this article by A. Ungar.

Ungar computed the Thomas rotation formula which is almost what you need. I'll describe the general procedure, and in some cases, I'll refer you to Ungar for the proof. I'll express (just like Ungar), the Boosts in terms of velocities rather than momenta. If you wish you can repeat the exercise with the momentum parametrization. From Wikipedia we have $$B(\mathbf{v}) = \begin{bmatrix} \gamma_{\mathbf{v}}& \frac{\gamma_{\mathbf{v}}}{c}\mathbf{v}^t \\ \frac{\gamma_{\mathbf{v}}}{c}\mathbf{v} & 1_{(3\times3)}+ \frac{\gamma_{\mathbf{v}}^2}{1+\gamma_{\mathbf{v}}} \frac{\mathbf{v}\mathbf{v}^t}{c^2} \end{bmatrix}$$

The key point is finding the Wigner rotation is the observation that every Lorentz transformation can be decomposed (nonuniquely) as a product of a Boost and a rotation: $$\Lambda = B(\mathbf{u}) R$$ Any choice of the decomposition method will do, but we need to work in a fixed method of decomposition. I'll describe to you a possible method at the end of the answer Now, when we multiply two boosts, we get a boost with the relativistic addition velocity + a rotation (often referred to as the Thomas rotation): $$ B(\mathbf{u}) B(\mathbf{v}) = B(\mathbf{u}\oplus \mathbf{v}) \mathrm{Tom}(\mathbf{u},\mathbf{v})$$ Where $\mathrm{Tom}$ is a rotation matrix Ungar found the general solution for the Thomas rotation (Equation : (13) in the article) $$\mathrm{Tom}(\mathbf{u},\mathbf{v}) = B(-\mathbf{u}\oplus \mathbf{v}) B(\mathbf{u}) B(\mathbf{v})$$ Now, we are in a position to solve the equation for the Wigner rotation. We need to solve: $$ \Lambda B(\mathbf{v}) = B(\Lambda \mathbf{v}) W$$ for $W$. We parametrize $\Lambda$, we get for the left handside:

$$\begin{align*} \Lambda B(\mathbf{v}) & = B(\mathbf{u}) R B(\mathbf{v}) \\ & = B(\mathbf{u}) R B(\mathbf{v}) R^{-1} R \\ & = B(\mathbf{u}) B(\mathbf{Rv}) R \\ & = B(\mathbf{u}\oplus R\mathbf{v}) \mathrm{Tom}(\mathbf{u},R\mathbf{v}) R \end{align*} $$

and for the right hand side $$ B(\Lambda \mathbf{v}) W = B(B(\mathbf{u}) R\mathbf{v}) W = B(\mathbf{u}\oplus R\mathbf{v}) W$$

Thus: $$W = \mathrm{Tom}(\mathbf{u},R\mathbf{v}) R$$ What is left is to describe a specific parametrization of a general Lorentz matrix into a boost and a rotation: We need to find $B(\mathbf{u})$ and $R$ such that: $$\begin{bmatrix} \lambda_0 &\mathbf{\xi}^t \\ \mathbf{\eta} & \Lambda_1 \end{bmatrix} = \begin{bmatrix} \gamma_{\mathbf{v}}& \frac{\gamma_{\mathbf{v}}}{c}\mathbf{v}^t \\ \frac{\gamma_{\mathbf{v}}}{c}\mathbf{v} & 1_{(3\times3)}+ \frac{\gamma_{\mathbf{v}}^2}{1+\gamma_{\mathbf{v}}} \frac{\mathbf{v}\mathbf{v}^t}{c^2} \end{bmatrix} \begin{bmatrix} 1&0 \\ 0 &R_1 \end{bmatrix}$$

We observe that $R_1$ needs to satisfy the following relation $$\mathbf{\xi} = R_1 \mathbf{\eta}$$ Thus $R_1$ needs to rotate the 3-vector $\mathbf{\mathbf{\eta}}$ into the 3-vector $\mathbf{\xi}$ The solution (using a similar method as Ungar) can be written as: $$R_1 = 1_{(3\times3)}+ \sin(\theta) \Omega + (\cos(\theta-1)) \Omega^2$$

Where $\theta$ is the angle between the vectors $\mathbf{\eta}$ and $\mathbf{\xi}$ $$\sin(\theta) = \frac{\mathbf{\eta} \times \mathbf{\xi}}{|\mathbf{\eta}||\mathbf{\mathbf{\xi}}|}$$ and $$\Omega_{ij} =\frac{ \eta_i \xi_j - \xi_i \eta_j}{|\mathbf{\eta}||\mathbf{\xi}|}$$