Why are $L_4$ and $L_5$ lagrangian points stable?

When you look at the dynamics in the rotating reference frame, there are 4 forces acting on the particle: the two gravitational pulls from the massive bodies, the centrifugal push away from the center of rotation (located between the massive objects) and the Coriolis force.

The first three forces depend on the position of the particle, and can be derived from a potential (that also depends on the position), whose level curves are shown in the picture presented with the question. This potential has local maxima at L4 and L5.

The Coriolis force depends on the velocity of the particle: it is perpendicular to it, contained in the plane of motion and proportional to the speed. It curves the motion of the particle to the right (if the massive bodies and the reference system rotate counterclockwise, which is what you see in our Solar System if you stand on the North pole of the Earth).

If a particle placed at L4 tries to leave the point with a mild speed, the Coriolis force curves its trajectory. The trajectory is too curly to get anywhere. See the animation at http://demonstrations.wolfram.com/OrbitsAroundTheLagrangePointL4/.

Of course this doesn't prove that the particle will stay near L4 forever. I don't know a proof. I've seen some computations that show that the dynamical equation linearised at L4 is stable if the mass ratio of the massive objects is sufficiently large, but this also is not enough to prove stability in the non-linearised problem.

I would be convinced that the equilibrium is stable if I were shown that there exists a conserved quantity (depending on the position and speed) that has a strict local extremum at that point of phase space (position=L4, speed=0).

The "energy" (potential discussed above + kinetic energy measured in our non-inertial reference system) is conserved, because the Coriolis force is perpendicular to the trajectory, so it doesn't perform work (in fact, in Lagrangian mechanics it is derived from a potential that depends on the position and speed of the particle). But this quantity doesn't have an extremum at our equilibrium point, because the potential has a local maximum at L4 and the kinetic term is minimum when the speed is 0.

So I can't prove that the equilibrium is stable.

Here's another way of looking at it. Let $M_1$, $M_2$, $M_3$ be our three masses. In the three body problem we're considering, the whole frame containing $M_1$, $M_2$ and $M_3$ is rotating.

You're right to think that if that frame was fixed then the points L4 and L5 would not be stable. After all if you perturb $M_3$ from L4 or L5 then it should just roll down the potential hill.

But there is another force at work here. Because the frame is rotating there is a fictitious force called the Coriolis force which you would feel. This is the same force that makes hurricanes rotate into spirals when seen from space.

When you take the Coriolis force into account, L4 and L5 become stable fixed points. That is, if you perturb $M_3$ from L4 a little bit it will just stay at it's perturbed distance and orbit the point L4.

If anyone wants to see the maths for this then check out this useful article.

As others have commented, including OP, the effective potential (consisting of gravity & the centrifugal potential) $$V~=~ -\frac{Gm_1}{|z_1|} - \frac{Gm_2}{|z_2|} -\frac{\Omega^2 |z|^2}{2} \tag{1} $$ in the orbital plane $\mathbb{R}^2\cong\mathbb{C}$ has a global maximum at the Lagrange points $L_4$ and $L_5$, $$\begin{align}z_1~=~&\hspace{5mm} R\exp\left\{\pm \frac{i\pi}{3} \right\} ~=~\hspace{5mm}\frac{R}{2}\pm\frac{\sqrt{3}iR}{2}, \cr z_2~=~&-R\exp\left\{\mp \frac{i\pi}{3} \right\}~=~-\frac{R}{2}\pm\frac{\sqrt{3}iR}{2} .\end{align} \tag{2}$$ I'm here using the same notation as my Phys.SE answer here: $$\begin{align}z_1~:=&~ z-r_1~\neq~0,\cr z_2~:=&~ z-r_2~\neq~0,\cr \Omega^2~:=&~\frac{G(m_1+m_2)}{R^3}.\end{align} \tag{3}$$ Usually a test particle doesn't want to be at a global maximum! However, we should not forget the Coriolis force.

Main statement. When the test particle tries to leave $L_4$ or $L_5$, the Coriolis force will prevent this via deflection iff one of the mass ratios $m_1/m_2$ or $m_2/m_1$ exceeds $$\frac{25}{2}+\frac{3}{2}\sqrt{69}~\approx~25.\tag{4}$$

In this answer, I would like to calculate this mass ratio condition (4), which is also mentioned on Wikipedia.

The Hessian of the effective potential (1) is $$ \begin{align}H^{ik}~:=~&\frac{\partial^2 V}{\partial z^i \partial z^k},\qquad i,k~\in~\{1,2\},\qquad {\bf z}~\in~\mathbb{R}^2,\cr {\bf H}~=~& -Gm_1 \left(\frac{3{\bf z}_1{\bf z}_1^T}{|{\bf z}_1|^5}-\frac{{\bf 1}_{2\times 2}}{|{\bf z}_1|^3}\right)\cr &-Gm_2 \left(\frac{3{\bf z}_2{\bf z}_2^T}{|{\bf z}_2|^5}-\frac{{\bf 1}_{2\times 2}}{|{\bf z}_2|^3}\right) -\Omega^2{\bf 1}_{2\times 2} . \end{align} \tag{5}$$ At $L_4$ or $L_5$, the Hessian becomes $$\begin{align} {\bf H}~=~&-\frac{3\Omega^2}{4} \begin{pmatrix} 1 & \pm \sqrt{3}\epsilon_{12} \cr \pm \sqrt{3}\epsilon_{12} & 3 \end{pmatrix}, \cr \epsilon_{12}~:=~&\epsilon_1-\epsilon_2~=~\frac{m_1-m_2}{m_1+m_2}~=~1-2\epsilon_2~\in~]-\!1,1[. \end{align} \tag{6}$$ The trace & determinant are $$ {\rm tr}({\bf H})~=~-3\Omega^2 \qquad\text{and}\qquad \det({\bf H})~=~\frac{27\Omega^4}{16}(1-\epsilon^2_{12})~>~0. \tag{7}$$

We now use the following theorem$^1$ mentioned in Ref. 1:

Theorem. Let there be given a Hessian ${\bf H}$ for an effective potential $V:\mathbb{R}^2\to\mathbb{R}$ at a Lagrange point. Let $\Omega$ be the angular velocity. The following 3 conditions are necessary & sufficient conditions for the Lagrange point to be stable:

1. $C~:=~\det({\bf H})~\geq~0. $

2. $B~:=~{\rm tr}({\bf H})+4\Omega^2~\geq~0. $

3. $D~:=~B^2-4C~\geq~0. $

NB: The above theorem ignores higher-order terms in $V$, which could become important if $\det({\bf H})=0$.

The first two conditions are satisfied: $$C~=~\frac{27\Omega^4}{16}(1-\epsilon^2_{12})~>~0 \qquad\text{and}\qquad B~=~\Omega^2~>~0.\tag{8} $$ The third condition reads $$0~<~D~=~\frac{27\epsilon^2_{12}-23}{4}~=~27\left(\epsilon^2_2-\epsilon_2 +\frac{1}{27} \right) \tag{9}. $$ The roots of the quadratic equation $D=0$ are $$ \frac{m_2}{m_1+m_2}~\equiv~\epsilon_2~=~\frac{1}{2} \pm \frac{\sqrt{69}}{18}~\approx~\left\{\begin{array}{c}25/26,\cr 1/26.\end{array}\right.\tag{10}$$ This leads to the condition (4). $\Box$

References:

1. J. Binney & S. Tremaine, Galactic Dynamics, 2nd edition (2008); p. 181-182.

--

$^1$ Proof of theorem: The linearized EOM at a Lagrange point ${\bf z}={\bf 0}$ in the orbital plane $\mathbb{R}^2$ reads

$$\ddot{\bf z} ~\approx~\underbrace{2 \Omega ~{\bf J} \dot{\bf z}}_{\text{Coriolis}} - {\bf H} {\bf z}, \qquad {\bf J}~:=~\begin{pmatrix} 0 & 1 \cr -1 & 0 \end{pmatrix},\tag{11}$$

where the first term on the rhs. is the Coriolis force. The Hessian is a real symmetric matrix, and therefore diagonalizable with 2 principal axes. After a possible coordinate rotation ${\bf z}\mapsto e^{J\phi}{\bf z}$, we may assume that the Hessian

$${\bf H}~=~\begin{pmatrix} H_1 & 0 \cr 0 & H_2 \end{pmatrix}\tag{12}$$

is diagonal. (The rotation commutes with the Coriolis term (11) and leaves the 3 conditions in the theorem invariant!) The EOMs (11) are 2 coupled 2nd-order homogeneous ODEs with constant coefficients. Their characteristic equation is a 4th order equation $$0~=~\left| \begin{array}{cc} \lambda^2 +H_1& -2\Omega \lambda \cr 2\Omega \lambda & \lambda^2 +H_2 \end{array} \right| ~=~(\lambda^2)^2 +B\lambda^2 +C \tag{13},$$ which has 4 roots $$ \lambda^2~=~\frac{-B\pm \sqrt{D}}{2}\tag{14} .$$ Eq. (13) is a 2nd order eq. in $\lambda^2$. The solution to the 2 coordinates $z^1$ and $z^2$ are linear combinations of exponentials $e^{\lambda t}$, where $\lambda$ is a root. The condition of stability is that ${\rm Re}(\lambda)\leq 0$ for all 4 roots.

However due to symmetry in eq. (13), if $\lambda$ is a root, so is $-\lambda$. So the condition of stability is that ${\rm Re}(\lambda)=0$ for all 4 roots, i.e. that $\lambda$ is imaginary. Or equivalently, that $\lambda^2 \leq 0$ is non-positive for all 4 roots.

This is only possible if the discriminant $D\geq 0$ in eq. (14) is non-negative, i.e. the 3rd condition in the theorem.

The 1st & 2nd conditions $$ C,B~\geq~ 0\qquad\Leftrightarrow\qquad\lambda^2_{\pm}~\leq~0 \tag{15}$$ follow from the well-known fact that $$ C~=~ \lambda^2_+\lambda^2_-\qquad\text{and}\qquad-B~=~\lambda^2_++\lambda^2_- \tag{16} $$ in the 2nd order eq. (13) is the product and sum of its roots, respectively. $\Box$