Finding a good Lyapunov function for $ \{ x'= y , y'= -4 x + 5x^3 - x^5 \}$

Hint.

The dynamical system has an integral which is

$$ \frac 12 y^2+\frac{x^6}{6}-\frac{5 x^4}{4}+2 x^2=C $$

Studying the level curves we have the following graphics

and for $0 < C < 0.915$ we have closed level curves around the origin characterizing a center.

This is a conservative system, since it's equivalent to $$ x'' = -4x + 5x^3 - x^5 = -V'(x) , $$ with the potential $$ V(x) = 2x^2 - \tfrac54 x^4 + \tfrac16 x^6 . $$ So $H(x,y)=\tfrac12 y^2 + V(x)$ is a constant of motion, and the trajectories follow the level curves of $H$, which near the origin look like ellipses $2 x^2 + \tfrac12 y^2 = C$. This means that the origin is neutrally stable (i.e., stable but not asymptotically stable).

Compare the two plots (on Wolfram Alpha):

• Phase portrait
• Level curves