Proving that $\cos(\frac{\arctan(\frac{11}{2})}{3}) = \frac{2}{\sqrt{5}}$

The fact you want to prove is equivalent to $$ \frac{\arctan\left(\frac{11}{2}\right)}{3} = \arccos\left(\frac{2}{\sqrt 5}\right), $$ that is, $$ \arctan\left(\frac{11}{2}\right) = 3 \arccos\left(\frac{2}{\sqrt 5}\right), $$ that is, $$ \frac{11}{2} = \tan\left(3 \arccos\left(\frac{2}{\sqrt 5}\right)\right). $$

The angle $\arccos\left(\frac{2}{\sqrt 5}\right)$ is the angle opposite the shorter leg in a right triangle with legs $1$ and $2$, so $\arccos\left(\frac{2}{\sqrt 5}\right) = \arctan\left(\frac12\right)$, and the fact you want to prove is therefore equivalent to $$ \tan\left(3 \arctan\left(\frac12\right)\right) = \frac{11}{2}. $$

Using the triple-angle formula $$ \tan(3x) = \frac{3 \tan x - \tan^3 x}{1 - 3 \tan^2 x} $$ with $x = \arctan\left(\frac12\right)$, so $\tan x = \frac12$ and \begin{align} \tan\left(3 \arctan\left(\frac12\right)\right) & = \frac{3 \tan x - \tan^3 x}{1 - 3 \tan^2 x} \\ & = \frac{3 \left(\frac12\right) - \left(\frac12\right)^3} {1 - 3 \left(\frac12\right)^2} \\ & = \frac{11}{2} \end{align} which is what you needed to show.

Starting with

$\cos(\dfrac{\arctan(\frac{11}{2})}{3}) = \frac{2}{\sqrt{5}}$

this also means starting with the triangle of trig ratios drawn and Pythagoras theorem:

$\tan(\dfrac{\arctan(\frac{11}{2})}{3}) = \frac{1}{2} = t ,$

Now use the $ \tan 3 \theta = \dfrac{3 t - t^3}{1-3 t^2} \rightarrow \dfrac{11}{2} $ triple angle formula and simplify, done!

Let $\cos\theta=\frac2{\sqrt5}$. Then $$\cos3\theta=4\cos^3\theta-3\cos\theta=\frac2{5\sqrt5}$$ and so $$\tan3\theta=\frac{\sqrt{(5\sqrt5)^2-2^2}}2=\frac{11}2\ .$$