Simplifying an expression containing inverse trigonometric functions

expr = 2/3 (J+2 Sqrt[3 B^2+4 J^2] Cos[1/3 ArcCos[-((J (9 B^2+8 J^2))/Sqrt[(3 B^2+4 J^2)^3])]]);
expr2 = Simplify[expr /. {B->r Cos[t]/√3, J->r Sin[t]/2 },{r>0, 0<t<2Pi}]
expr3 = expr2 /. e:Cos[ArcCos[_]/3 ] :> First[y/.Solve[Reduce[{y==e, 0<t<2Pi},t,Reals]/.c:_Cos:>TrigExpand[c],y,Reals]//Simplify//Normal//Union]//Simplify

Simplify[TrigExpand //@ expr3 /. {r->Sqrt[3 B^2 + 4 J^2], t->ArcTan[√3B, 2J]}, _∈ Reals]

Output

$\frac{1}{3} r \left(4 \cos \left(\frac{1}{3} \cos ^{-1}\left(-\frac{1}{4} (5+\cos (2 t)) \sin (t)\right)\right)+\sin (t)\right)$

$\frac{r \sqrt{\cos (2 t)+7}}{\sqrt{6}}$

$2 \sqrt{B^2+J^2}$

Another way to reduce expression.

expr = 2/3 (J + 
  2 Sqrt[3 B^2 + 4 J^2] Cos[
    1/3 ArcCos[-((J (9 B^2 + 8 J^2))/Sqrt[(3 B^2 + 4 J^2)^3])]]) //
Expand;

rule = Cos[
1/3 ArcCos[-((J (9 B^2 + 8 J^2))/Sqrt[(3 B^2 + 4 J^2)^3])]
] -> cc;

Solve for unknown simplified expression x

sol = Solve[x == expr /. rule, cc, Reals][[1, 1]] // Normal

(*   cc -> (-2 J + 3 x)/(4 Sqrt[3 B^2 + 4 J^2])   *)

eq1 = Equal @@ sol /. Reverse[rule]

(*   Cos[1/3 ArcCos[-((J (9 B^2 + 8 J^2))/
 Sqrt[(3 B^2 + 4 J^2)^3])]] == (-2 J + 3 x)/(
 4 Sqrt[3 B^2 + 4 J^2])   *)

eq2 = ArcCos[#] & /@ eq1 // 
   Simplify[#, 
0 < ArcCos[-((J (9 B^2 + 8 J^2))/Sqrt[(3 B^2 + 4 J^2)^3])] < Pi] &

(*    ArcCos[-((J (9 B^2 + 8 J^2))/Sqrt[(3 B^2 + 4 J^2)^3])] == 
      3 ArcCos[(-2 J + 3 x)/(4 Sqrt[3 B^2 + 4 J^2])]   *)

eq3 = TrigExpand[Cos /@ eq2]

(*   -((9 B^2 J Sqrt[(3 B^2 + 4 J^2)^3])/(3 B^2 + 4 J^2)^3) - (
8 J^3 Sqrt[(3 B^2 + 4 J^2)^3])/(3 B^2 + 4 J^2)^3 == -(J^3/(
2 (3 B^2 + 4 J^2)^(3/2))) + (3 J)/(2 Sqrt[3 B^2 + 4 J^2]) + (
9 J^2 x)/(4 (3 B^2 + 4 J^2)^(3/2)) - (9 x)/(
4 Sqrt[3 B^2 + 4 J^2]) - (27 J x^2)/(8 (3 B^2 + 4 J^2)^(3/2))  + (
27 x^3)/(16 (3 B^2 + 4 J^2)^(3/2))   *)

Solve[eq3, x, Reals] // Simplify

(*   {{x -> ConditionalExpression[2 J, B != 0]}, 
      {x -> ConditionalExpression[-2 Sqrt[B^2 + J^2], B != 0]}, 
      {x -> ConditionalExpression[2 Sqrt[B^2 + J^2], B != 0]}}   *)

I didn't examine further, why 2 J is given as solution for B!=0. It is the solution for B==0.

expr /. B -> 0 // Simplify[#, J \[Element] Reals] &