Where was my mistake (integration by trig-substitution problem)?

This answer was posted prior to an edit made by the OP.

Note that we have

$$x^2+x+1=(x+1/2)^2+3/4\ne \frac34 \left(\frac43 (x+1/2)^2+\frac34 \right)=x^2+2+5/2$$

Your only mistake appears to be a failure to notice that $\displaystyle \frac{4}{3} \frac{\sqrt{3}}{2} \tan^{-1} \left( \sqrt{\frac 4 3} \left(x+\frac 1 2 \right)\right) $ is exactly the same thing as $ \displaystyle \frac 2 {\sqrt 3} \tan^{-1} \left( \frac{2x+1}{\sqrt 3} \right).$

First you have $$ \frac 4 3 \cdot \frac{\sqrt 3} 2 = \frac{4\sqrt 3}{\sqrt 3\sqrt 3 \cdot 2} = \frac 2 {\sqrt 3}. $$ And then $$ \sqrt{\frac 4 3} \left( x + \frac 1 2 \right) = \frac 2 {\sqrt 3} \left( x + \frac 1 2 \right) = \frac 1 {\sqrt 3} (2x+1). $$