How do I bypass the limit for the tan function on the calculator?
You cannot do the inverse tangent because the range of inverse tangent is $(-\frac{\pi}{2},\frac{\pi}{2})$, and so your answer for step 2 (and also the answer from your calculator) is already incorrect.
I would suggest you do this:
Draw the triangle in the third quadrant (because it is in the range $(\pi, \frac{3\pi}{2})$, with adjacent side (on the x-axis) $x=-4$ and opposite side (on the y-axis) $y=-3$. The hypotenuse is $5$ by Pythagoras. Then, $\cos \theta = -\frac{4}{5}$. Using the cosine half-angle formula, $\cos \frac{\theta}{2}=\pm\sqrt{\frac{1+\cos \theta}{2}}=\pm\sqrt{\frac{1+(-\frac{4}{5})}{2}}=\pm\sqrt{\frac{1}{10}}=\pm \frac{\sqrt{10}}{10}$ Because the angle is in the range $(\pi, \frac{3\pi}{2})$, then half the angle is in the range $(\frac{\pi}{2}, \frac{3\pi}{4})$, so $\cos$ is negative. Therefore, your answer is $\boxed{-\frac{\sqrt{10}}{10}}$.
The "limit" you're trying to bypass is the fact that $\tan^{-1}$ only produces output values between $-\frac\pi2$ and $\frac\pi2.$ It will never directly give you an angle in the third quadrant.
The way you "get around" this is you remember that $$ \tan\theta = \tan(\theta + \pi) = \tan(\theta + 2\pi) = \tan(\theta - \pi),$$ that is, no matter what whole multiple of $\pi$ you add or subtract from $\theta,$ the tangent of the resulting angle is always the same as $\tan\theta.$
Since you know $\pi < \theta<\frac{3\pi}{2}$, you know that if you subtract $\pi$ from $\theta$ you will get an angle $\alpha = \theta - \pi$ where $0 < \alpha < \frac\pi2.$ Since $\tan\alpha = \tan\theta = \frac34,$ and $\alpha$ is an angle that you can get as an answer from $\tan^{-1},$ you know that $\alpha = \tan^{-1}\frac34.$ Therefore $\theta = \alpha + \pi = \pi + \tan^{-1}\frac34.$
To continue along the lines you were trying to follow, you will have to compute $\cos\frac{\pi + \tan^{-1}(3/4)}{2}.$ One way to continue is to use the cosine half-angle formula, just as you attempted, but with the correct angle, so you get $$ \cos\frac\theta2 = \cos\frac{\pi + \tan^{-1}\frac34}{2} = \pm\sqrt{\frac{1 + \cos\left(\pi + \tan^{-1}\frac34\right)}{2}}. $$ Now use the fact that $\cos(\pi + \beta) = -\cos(\beta).$ So now you have $$ \cos\frac\theta2 = \pm\sqrt{\frac{1 - \cos\left(\tan^{-1}\frac34\right)}{2}}. $$ You can get to the correct answer from there using similar methods to what you already used in your attempt for this question, but you get a different final answer since you have a difference inside the square root rather than a sum.
Alternatively, write $\cos\frac{\pi + \tan^{-1}(3/4)}{2} = \cos\left(\frac\pi2 + \frac{\tan^{-1}(3/4)}{2}\right),$ and use the fact that $\cos\left(\frac\pi2 + \phi\right) = -\sin\phi.$ Therefore $$ \cos\frac\theta2 = -\sin\left(\frac{\tan^{-1}\frac34}{2}\right) = \pm\sqrt{\frac{1 - \cos\left(\tan^{-1}\frac34\right)}{2}}. $$ The same result follows.
Having said all that, I think the easier way to solve this is not to "get around" the $\tan^{-1}$ function at all, but rather draw a triangle as recommended in the other answer. That answer uses straightforward geometric intuition that is much less likely to lead you astray.