Adding Absolute value to a complex number: $ z+| z|=2+8i$

I would go about this differently. Since $|z| \in \mathbb{R}$, you know that $b=8$ immediately since $bi$ is the only imaginary term on the left and $8i$ - on the right.

Now the only thing is to find $a$...

UPDATE

We have the equation $$a + \sqrt{a^2+64} = 2$$ (hence $a<0$), which implies $$\sqrt{a^2+64} = 2-a$$ and now squaring will yield the desired result.

Yet another way: $\,z=2+8i-|z|\,$, so $\,\bar z =2-8i-|z|\,$, then multiplying the two:

$$\require{cancel} z \bar z = (2+8i-|z|)(2-8i-|z|) \;\iff\; \cancel{|z|^2} = \cancel{|z|^2} - 4|z| + 68 \;\iff\; |z| = 17 $$

Then, substituting back in the first equation: $\,z=2+8i-|z|=\ldots\,$

$a+bi+\sqrt{a^2+b^2} = 2 + 8i$ so

$a + \sqrt{a^2 + b^2} = 2$ and $b = 8$.

So $a + \sqrt{a^2 + 64} = 2$

So $\sqrt{a^2 + 64} = 2- a$

$a^2 + 64 = 4 -4a + a^2$

$4a = -60$

$a = -15$.

$z = -15 + 8i$.

....

To do what you were attempting

You have to realize that the $Re(z) = a + \sqrt{a^2 + b^2}$ and $Im(z) = b$. I think somehow you were thinking there were threee parts $Re(z)=a$ and $Im(z) = b$ and some $Weird(z) = \sqrt{a^2 + b^2}$ and that $z\overline z = Re^2(z) - Im^2(z) + Weird^2(z)$. That simply isn't true....

$(a+bi+\sqrt{a^2+b^2})(a - bi +\sqrt{a^2 + b^2}) = (2 + 8i)(2-8i)$

$(a + \sqrt{a^2 + b^2})^2 - b^2 = 4 - 64$

$2a^2 + b^2 + 2a\sqrt{a^2 + b^2} -b^2 = -60$

$a^2 + a \sqrt{a^2 + b^2} = -30$

which is a pain to solve but can be done.