Finding the sign of a dot product

If you have vectors $\vec a$ and $\vec b$ that satisfy the given conditions, then replacing $\vec a$ by its inverse will flip the sign of $\vec a\cdot \vec b$, but leave the givens unchanged.

So you can't hope to find the sign of $\vec a\cdot \vec b$ based on those given values.

This question is equal to say:

One parallelogram has sides with lengths $2$ and $5$, and area $8$, what is the angle between them?

And it's clear that we have two answer:

The fact that $\vec{a}$ and $\vec{b}$ are vectors is really a red herring. With the information given, the equation is $64+ x^2= 100$ so $x^2= 36$ which has roots x= 6 and x= -6. There is no reason why the dot product of two vectors cannot be negative so both are solutions.