$x^2+y^2+z^2=5(xy+yz+zx)$ -- Is this all solutions?
Yes, all primitive solutions come from this, with the symmetries and negating all three entries at once. To get absolutely all, multiply these triples by any nonzero integer.
The matrix equation being solved by the computer, with a bound (9) on the matrix entries supplied by me, was $R^T G R = 196 H,$ where $$ G = \left( \begin{array}{rrr} 2 & -5 & -5 \\ -5 & 2 & -5 \\ -5 & -5 & 2 \end{array} \right) $$ and $$ H = \left( \begin{array}{rrr} 0 & 0 & -1 \\ 0 & 2 & 0 \\ -1 & 0 & 0 \end{array} \right) $$
The matrices are to be applied to the column vector
$$
\left(
\begin{array}{c}
u^2 \\
uv \\
v^2
\end{array}
\right)
$$
which gives all primitive solution vectors $(x,y,z)$ (up to $\pm$) to
$$ y^2 - zx = 0. $$
Put another way, all solutions come from these by multiplying by a nonzero integer.
Existence of an integer matrix of this type is guaranteed by Theorem I.9 on page 15 of PLESKEN; THAT IS, Automorphs of Ternary Quadratic Forms, by William Plesken, pages 5-30 in Ternary Quadratic Forms and Norms, (1982), edited by Olga Taussky. This is originally in pages 507-508 of Fricke and Klein (1897), which can be read online
I typed some extra characters at your matrix to make it easier to locate.
./homothety_indef 1 1 1 -5 -5 -5 0 196 0 0 -196 0 9
note that the final 9 is the bound on absolute values of matrix entries
Mon Mar 23 12:01:09 PDT 2015
-5 -9 -3
-3 3 1
1 -1 -5
-5 -1 1
-3 -9 -5
1 3 -3
-5 1 1
-3 9 -5
1 -3 -3
-5 9 -3
-3 -3 1
1 1 -5
-5 -9 -3
1 -1 -5
-3 3 1
-5 -1 1
1 3 -3
-3 -9 -5
-5 1 1
1 -3 -3
-3 9 -5
-5 9 -3
1 1 -5
-3 -3 1
-3 -9 -5
-5 -1 1
1 3 -3
-3 -3 1
-5 9 -3
1 1 -5
-3 3 1
-5 -9 -3
1 -1 -5
-3 9 -5
-5 1 1
1 -3 -3
-3 -9 -5
1 3 -3
-5 -1 1
-3 -3 1
1 1 -5
-5 9 -3
-3 3 1
1 -1 -5
-5 -9 -3
-3 9 -5
1 -3 -3
-5 1 1
-1 -3 3
3 9 5
5 1 -1
-1 -1 5
3 3 -1
5 -9 3
-1 1 5
3 -3 -1
5 9 3
-1 3 3
3 -9 5
5 -1 -1
-1 -3 3
5 1 -1
3 9 5
-1 -1 5
5 -9 3
3 3 -1
-1 1 5
5 9 3
3 -3 -1
-1 3 3
5 -1 -1
3 -9 5
1 -3 -3
-5 1 1
-3 9 -5
1 -1 -5
-5 -9 -3
-3 3 1
1 1 -5 =-=-=-=-=-=-=-=-=-=-=-=
-5 9 -3
-3 -3 1
1 3 -3
-5 -1 1
-3 -9 -5
1 -3 -3
-3 9 -5
-5 1 1
1 -1 -5
-3 3 1
-5 -9 -3
1 1 -5
-3 -3 1
-5 9 -3
1 3 -3
-3 -9 -5
-5 -1 1
3 -9 5
-1 3 3
5 -1 -1
3 -3 -1
-1 1 5
5 9 3
3 3 -1
-1 -1 5
5 -9 3
3 9 5
-1 -3 3
5 1 -1
3 -9 5
5 -1 -1
-1 3 3
3 -3 -1
5 9 3
-1 1 5
3 3 -1
5 -9 3
-1 -1 5
3 9 5
5 1 -1
-1 -3 3
5 -9 3
-1 -1 5
3 3 -1
5 -1 -1
-1 3 3
3 -9 5
5 1 -1
-1 -3 3
3 9 5
5 9 3
-1 1 5
3 -3 -1
5 -9 3
3 3 -1
-1 -1 5
5 -1 -1
3 -9 5
-1 3 3
5 1 -1
3 9 5
-1 -3 3
5 9 3
3 -3 -1
-1 1 5
-196 : 1 1 1 -5 -5 -5
-7529536 : 0 196 0 0 -196 0
Mon Mar 23 12:01:09 PDT 2015
Note that there is an annoying complication here about the prime $7.$ It is possible, precisely when my $v \equiv 5 u \pmod 7,$ to have all three of my $x,y,z$ divisible by $7.$ However, and i am still hand-waving here, when that happens, we can divide through by $7$ and produce the result from a different $(u,v)$ pair. The point, really, is that all the binary quadratic forms used are equivalent to $u^2 + 3 uv - 3 v^2$ of discriminant $21.$ When you divide through by $7,$ you get right back to that.
Tuesday, 24 March: pleased I was able to fill in the blanks as far as the prime $7.$ If we have the triple divisible by $7,$ it means we can write (using the original $(m,n)$ letters, $$ n = 5m + 7 t, $$ because we have $n \equiv 5m \pmod 7.$ All three formulas in the original question become divisible by $7,$ and we divide that out to get $$ \frac{m^2 +mn-5n^2}{7} = -17 m^2 - 49 mt - 35 t^2, $$ $$ \frac{-5m^2 +9mn-3n^2}{7} = -5 m^2 - 21 mt - 21 t^2, $$ $$ \frac{-3m^2 -3mn +n^2}{7} = m^2 + 7 mt + 7 t^2. $$
Proceeding by hand from here would be a mess, but I did a computer search to find a simultaneous substitution that does the desired thing; the (integer invertible!!) change of variables $$ m = 3r-2s , \; \; \; \; t = -2r + s. $$ The results are gratifying: we get $$ \frac{m^2 +mn-5n^2}{7} = r^2 + rs - 5 s^2, $$ $$ \frac{-5m^2 +9mn-3n^2}{7} = -3 r^2 - 3rs + s^2, $$ $$ \frac{-3m^2 -3mn +n^2}{7} = -5r^2 + 9rs - 3 s^2. $$ Combine this with a permutation and we have it.
Final comment: the matrices I called $R$ provided by the first computer run have determinant $\pm 196.$ That is, they are singular in the field of $7$ elements, but also singular in the field of $2$ elements. This seems bad, but is not. The eigenvectors with eigenvalue $0$ in $\mathbb Z / 2 \mathbb Z$ are $(0,1,1), \; \; $ $(1,0,1), \; \; $ $(1,1,0). \; \; $ However, we are applying such an $R$ only to $(u^2, uv,v^2)$ with integers $u,v$ relatively prime (not both even). The possible such vectors are $(1,1,1), \; \; $ $(1,0,0), \; \; $ $(0,1,0), \; \; $ $(0,0,1). \; \; $ So, this is never a problem.
If we rewrite the equation we qget: $16z^2−(3y+5z)^2+21(\frac{2x−5y−5z}{7})^2=0$
which is of the form $$16X^2+21Y^2=Z^2$$ Now to find all solutions, we need a primitive solution to this equation which can be determined by your formulas, and all solutions to this equation can be found by a quadratic formulas for more details see this answer (@Will Jagy)
And finally you can check if you have the same formals , if it's the case then you have covered all primitive solutions, if you don't have the same formulas you have to made an adequate change of variables.
November 2015:
I did an awful lot of work on the problem of integer variables $(x,y,z)$ in $$ A (x^2 + y^2 + z^2) - B(yz + zx + xy) = 0, $$ with integers $B > A > 0,$ also $\gcd(A,B) = 1,$ from about February to April, 2015. If there are any solutions, which requires both $B-A$ and $B + 2A$ to be expressible as $s^2 + 3 t^2$ in integers, then there is a very attractive type of solution. The fundamental observation is on pages 507-508 of FRICKE KLEIN (1897). The trick that can be used in this particular problem, any $(A,B),$ is that there are elements of order $3$ in the modular group $SL_2 \mathbb Z.$ The three binary quadratic forms displayed are "equivalent" to each other by the action of an order three element, with its square and cube (the identity). Very pretty the way it worked out, not something I could have understood ahead of time.
With $A=1, B=5,$ we need only one "recipe," $$ X_0 = 5 u^2 + 9 u v + 3 v^2, $$ $$ Y_0 = 3 u^2 -3 u v + v^2, $$ $$ Z_0 = - u^2 + u v + 5 v^2. $$ We rename these as $x,y,z$ and permute such that $|x| \geq |y| \geq |z|.$ Next, if $x < 0,$ we negate all three, with the overall convention that $$ x \geq |y| \geq |z|. $$ It is not obvious, but it turns out that $y$ is also positive here, this is just some inequalies with real numbers, nothing to do with integers. We wind up with $$ x \geq y \geq |z|. $$ With this in mind, we get all solutions by taking $u,v$ with $\gcd(u,v)=1.$ The part that was surprising, and quite unusual, was that we may also demand $u,v \geq 0,$ and still get all solutions. Finally, it is possible for $X_0, Y_0,Z_0$ to have a common factor, even though $u,v$ do not. We discard such imprimitive triples. Also, it is quite quick to find all solutions in a given large sphere around the origin, because $$ x^2 + y^2 + z^2 = 35 \left( u^2 + uv + v^2 \right)^2 $$
In the output below, compare the raw list of such ordered solutions, after the command line
isotropy_just_ordered 1 5 500
with the solutions produced from the triple of binary quadratic forms, after the command line
isotropy_binaries_combined 1 5 500 | sort -n
.................................
jagy@phobeusjunior:~$ ./isotropy 1 5
A = 1 B = 5
5 9 3
3 -3 -1
-1 1 5
end of A = 1 B = 5
B - 2 A = 3 B - A = 4 B + 2 A = 7
gcd( 4B-4A, B+2A) = 1
lambda = 7 t = 1 lambda t = 7
2 alpha - beta + 2 gamma = 7
alpha^2 + (alpha - beta + gamma)^2 + gamma^2 = 35
beta^2 - 4 alpha gamma = 21
matrix determinants = +/- 196 = 2^2 * 7^2
=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=
jagy@phobeusjunior:~$
jagy@phobeusjunior:~$ ./isotropy_just_ordered 1 5 500
5 3 -1
17 5 -1
41 5 3
59 47 -15
75 17 -1
89 83 -25
101 47 -15
111 17 5
129 125 -37
173 59 -15
185 131 -43
185 167 -51
201 83 -25
215 41 3
227 41 5
237 89 -25
251 215 -67
255 131 -43
293 255 -79
311 125 -37
327 269 -85
335 129 -37
353 75 -1
381 257 -85
383 101 -15
395 167 -51
425 419 -123
453 335 -109
461 75 17
479 257 -85
489 215 -67
=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=
jagy@phobeusjunior:~$ ./isotropy_binaries_combined 1 5 500 | sort -n
x y z first binary form u v
5 3 -1 < 5, 9, 3 > 1 0
17 5 -1 < 5, 9, 3 > 1 1
41 5 3 < 5, 9, 3 > 2 1
59 47 -15 < 5, 9, 3 > 1 3
75 17 -1 < 5, 9, 3 > 3 1
89 83 -25 < 5, 9, 3 > 1 4
101 47 -15 < 5, 9, 3 > 2 3
111 17 5 < 5, 9, 3 > 3 2
129 125 -37 < 5, 9, 3 > 1 5
173 59 -15 < 5, 9, 3 > 5 1
185 131 -43 < 5, 9, 3 > 2 5
185 167 -51 < 5, 9, 3 > 1 6
201 83 -25 < 5, 9, 3 > 3 4
215 41 3 < 5, 9, 3 > 4 3
227 41 5 < 5, 9, 3 > 5 2
237 89 -25 < 5, 9, 3 > 6 1
251 215 -67 < 5, 9, 3 > 1 7
255 131 -43 < 5, 9, 3 > 3 5
293 255 -79 < 5, 9, 3 > 2 7
311 125 -37 < 5, 9, 3 > 7 1
327 269 -85 < 5, 9, 3 > 1 8
335 129 -37 < 5, 9, 3 > 4 5
353 75 -1 < 5, 9, 3 > 5 4
381 257 -85 < 5, 9, 3 > 3 7
383 101 -15 < 5, 9, 3 > 7 2
395 167 -51 < 5, 9, 3 > 8 1
425 419 -123 < 5, 9, 3 > 2 9
453 335 -109 < 5, 9, 3 > 3 8
461 75 17 < 5, 9, 3 > 7 3
479 257 -85 < 5, 9, 3 > 4 7
489 215 -67 < 5, 9, 3 > 9 1
jagy@phobeusjunior:~$
In case anyone looks at the output, $(u,v) = (0,1)$ just repeats $(1,0),$ so I don't print that. With $(u,v)= (1,2),$ we get $x=35, y=21,z=-7,$ with a gcd of $7,$ so that is not printed either.