Why does dividing both sides of this system of equations to each other yields infinite "incorrect solutions"?
Dividing (or, in general, "combining") several equations can be used to derive new equations that must necessarily hold, but they are not equivalent to the original system. Consider for example:
$$ \begin{cases} x = 1 \\ y = 1 \end{cases} $$
The above is a well defined system of two equations with two unknowns, and obviously has the unique solution $x=y=1\,$. Dividing the equations, however, gives $\,\dfrac{x}{y}=1 \iff x=y\,$ with infinitely many solutions. But not all of those satisfy the original system, in fact just one does.
[ EDIT ] About the "how to solve the system" part, the following outlines one possible approach, though the calculations become rather tedious.
Let $\,r=\sqrt{x^2+y^2} \ge0\,$, then the system can be written as:
$$ \begin{cases} \begin{align} r(x-5) &= 6x+y \\ r(y-1) &= 6y-x-2 \end{align} \end{cases} \quad\iff\quad \begin{cases} (r-6)x - y = 5r \\ x + (r-6)y = r - 2 \end{cases} $$
Solving for $\,x,y\,$ gives:
$$ \begin{cases} x = \dfrac{5 r^2 - 29 r - 2}{r^2 - 12 r + 37} \\[10px] y = \dfrac{r^2 - 13 r + 12}{r^2 - 12 r + 37} \end{cases} \tag{*} $$
Substituting the above into $\,r^2=x^2+y^2\,$ then gives, successively:
$$ \begin{alignat}{2} && r^2(r^2 - 12 r + 37)^2 &= (5 r^2 - 29 r - 2)^2 + (r^2 - 13 r + 12)^2 \\ && r^6 - 24 r^5 + 192 r^4 - 572 r^3 + 355 r^2 + 196 r - 148 &= 0 \\ && (r - 1) (r^2 - 12 r + 37) (r^3 - 11 r^2 + 4) &= 0 \end{alignat} $$
The latter factorization gives the root $\,r=1\,$, and the cubic factor gives two more positive real roots. Substituting those back into $\,(*)\,$ gives the solutions in $\,x,y\,$.
When you divide the two equations, you're producing another consistent equation that must be satisfied (except, perhaps, where a denominator of the resulting equation vanishes, where the division operation entails dividing by zero) but not all solutions of the new equation need be a solution of the original equations.
In this case, you've found a new condition, namely, that all solutions must lie on the circle with the given equation, which may help you in finding the solution(s) of the system.
From this comment:
it produces only one equation while the original system had two. One equation generally removes one degree of freedom from the in this case original 2 degrees, so you get a 1-dimensional solution set.
Take an easier example:
$$ \begin{cases} x+y = 1 \\ x-y = 1 \end{cases} $$
The only solution is $(x,y)=(1,0)$. Normally you would get to the solution by adding both sides to get
$$ 2x=2 \tag{*}$$
and solve it as
$$ x=1 $$
But then you get back to (the one of) the original equations, substitute $x$ you already know and solve for $y$. This means when writing (*) you do really mean something like:
$$ \begin{cases} x+y = 1 \\ 2x = 2 \end{cases} $$
or
$$ \begin{cases} 2x = 2 \\ x-y = 1 \end{cases} $$
Note if your original system included a parameter $\alpha$ like this
$$ \begin{cases} x+y = 1 +\alpha \\ x-y = 1 -\alpha \end{cases} \tag{**} $$
then you could get the same (*)! When you solve it and get $x=1$, the form may suggest a single solution. But your domain is still two-dimensional and what you really get is
$$1x+0y=1$$
solved by
$$\{(x,y):x=1, y \in R \}$$
(or it may be $C$ instead of $R$ if you allow complex numbers). This is not a single solution. Choose any single solution from within this set and there is a certain value of $\alpha$ that makes the chosen solution solve (**).
In fact (*) aggregates solutions to all possible versions of (**). By going from (**) to (*) and solving the sole latter, you lose the information what $\alpha$ is.
So where is a similar parameter in your case, since you divide your original equations? Here:
$$ \begin{cases} \sqrt{x^2+y^2}\cdot \left(x-5\right)=\left(6x+y\right)\cdot\beta \\ \sqrt{x^2+y^2}\cdot \left(y-1\right)=\left(6y-x-2\right)\cdot\beta \\ \beta\neq0 \end{cases} \tag{***} $$
It leads to the circle you got but the circle doesn't depend on $\beta$. I haven't investigated if every single point of the circle solves (***) for some value of $\beta$ but it seems to me that every solution of (***) for any possible $\beta$ belongs to the circle. The circle aggregates solutions to different systems of equations.
Taking a single point that belongs to the circle, you don't know if it solves (***) for $\beta=1$ (which is your case) until you confront it with something that specifies $\beta=1$ somehow. It may be any of your two original equations, they implicitly set $\beta=1$.