How can you find a Pythagorean triple with $a^2+b^2=c^2$ and $a/b$ close to $5/7$?

Pythagorean triplets are characterized by being of the form $$a= m^2-n^2 \\ b= 2mn \\ c=m^2+n^2$$ So you could look for two integers $m>n$ such that $$\frac{m^2-n^2}{2mn} \approx \frac{5}{7}$$ or, equivalently, $$\frac{m}{n}- \frac{n}{m} \approx 2 \cdot \frac{5}{7}$$

Now, let $x$ be unique positive solution of $$x-x^{-1} = 2 \cdot \frac{5}{7}$$ one can easily compute that $x= \frac{5 + \sqrt{74}}{7} \approx 1.94$. The idea is to appximate $x= m/n$with a rational number. The continued fraction of $x$ is periodic and it is $[1; \overline{1, 16, 1,1,1}]$. This gives us the first convergents of $x$: $$\frac{1}{1} , \frac{2}{1} , \frac{33}{17} , \frac{35}{18} , \frac{68}{35} , \frac{103}{53} , \frac{171}{88},\dots$$ which are the best rational approximations of $x$. Taking $m= 33$ and $n= 17$ you get the triplet $$a= 800 ; \ \ b= 1122 ; \ \ c= 1378$$ which is not a primitive triplet. Dividing it by $2$ you get the primitive triplet $$a= 400 ; \ \ b= 561 ; \ \ c= 689$$ with $$\left| \frac{a}{b} - \frac{5}{7} \right|= \left| \frac{400}{561} - \frac{5}{7} \right|= \frac{5}{3927} \approx 0.0013$$

Going on like this ($m= 35 , n=18$) you get the next solution $$a= 901 ; \ \ b= 1260 ; \ \ c= 1549$$ with $$\left| \frac{a}{b} - \frac{5}{7} \right|= \left| \frac{901}{1260} - \frac{5}{7} \right|= \frac{1}{1260} \approx 0.0008$$ and so on.


The parametrization $c=m^2+n^2, a=m^2-n^2, b=2mn$ comes from the fact that $-m/n$ is the slope of the line connecting points $Q=(1,0)$ and $P=(a/c,b/c)$ of the unit circle.

When you want $a:b=5:7$, you want to be near the point $P=(5/\sqrt{74},7/\sqrt{74})$. In other words, you want $$ \frac mn\approx\frac{7}{\sqrt{74}-5}. $$ A way to find progressively better rational approximations to a given real number such as $7/(\sqrt{74}-5)$ is to expand it as a continued fraction.

Some such approximations $$ \begin{array}{c|c|c|c|c} m&n&c&a&b\\ \hline 33&17&1378&800&1122\\ 68&35&5849&3399&4760 \end{array} $$


$$\left(a \over b\right)^2 + 1 = \left(c \over b\right)^2 $$ $$\implies 1 = \left(c \over b\right)^2 - \left(a \over b\right)^2$$ $$\implies 1 = x^2 - y^2 \textrm{ where } x = {c \over b} \textrm{ and } y={a \over b}$$

So what you want to find is a rational point on the unit rectangular hyperbola for which $y \approx {5 \over 7}$.

The following is an exhaustive formula for the rational points on the unit hyperbola (derivable from Euclid's formula for the Pythagorean triples, and with a geometric meaning described by J. Lahtonen):

$$x = {1 + t^2 \over 1 - t^2}, y = {2t \over 1 - t^2}$$

Now we find the exact solution for ${2t \over 1 - t^2} = {5 \over 7}$, and then find rational approximations to the resulting $t$.

The exact value we get is $t = {\sqrt{74} \over 5} - {7 \over 5}$. We can get rational approximations using the continued fraction $[0; \overline{3, 8,3}]$. This gets you values of $t$ of $1/3, 8/25, 25/78...$