Find $(a,b)$ if $x^2-bx+a = 0, x^2-ax+b = 0$ both have distinct positive integers roots

Let the roots of $x^2 - ax +b$ be $r$ and $s$, and the roots of $x^2 - bx + a$ be $u$ and $v$. Then

$$\begin{align} r+s &= a\\ rs &= b\\ u+v &= b\\ uv &= a. \end{align}$$

Let, without loss of generality, $a \leqslant b$. Thus

$$\begin{align} uv &\leqslant u+v\\ \iff uv - u - v + 1 & \leqslant 1\\ \iff (u-1)(v-1) & \leqslant 1. \end{align}$$

So $u = 1$ or $v = 1$ or $u = v = 2$. But the roots are supposed to be distinct, hence $u = 1$ or $v = 1$. Without loss of generality, $u = 1$.

Thus $a = v = r+s$, $b = v+1 = rs$, so

$$rs = r+s+1 \iff (r-1)(s-1) = 2,$$

and that leaves $r = 2$, $s = 3$ (or vice versa), so $a = 5, b = 6$.


Suppose $x^2-ax+b$ and $x^2-bx+a$ have positive integer roots. Then $$x^2-ax+b=(x-c)(x-d)\qquad\text{and}\qquad x^2-bx+a=(x-e)(x-f),$$ for some positive integers $c$, $d$, $e$ and $f$. As the roots are must be distinct, we may assume without loss of generality that $c>d$ and $e>f$. Now compare coefficients.

Hint 1:

It follows that $c+d=a=ef$ and $cd=b=e+f$.

Hint 2:

If $c>d>1$ then $cd>c+d$.

Hint 3:

If $d>1$ and $f>1$ then $cd>c+d=ef>e+f=cd$, a contradiction.

Hint 4:

Without loss of generality we have $f=1$, so $e=c+d$ and $e+1=cd$.

Hint 5:

It follows that $(c-1)(d-1)=2$, so $c=3$ and $d=2$, and hence $e=5$.