Which is faster between the tiger and the leopard given the relation between the number of paces they take and length of their paces?
I agree with David K that the book is wrong.
We can avoid both fractions and algebra (and thus reduce the opportunity for committing silly mistakes) by letting the animals run for a longer period of time -- for example five times the situation described in (I).
Then the tiger has run $20$ tiger paces while the leopard has run $25$ leopard paces.
By (II), the tiger's distance is less than $18$ leopard paces, and therefore certainly shorter than the $25$ leopard paces the leopard has run. So the leopard is definitely faster.
Or, for that matter, entirely without numbers. Derive from each of the givens:
Ia: The tiger takes fewer paces than the leopard.
IIa: A tiger pace is shorter than a leopard pace.
Your answer is correct (based on the paragraph where you explain the result in words). The book's solution is wrong.
The book makes at least two mistakes. It says that $\frac54$ paces of the leopard covers a distance less than $\frac{9}{10}x \cdot \frac54$ meters.
But the pace of the leopard was already defined to be $x$ meters exactly. Therefore $\frac54$ paces of the leopard covers a distance of $\frac54 x$ meters. (Mistake number one.) And that's exactly $\frac54 x$ meters, not "less than" as your book says. (Mistake number two.)
It is the pace of the tiger which must be less than $\frac{9}{10}x $ meters, given that the leopard's pace is $x$ meters. And indeed $\frac{9}{10}x < \frac54 x$ no matter what the value of $x$ is (provided that $x$ is positive, of course).
It appears that in the middle of writing the book's answer, whoever wrote it got confused about which was the tiger and which was the leopard.
I would use a bit of caution when expressing your arguments in formulas. According to your assignment of variables, the tiger travels the distance $4x$ in the same amount of time that the leopard travels the distance $5y.$ So there is a correspondence between $4x$ and $5y,$ but the two are not equal. As you observed, $x<\frac{9}{10}y.$ This implies that $4x<3.6y,$ and (as you also observed) $3.6y<5y.$ What you did not exactly say (but you appear to have been thinking) is that the two inequalities chain together to give $4x<5y,$ thereby proving that the leopard travels farther.