Find the group with presentation $\langle x, y: xy=yx, x^5=y^3\rangle$.

Your answer can't be right, since multiplying any element in $ \mathbb Z/5\mathbb Z \times \mathbb Z $ by $ 5 $ gives you something of the form $ (0, 5x) $, and such an element can't be divided by $ 3 $ in this group, so there's no element to play the role of $ y $.

To find the isomorphism class, note that there is a surjective map $ \mathbb Z^2 \to G $ given by $ (n, m) \to x^n y^m $ (where $ G $ denotes your group) whose kernel is precisely $ \langle (5, -3) \rangle $. In other words, we're trying to find the quotient $ \mathbb Z^2/\langle (5, -3) \rangle $.

This quotient is isomorphic to $ \mathbb Z $, as you can see by noting that $ (5, -3) $ and $ (3, -2) $ form a basis of $ \mathbb Z^2 $, so $ G \cong \mathbb Z $.


Once you know the group is abelian, then your question can be converted into linear algebra. Namely, the given presentation tells you that your group is isomorphic to the kernel of the homomorphism $h : \mathbb Z^2 \to \mathbb Z$ given by $h(x,y) = 5x-3y$, in other words the kernel of the matrix $\begin{pmatrix} 5 \\ -3 \end{pmatrix}$, or to put it another way the solution set of the matrix equation $$(x \quad y) \begin{pmatrix} 5 \\ -3 \end{pmatrix} = 0 $$ To determine what that kernel is you can convert the matrix $\begin{pmatrix} 5 \\ -3 \end{pmatrix}$ into Smith normal form, which gives the matrix $$\begin{pmatrix} 1 \\ 0 \end{pmatrix} $$ From this it now easily follows that the kernel is isomorphic to $\mathbb Z$.