If a group is $3$-abelian and $5$-abelian, then it is abelian

(I'll write $xy$ instead of $x*y$ for simplicity.)

$(ab)^5=a^5b^5\implies ababababab=aaaaabbbbb\implies babababa=aaaabbbb$

$(ab)^3=a^3b^3\implies ababab=aaabbb\implies baba=aabb$

From $baba=aabb$, we get $babababa=aabbaabb$. Combining with $babababa=aaaabbbb$ gives $aabbaabb=aaaabbbb$, so $bbaa=aabb$. Combining again with $baba=aabb$ gives $bbaa=baba$, so $ba=ab$. This is true for any $a,b\in Z$, so $Z$ is abelian.


A group $G$ is said to be $n$-abelian if $(ab)^n = a^nb^n$ for all $a,b\in G$. Given a group $G$, the "exponent semigroup of $G$" is defined to be $$\mathcal{E}(G) = \{n\in\mathbb{Z}\mid (ab)^n =a^nb^n\text{ for all }a,b\in G\}.$$ Properties of $\mathcal{E}(G)$ established by Levi (see this previous answer) imply that if $3,5\in\mathcal{E}(G)$, then $\mathcal{E}(G)=\mathbb{Z}$, giving your desired conclusion.

For a slightly less high-powered proof, note that in general we have that $$(ab)^n = a^nb^n \iff (ba)^{n-1}=a^{n-1}b^{n-1}$$ which can be obtained by cancelling a single $a$ and a single $b$. Therefore, if $n\in\mathcal{E}(G)$, then $$(xy)^{(n-1)^2} = \bigl( (xy)^{n-1}\bigr)^{n-1} = \bigl( y^{n-1}x^{n-1}\bigr)^{n-1} = (x^{n-1})^{n-1}(y^{n-1})^{n-1} = x^{(n-1)^2}y^{(n-1)^2}.$$ That is, $n\in\mathcal{E}(G)$ implies $(n-1)^2\in\mathcal{E}(G)$.

Here, you know $3\in\mathcal{E}(G)$, hence $(3-1)^2 = 4\in\mathcal{E}(G)$. So now we have that $3$, $4$, and $5$ are all in $\mathcal{E}(G)$, and as you note, it is known that if $\mathcal{E}(G)$ contains three consecutive integers then it contains all integers, hence $G$ is abelian.

(This is essentially jgnr's answer, only cast a bit more generally)

As an alternative, but also somewhat high-powered proof, we can follow Steve Ds good observation: by the theorem of Alperin quoted in the answered linked to above, a $3$-abelian group is a quotient of a subgroup of a direct product of abelian groups, groups of exponent $3$, and groups of exponent $2$; but the latter are abelian as well, so $G$ will be a quotient of a subgroup of a direct product of abelian groups and groups of exponent $3$. If you had an element that is not central (that does not commute with everything) then it would have to be an element of order $3$. But by the same characterization, since $G$ is $5$-abelian it is a quotient of a subgroup of a direct product of abelian groups, groups of exponent $4$, and groups of exponent $5$. So a noncentral element would have to have order dividing $20$. Since $\gcd(3,20)=1$, there can be no noncentral elements (it would have to have exponent $3$ and exponent $20=4\times 5$, hence exponent $1$, hence be trivial, hence it is central). So $G=Z(G)$, hence $G$ is abelian.