Another solution from relations:
You have the relation $xy=yx^2=yx^{-1}$. This means $y^{-1}xy=x^{-1}$. So $y$ conjugates $x$ to $x^{-1}$, and then $y^2$ conjugates $x$ to $(x^{-1})^{-1}=x$, and finally $y^3=1$ conjugates $x$ to $x^{-1}$. Thus $x=x^{-1}$, which is the same as saying $x^2=1$, and multiplying both sides by $x$ we get $1=x^3=x$.
EDIT: Here's a proof of my comment, that the group $G=\langle x,y\mid x^n=y^m=1, xy=yx^r\rangle$ has order $mn$ precisely when $r^m=1\pmod{n}$. I'll use the fact already given in the question body, that $G$ can have order at most $mn$ [this is seen by just listing all possible elements].
First, note that just like in my answer, we have $y^{-1}xy=x^r$, and thus $y^{-k}xy^k=x^{r^k}$. Letting $k=m$, we get
\begin{align}
x &= y^{-m}xy^m\\
&= x^{r^m}
\end{align}
And thus $r^m-1$ divides $|x|$. So if $r^m\neq1\pmod{n}$, then $|x|\le\gcd(n, r^m-1)<n$, and so $|G|<mn$.
Now suppose $r^m=1\pmod{n}$. We can show $|G|=mn$ by giving an explicit (permutation) representation of $G$, that has order $mn$. [To see why, this shows $|G|\ge mn$, and together with $|G|\le mn$ we get $|G|=mn$.]
We will work in the symmetric group $S_{n+m}$, and let our representative for $x$ be the $n$-cycle $(1, 2, \ldots, n)$. Note that $x^r$ then looks like $(1, r+1, 2r+1, \ldots)$. If we define the permutation $\alpha$ as
$$ \begin{pmatrix}
1 & 2 & \cdots & n\\
1 & r+1 & \cdots & (n-1)r+1
\end{pmatrix}$$
then we have $\alpha^{-1}x\alpha=x^r$. In fact, we can give an explicit formula for $\alpha$:
$$ \alpha(i) = (i-1)r+1\pmod{n}$$
Note that, for any positive integer $k$, we get $\alpha^k(i)=(i-1)r^k+1\pmod{n}$. Thus, once again letting $k=m$, we have
$$ \alpha^m(i) = (i-1)r^m+1\pmod{n}$$
Since $r^m=1\pmod{n}$, we see then that $\alpha^m(i)=i$ for all $i\in\{1, \ldots,n\}$.
Thus $|\alpha|$ divides $m$. Unfortunately, we're not quite done, since it's not always true $\alpha$ has order $m$ (to see an explicit example, try $n=m=4$ and $r=3$).
However, this is why we work in $S_{n+m}$, and not just $S_n$. We can take the representative for $y$ to be $\alpha\cdot(n+1, \ldots, n+m)$, and then $|y|=m$ and $y$ acts on $x$ the same way $\alpha$ does: $y^{-1}xy=x^r$.
Finally, if $H=\langle x,y\rangle\le S_{n+m}$, then $K=\langle x\rangle$ is normal in $H$, and $H/K\cong\langle y\rangle$. Since $|x|=n$ and $|y|=m$, we have $|H|=mn$, as desired.
