Every element of $G$ can be written as product of zero or more factors $\in\{a,b,a^{-1},b^{-1}\}$, i.e., for $g\in G$ there exist $n\in\mathbb N_0$ and $x_i\in\{a,b,a^{-1},b^{-1}\}$, $1\le i\le n$, such that $g= x_1x_2\ldots x_n$. For $g\in G$ let $l(g)$ be the smallest $n$ that can be used.
Fix $g\in G$ and let $g=x_1x_2\ldots x_n$ with $n=l(g)$, $x_i\in\{a,b,a^{-1},b^{-1}\}$.
Lemma 1. If $1\le k<n-2$ then $x_{k+2}=x_k^{\pm1}$.
Proof. As $aa$ can be shortend to $a^{-1}$ and so on, we know that $x_{k+1}$ "uses the other letter" than $x_k$, hence $x_{k+2}$ "uses the same letter" again as $x_k$ and hence can be either the same as or the inverse of $x_k$. $_\square$
Lemma 2. If $1\le k\le n-3$ then $x_k=x_{k+2}^{-1}$.
If $2\le k\le n-2$ then $x_k=x_{k+2}^{-1}$.
Proof. Otherwise, with $u=x_k$, $v=x_{k+1}$ we have one of the following for $x_kx_{k+1}x_{k+2}x_{k+3}$:
$$\underbrace{uvuv}_4=(uv)^2=(uv)^{-1}=\underbrace{v^{-1}u^{-1}}_2$$
or
$$\underbrace{uvuv^{-1}}_4=uvuv\cdot v=\underbrace{v^{-1}u^{-1}v}_3$$
contradicting minimality of $n$.
The second claim follows by symmetry / considering the reverse word. $_\square$
This allows us to estimate the number of elements in $G$ with given length:
- There is one element, $e$, of length $0$
- There are up to four elements of length $1$
- There are up to $8$ elements of length $2$: Choose one of four first symbols, then choose one of two symbols with "the other letter" as second.
- There are up to $16$ elements of length $3$: Continuing from above, there are two choices for the third symbol (by lemma 1)
- If $n>3$ then there are at most $8$ elements of length $n$: Four choices for the first symbol, two for the second, then $x_3,\ldots, x_{n-1}$ are determined by lemma 2 (first sentence) and $x_n$ by lemma 2 (second sentence).
Especially, elements of length $7$ are of the form
$$\begin{align}\underbrace{xyx^{-1}y^{-1}xyx}_7 &=(xyx^{-1}y^{-1})^2yx^2\\
&=(xyx^{-1}y^{-1})^{-1}yx^{-1}\\
&=\underbrace{yxy^{-1}x^{-1}yx^{-1}}_6\end{align}$$
and hence cannot occur; nor can greater lengths.
We conclude
$$ |G|\le 1+4+8+16+8+8+8=53.$$
As $|G|$ must be a power of $3$ (Sylow), we conclude
$$ |G|\le 27.$$