The following summarises the example by @bof in the comments . . .
Let $E$ be any infinite group. Consider the power set $\mathcal{P}(E)=\{X\mid X\subseteq E\}$.
For $X, Y\in \mathcal{P}(E)$, the symmetric difference $X\vartriangle Y$ is given by $$X\vartriangle Y=(X\setminus Y)\cup(Y\setminus X).$$
Theorem: $(\mathcal{P}(E), \vartriangle)$ is an infinite group in which every non-identity element has order $2$.
Proof:
If $\mathcal{P}(E)$ were finite, so then must be $E$. Thus $\mathcal{P}(E)$ is infinite.
A proof that $(\mathcal{P}(E), \vartriangle)$ is a group with identity $\varnothing $ can be found here.
Finally, consider $\varnothing \neq A\in\mathcal{P}(E)$. Then $$\begin{align}
A\vartriangle A&=(A\setminus A)\cup(A\setminus A)\\
&=\varnothing \cup \varnothing\\
& =\varnothing.
\end{align}$$ Hence the order of each non-identity element of $(\mathcal{P}(E), \vartriangle)$ has order two.$\square$