One answer is reached through the comment section. I thought I would add my own solution here, by showing that if one of $AB\setminus BA$ and $BA\setminus AB$ is non-empty, then they both are non-empty.
Assume $AB\setminus BA$ is non-empty, and pick an element $x = ab\in AB\setminus BA$ with $a\in A$ and $b\in B$. Then consider $x^{-1} = b^{-1}a^{-1}$. Clearly it's an element of $BA$. Assume, for contradiction, that $x^{-1}\in AB$. Then it would be possible to write $x^{-1} = a_0b_0$ with some $a\in A, b\in B$. But then $$x = (x^{-1})^{-1} = (a_0b_0)^{-1} = b_0a_0\in BA$$which contradicts the choice of $x$.
So if there exists an $x\in AB\setminus BA$, then $x^{-1}\notin AB$, and therefore $x^{-1} = b^{-1}a^{-1}\in BA\setminus AB$. Therefore, if $AB\setminus BA$ is non-empty, then so is $BA\setminus AB$. By repeating the exact same proof, but with $A$ and $B$ swapped, this implication also goes the other way.