At least for finite $A$, yes, that suffices to have a group.
Call $\theta_a$ and $\gamma_a$, respectively, the left and right translation maps by a fixed element $a\in A$. Now, by assumption, $\theta_a,\gamma_a\in \operatorname{Sym}(A)$ and (associativity) $\theta_a\theta_b=\theta_{ab}$. Therefore (closure) $\{\theta_a, a \in A\}\le \operatorname{Sym}(A)$, and hence $\exists \tilde e\in A$ such that $\theta_{\tilde e}=Id_A$. Likewise, being $\gamma_a\gamma_b=\gamma_{ba}$, $\exists \hat e\in A$ such that $\gamma_{\hat e}=Id_A$; but $\tilde e=\tilde e\hat e=\hat e$ and hence the left and the right identities coincide, say $e:=\tilde e=\hat e$.
Now, since $\theta_a,\gamma_a\in \operatorname{Sym}(A)$, then $\exists(!) \tilde b,\hat b \in A$ such that $\theta_a(\tilde b)=\gamma_a(\hat b)=e$ or, equivalently, $a\tilde b=\hat ba=e$; from this latter we get e.g. $\hat ba=a\hat b$, whence $a\tilde b=a\hat b$ or, equivalently, $\theta_a(\tilde b)=\theta_a(\hat b)$, and finally $\tilde b=\hat b=:a^{-1}$.