The point of this answer is to try to avoid using the fact that I already know how to construct the quaternions.
Looking at the relations in the algebra $A=\mathbb R\langle X,Y\rangle/(X^2+1, Y^2+1,XY+YX)$, note that since $X^2=Y^2=-1$, they are both units, and hence the third relation can be rewritten as $XYX^{-1}Y^{-1}=-1$, so that $A= \mathbb R\langle X,Y| X^2=Y^2=XYX^{-1}Y^{-1} = -1\rangle$. Now this is obviously a quotient of the group algebra of the group $Q= \langle a,b\mid a^2=b^2 = aba^{-1}b^{-1}, a^4=e\}$, a finitely generated group (since all we have done is weaken the condition that $a^2=-1$ to $a^4=1$, thus we immediately obtain:
Fact 1: There is a well-defined homomorphism of groups $\varphi\colon Q \to A^{\times}$ which induces a surjective homomorphism of algebras $\varphi\colon \mathbb R[Q]\to A$.
Next we show:
Fact 2: $4\leq |Q|\leq 8$, where $|Q|=8$ if and only if $a^2\neq e$ in $Q$.
Proof:
Let $z=a^2=b^2$. Since $a^4=(a^2)^2=(b^2)^2=e$ it follows that $z$ is a central involution. Since $z=aba^{-1}b^{-1}$ we also have $zba=ab$. Now for any $n \in \mathbb Z$, $a^n \in \{z,a,za=za\}$, $b^{n} \in \{b,z,bz=zb\}$, hence using $ab=zba$ it follows that any word $w$ in $\{a,b\}$ must be equal to one of
$$
\{e,a,b,za,zb,ab=zba,ba=zab,z\}.
$$
It follows that $|Q|\leq 8$. To see that $|Q|\geq 4$, note that $\theta\colon Q\to (\mathbb Z/2\mathbb Z)^2$ given by $\theta(a)=(1,0)$ and $\theta(b)=(0,1)$ is clearly well-defined and surjective, and $\ker(\theta)=\{e,z\}$. In particular, the words $\{e,a,b,ab\}$ give distinct elements of $Q$ (since their images in $(\mathbb Z/2\mathbb Z)^2$ are) and if $|Q|=4$ then we must have $z=a^2=e$. $\fbox{$\phantom{\cdot}$}$
Now if $z=e$ in $Q$, then $\varphi\colon \mathbb R[Q]\to A$ has $\varphi(a)^2=\varphi(e)=1$, so that $1=-1$ in $A$ and hence $A=\{0\}$. Conversely, if $z \neq e$ so that $|Q|=8$, then since $z \in Z(Q)$ and $z^2=e$, $z$ acts diagonalizably on any $Q$-representation $V$ with eigenvalues $\pm 1$, and the corresponding $z$-eigenspaces $V_{\pm 1}$ are $Q$-subrepresentations, so that $V_{-1}$ becomes an $A$-module. Since in $\mathbb R[Q]$ we have $1=(e+z)/2+(e-z)/2$, it follows that if $|Q|=8$ then $\mathbb R[Q]_{-1}$ is a $4$-dimensional, and hence $A\cong\mathbb R[Q]_{-1}$ as a $Q$-representation, so that $\dim(A)=4$.
Thus we have shown:
Fact 3: $\dim(A)=4$ if and only if $z\neq e$ in $Q$.
Now $z\neq e$ in $Q$, if and only if $Q$ has an irreducible complex representation $(V,\rho)$ with $\rho(z)=-1_V$, and since $ab=zba$, so that $\rho(a)\rho(b)=-\rho(b)\rho(a)$, we must have $\dim(V)>1$. Moreover, since $Q/\langle z\rangle \cong (\mathbb Z/2\mathbb Z)^2$, which has four $1$-dimensional irreducible representations all of which lift to irreducible $Q$-representations, and the sum of the squares of the dimensions of the isoclasses of irreducibles is equal to $|Q|$, there can only be one such irreducible $(V,\rho)$. Since $Q$ is finite, this representation is unitary, and since $Q$ has at most one isomorphism class of irreducible of dimension $2$, $V \cong V^*$ so that we may assume $V$ has a positive definite Hermitian inner product and $\rho\colon Q \to \mathrm{SU}(V)$.
Fact 4: There exists a unique up to conjugation group homomorphism $\rho\colon Q \to \mathrm{SU}(V)$.
Proof:
Such a homomorphism is given by a pair $(A,B)$ of elements of $\mathrm{SU}(V)$ satisfying $B^2=C^2=-1$ and $BCB^{-1}C^{-1}=-1$, or equivalently, $BCB^{-1} = C^{-1}$. Now every element of $\mathrm{SU}(V)$ is diagonalizable with orthogonal eigenvectors having eigenvalues $\lambda$ and $\lambda^{-1}$ for some $\lambda \in \mathbb C$, $|\lambda|=1$. Since $B^2=C^2=-1_V$ it follows both $B$ and $C$ have eigenvalues $\pm i$. Thus since $CBC^{-1} = B^{-1} = -B$, it follows that if $u_+$ is a basis of the $+i$ eigenspace of $B$ with $\|u_+\|=1$, then $u_- = C(u_+)$ is an eigenvector for $B$ with eigenvalue $-i$, and hence $\{u_+,u_-\}$ is an orthonormal basis of $V$ with respect to which $C$ and $B$ have matrices: $\left(\begin{array}{cc} 0 & -1 \\ 1 & 0\end{array}\right)$ and $\left(\begin{array}{cc} i & 0 \\ 0 & -i\end{array}\right)$ respectively. But then clearly $A^2=B^2=ABA^{-1}B^{-1}=-1_V$, so that they define the required two-dimensional irreducible representation. $\fbox{$\phantom{\cdot}$}$
By the preceding discussion, this establishes the fact that $|Q|=8$, and hence $\dim(A)=4$. Indeed the proof shows that $X\mapsto B$, $Y \mapsto C$ gives $V$ the structure of a complex $A$-module, and clearly the image of $A$ in $\text{End}(V)$ is $4$-dimensional over $\mathbb R$. Thus we could have simply decided to consider whether $A^{\times}$ had any self-dual two-dimensional unitary representations, but I have no idea why you might think to consider that without being led to it through the group $Q$. On the other hand, I don't know what led the OP to this definition of the algebra $A$, maybe it is possible to motivate this from other points of view?