If I have a matrix of the form
$$\begin{pmatrix} A & -B \\ B & A \end{pmatrix}$$
where all blocks are square and have the same dimensions, how do I turn it into something like
$$\begin{pmatrix} X & Y \\ 0 & Z \end{pmatrix}$$
so that the determinant is easy to calculate using row and column operations?
By diagonalization in the $2\times 2$ case, we obtain the similarity relation $$ \pmatrix{A&-B\\B&A}=\pmatrix{-iI&iI\\I&I}\pmatrix{A-iB&0\\0&A+iB}\pmatrix{(i/2)I&(1/2)I\\-(i/2)I&(1/2)I}. $$ Whence $$ \det \pmatrix{A&-B\\B&A}=\det(A-iB)\det(A+iB)=\det(A^2+i(AB-BA)+B^2). $$ If $A$ and $B$ commute, this yields $\det(A^2+B^2)$.
By using Schur complement, the determinant of your block matrix can be calculated as $\det(A)\det(A+BA^{-1}B)$ if $A$ is invertible, or $\det(B)\det(B+AB^{-1}A)$ if $B$ is invertible. In the course of derivation of these formulae, the matrix is first converted to a block upper triangular form. See the above-linked Wikipedia article for details.
Edit: If you want a method that works in all cases or know nothing about $A$ and $B$ in advance, I think you'd better compute the determinant directly, rather than converting it into a block triangular matrix first, and then evaluating the determinants of the diagonal blocks individually. If you insist in turning the matrix into a block upper triangular form, you may perform an LU decomposition (see also here), but then the resulting matrices are triangular (which, of course, are also block triangular). You may consider other matrix decompositions as well (such as QR decomposition or singular value decomposition).
