Sylvester's determinant identity states that if $A$ and $B$ are matrices of sizes $m\times n$ and $n\times m$, then
$$ \det(I_m+AB) = \det(I_n+BA)$$
where $I_m$ and $I_n$ denote the $m \times m$ and $n \times n$ identity matrices, respectively.
Could you sketch a proof for me, or point to an accessible reference?
Hint $ $ Work universally, i.e. consider the matrix entries as indeterminates $\:\!\rm a_{\:\!ij},b_{\:\!ij}.\,$ Adjoin them to $\,\Bbb Z\,$ to get the polynomial ring $\rm R = \mathbb Z[a_{\:\!ij},b_{\:\!ij}].\, $ In this polynomial ring $\rm R,$ compute the determinant of $\rm\, (1+A B) A = A (1+BA)\,$ then cancel the nonzero polynomial $\rm\, det(A)\, $ (valid by $\rm R$ a domain). $ $ Extend to non-square matrices by padding appropriately with $0$'s and $1$'s to get square matrices. Note that the proof is purely algebraic - it does not require any topological notions (e.g. (Zariski) density).
Alternatively we may employ Schur decomposition as follows
$$\rm\left[ \begin{array}{ccc} 1 & \rm A \\ \rm B & 1 \end{array} \right]\, =\, \left[ \begin{array}{ccc} 1 & \rm 0 \\ \rm B & 1 \end{array} \right]\ \left[ \begin{array}{ccc} 1 & \rm 0 \\ \rm 0 & \rm 1\!-\!BA \end{array} \right]\ \left[ \begin{array}{ccc} 1 & \rm A \\ \rm 0 & 1 \end{array} \right]\qquad$$
$$\rm\phantom{\left[ \begin{array}{ccc} 1 & \rm B \\ \rm A & 1 \end{array} \right]}\, =\, \left[ \begin{array}{ccc} 1 & \rm A \\ \rm 0 & 1 \end{array} \right]\ \left[ \begin{array}{ccc} \rm 1\!-\!AB & \rm 0 \\ \rm 0 & \rm 1 \end{array} \right]\ \left[ \begin{array}{ccc} 1 & \rm 0 \\ \rm B & 1 \end{array} \right]\qquad$$
See this answer for more on universality of polynomial identities, universal cancellation (before evaluation) and closely relation topics, and see also this sci.math thread on 9 Nov 2007.
(1) Start, for fun, with a silly proof for square matrices:
If $A$ is invertible, then $$ \det(I+AB)=\det A^{-1}\cdot\det(I+AB)\cdot\det A=\det(A^{-1}\cdot(I+AB)\cdot A)=\det(I+BA). $$ Now, in general, both $\det(I+AB)$ and $\det(I+BA)$ are continuous functions of $A$, and equal on the dense set where $A$ is invertible, so they are everywhere equal.
(1) Now, more seriously:
$$ \det\begin{pmatrix}I&-B\\\\A&I\end{pmatrix} \det\begin{pmatrix}I&B\\\\0&I\end{pmatrix} =\det\begin{pmatrix}I&-B\\\\A&I\end{pmatrix}\begin{pmatrix}I&B\\\\0&I\end{pmatrix} =\det\begin{pmatrix}I&0\\\\A&AB+I\end{pmatrix} =\det(I+AB) $$
and
$$ \det\begin{pmatrix}I&B\\\\0&I\end{pmatrix} \det\begin{pmatrix}I&-B\\\\A&I\end{pmatrix} =\det\begin{pmatrix}I&B\\\\0&I\end{pmatrix} \begin{pmatrix}I&-B\\\\A&I\end{pmatrix} =\det\begin{pmatrix}I+BA&0\\\\A&I\end{pmatrix} =\det(I+BA) $$
Since the leftmost members of these two equalities are equal, we get the equality you want.
We will calculate $\det\begin{pmatrix} I_m & -A \\ B & I_n \end{pmatrix}$ in two different ways. We have $$ \det\begin{pmatrix} I_m & -A \\ B & I_n \end{pmatrix} = \det\begin{pmatrix} I_m & 0 \\ B & I_n + BA \end{pmatrix} = \det(I_n + BA). $$ On the other hand, $$ \det\begin{pmatrix} I_m & -A \\ B & I_n \end{pmatrix} = \det\begin{pmatrix} I_m+AB & 0 \\ B & I_n \end{pmatrix} = \det(I_m + AB). $$
here is another proof of $det(1 + AB) = det(1+BA).$ We will use the fact that
the nonzero eigen values of $AB$ and $BA$ are the same and the determinant of a matrix is product of its eigenvalues. Take an eigenvalue $\lambda \neq 0$ of $AB$ and the coresponding eigenvector $x \neq 0.$ It is claimed that $y = Bx$ is an eigenvector of $BA$ corresponding to the same eignevalue $\lambda.$
For
$ABx = Ay = \lambda x \neq 0,$ therefore $y \neq 0.$ Now we compute $BAy = B(ABx) = B(\lambda x) = \lambda y.$ We are done with the proof.
Here's a determinant free proof.
Lemma: Let $A$ be an $m\times n$ matrix and $B$ be an $n\times m$ matrix over some field $F$. If $\lambda\,(\neq 0)\in F$ then $\dim\ker(\lambda I_m-AB)^k = \dim\ker(\lambda I_n-BA)^k$ for all $k\in\mathbb N$.
Proof:
Consider this linear map, which is in essence, restriction of $B$ to a subspace of its domain.
$\begin{array}{ccc}T_1:&\ker(\lambda I_m-AB)^k& \longrightarrow & \ker(\lambda I_n-BA)^k\\ &v&\longmapsto&Bv\end{array}\tag*{}$
Show that $T_1$ is injective. Well that's easy, just show that $\ker T_1$ is trivial.
Let $v\in\ker(T_1)\subseteq \ker(\lambda I_m-AB)^k$ then we have $T_1(v)=Bv =0$.
Also, $(\lambda I_m-AB)^k\,v=0$
$ \displaystyle\implies \left(\lambda^k I_m + \sum_{i\,=\,1}^k\binom{k}i\lambda^{k-i}(-AB)^i\right)v=0$
$\displaystyle \implies \lambda^k\, v - \left(\sum_{i\,=\,1}^k\binom{k}i\lambda^{k-i}(-AB)^{i-1}A\right)\!Bv=0$
$\displaystyle\implies \lambda^k \,v = 0$
$\implies v=0\quad\because\lambda\neq 0$
Thus, $v\in\ker T_1\implies v=0$.
Using first isomorphism theorem on $T_1$, we see that $\ker(\lambda I_m-AB)^k$ is isomorphic to some subspace of $\ker(\lambda I_n-BA)^k$.
$\therefore \dim\ker(\lambda I_m-AB)^k\leq \dim\ker(\lambda I_n-BA)^k$.
Likewise define,
$\begin{array}{ccc}T_2:&\ker(\lambda I_n-BA)^k& \longrightarrow & \ker(\lambda I_m-AB)^k\\ &w&\longmapsto & Aw\end{array}\tag*{}$
Show that $T_2$ is injective.
It follows that $\dim\ker(\lambda I_n-BA)^k\leq \dim\ker(\lambda I_m-AB)^k$.
Hence, we conclude that $\dim\ker(\lambda I_m-AB)^k = \dim\ker(\lambda I_n-BA)^k$.
Conclusions drawn from the lemma:
Considering $AB$ and $BA$ as matrices over a field in which both their characteristic polynomials split and using the above argument, we conclude that $\chi_{AB}(x)$ and $\chi_{BA}(x)$ are same except for multiplicity of the factor $(x-0)$.
Thus, $x^{n}\,\chi_{AB}(x) = x^{m}\,\chi_{BA}(x)$. (This balances out the degree on both sides.)
We have $x^{n}\det(I_m-AB)=x^{m}\det(I_n-BA)$.
Now, Sylvester's determinant equality is a particular case of $x=-1$. $\blacksquare$
Since this thread continues to see action, I guess maybe it's worth noting in passing that this can also be proved using the box product from mixed exterior algebra (at least over fields of characteristic zero): $$\begin{align*} \det(AB+I_m)&=\sum_{k=0}^m\mathop{\mathrm{tr}}\bigl(\,(AB)^{(k)}\mathbin{\square}I_m^{(m-k)}\,\bigr)\\ &=\sum_{k=0}^m\mathop{\mathrm{tr}}\bigl(\,(AB)^{(k)}\,\bigr)\\ &=\sum_{k=0}^n\mathop{\mathrm{tr}}\bigl(\,(BA)^{(k)}\,\bigr)\\ &=\sum_{k=0}^n\mathop{\mathrm{tr}}\bigl(\,(BA)^{(k)}\mathbin{\square}I_n^{(n-k)}\,\bigr)\\ &=\det(BA+I_n) \end{align*}$$ Here $M^{(k)}$ denotes the $k\times k$ compound matrix of $M$ consisting of $k\times k$ minor determinants.
