Indeed the observations of Exodd and Mason are the key.
Consider a Jordan block of the form $J=\begin{pmatrix} \lambda & 1 \\ 0 & \lambda \end{pmatrix}$. Then $J^k=\begin{pmatrix}\lambda^k & k\lambda^{k-1}\\ 0 & \lambda^k\end{pmatrix}$ by induction for $k\geq 1$. Therefore $J^k\begin{pmatrix}0\\ 1\end{pmatrix}=\begin{pmatrix}k\lambda^{k-1}\\ \lambda^k\end{pmatrix}$. So if $\|\lambda\|=1$, then $\lim_{k\to\infty}\|J^k\|=\infty$. Clearly this can be done for Jordan blocks with $1$s off the main diagonal in higher dimension too (just multiply with $(0,1,0,0,\dots,0)^T$ instead).
This implies that the Jordan normal form of $A$ is $\begin{pmatrix}Id & 0 &0\\ 0& B&0\\ 0 &0&C\end{pmatrix}$ for some diagonal matrix $B$ with all diagonal entries having norm $1$ but being distinct from $1$, a matrix $C$ with $\rho(C)<1$ and the size of the identity block being equal to the multiplicity of the $1$ eigenvalue.
Since $(Id-C)\sum_{k=0}^{\infty}C^k=Id$, it follows that $\frac{1}{n}\sum_{k=0}^{n-1} C^k \to 0$.
Thanks to Exodd for the suggestion about the $B$ component.
Now let $\lambda\in\mathbb{C}\setminus\{1\}$ such that $|\lambda|=1$, i.e. let us take $\lambda=e^{2\pi i\alpha}$ for some $\alpha\in[0,1)$. If $\alpha\in\mathbb{Q}\cap[0,1)$, the sequence $\{\lambda^n\}_{n\in\mathbb{N}}$ is periodic and runs through the values $e^{2\pi i\frac{k}{N}}$ for $1\leq k\leq N$ for some $N$. Since $\sum_{k=0}^{N-1} e^{2\pi i\frac{k}{N}}=0$ (this can be seen geometrically or by multiplying the expression with $1-e^{2\pi i\frac{1}{N}}$), it follows that $\frac{1}{n}\sum_{k=0}^{n-1} \lambda^k\to 0$ as $n\to\infty$.
If $\alpha$ is irrational, one can use Weil's equidistribution theorem or something similar to argue that for an arbitrary $\varepsilon>0$ the frequency of naturals $n$ such that $\lambda^n=e^{2\pi i\beta}$ with $\beta\in[\beta_0-\varepsilon,\beta_0+\varepsilon]$ is the same as the frequency of $n$s such that $\beta\in[\frac{1}{2}+\beta_0-\varepsilon,\frac{1}{2}+\beta_0+\varepsilon]$ for any $\beta_0\in[0,1)$. Once again this implies $\frac{1}{n}\sum_{k=0}^{n-1} \lambda^k\to 0$ as $n\to\infty$. More formally, I think the exact fact that the limit is $0$ follows from Kinchin's identity by taking $f$ to be the exponential, see e.g. https://en.wikipedia.org/wiki/Equidistribution_theorem .
Therefore, $\frac{1}{n}\sum_{k=0}^{n-1} A^k\to\begin{pmatrix}Id & 0 \\0 & 0\end{pmatrix}$ as $n\to \infty$ (in the appropriate Jordan normal form) with the size of the $Id$ block as described before.