The trick is to write :
$$\begin{align*}A_3^2 &= \left\|\hat{h}_{\ell}- \mathbb E[\hat{h}_{\ell}] +\mathbb E[\hat{h}_{\ell}]-h\right\|_2^2\\
&=\left\|\hat{h}_{\ell}- \mathbb E[\hat{h}_{\ell}]\right\|_2^2 + \left\|\mathbb E[\hat{h}_{\ell}]-h\right\|_2^2 + 2\left\langle\hat{h}_{\ell}- \mathbb E[\hat{h}_{\ell}],\mathbb E[\hat{h}_{\ell}]-h\right\rangle\tag1\end{align*} $$
Using Fubini's theorem and the fact that $\mathbb E[\hat{h}_{\ell}]-h $ is non random, we have :
$$\begin{align*}\mathbb E\left[\left\langle\hat{h}_{\ell}- \mathbb E[\hat{h}_{\ell}],\mathbb E[\hat{h}_{\ell}]-h\right\rangle\right] &=\int_\Omega\int_\mathbb R\left(\hat{h}_{\ell}(x,\omega)- \mathbb E[\hat{h}_{\ell}(x)]\right)\cdot\left(\mathbb E[\hat{h}_{\ell}(x)] - h(x)\right) dx\ d\mathbb P(\omega)\\
&=\int_\mathbb{R}\int_\Omega\left(\hat{h}_{\ell}(x,\omega)- \mathbb E[\hat{h}_{\ell}(x)]\right)\cdot\left(\mathbb E[\hat{h}_{\ell}(x)] - h(x)\right) d\mathbb P(\omega)\ dx\\
&=\int_\mathbb{R}\left(\mathbb E[\hat{h}_{\ell}(x)] - h(x)\right)\int_\Omega\left(\hat{h}_{\ell}(x,\omega)- \mathbb E[\hat{h}_{\ell}(x)]\right)\ d\mathbb P(\omega)\ dx\\
&=\int_\mathbb{R}\left(\mathbb E[\hat{h}_{\ell}(x)] - h(x)\right)\mathbb E\left[\hat{h}_{\ell}(x)-\mathbb E[\hat{h}_{\ell}(x)]\right]\ dx\\
&= 0\tag2\end{align*} $$
(Note : I spelled it out in detail to make it clear, but it just boils down to $\mathbb E[\langle X, c\rangle] = \langle \mathbb E[X],c\rangle $ whenever $X$ is random and $c$ isn't)
Putting $(1)$ and $(2)$ together, we get
$$\begin{align*} \mathbb{E}[A_3^2]&=\mathbb{E}\left[\left\|\hat{h}_{\ell}-\mathbb{E}[\hat{h}_{\ell}]\right\|_2^2\right] + \mathbb E\left[\left\|\mathbb{E}[\hat{h}_{\ell}]-h\right\|_2^2\right]\\
&\stackrel{(\star)}{=}\mathbb{E}\left[\left\|\hat{h}_{\ell}-\mathbb{E}[\hat{h}_{\ell}]\right\|_2^2\right] + \left\|\mathbb{E}[\hat{h}_{\ell}]-h\right\|_2^2\end{align*} $$
As desired. (Equality $(\star)$ follows again from the fact that $\left\|\mathbb{E}[\hat{h}_{\ell}]-h\right\|_2^2$ is not random)
