We can see that our region of integration is given by $D = \{\,(x,y) \mid 0 \le x \le \sqrt{y}, y \in [0,2]\,\}$.
We can partition this region into disjoint subregions $D_1 = \{\,(x,y) \mid \frac{\sqrt{2}}{2}y \le x \le \sqrt{y}, y \in [0,2]\,\}$ and $D_2 = \{\,(x,y) \mid 0 \le x \le \frac{\sqrt{2}y}{2}, y \in [0,2]\,\}$.
Then, for $D_1$, our polar region of integration will be such that $x \le \sqrt{y}\,$:
$$x \le \sqrt{y} \, \longrightarrow \, r^2\cos^2 \theta \le r\sin \theta \, \longrightarrow \, r \le \frac{\sin \theta}{\cos^2 \theta} = \tan{(\theta)} \sec{(\theta)}$$
And $\frac{\sqrt{2}}{2}y \le x\,$:
$$\frac{\sqrt{2}}{2}y \le x \, \longrightarrow \, \frac{y}{x} \le \sqrt{2} \, \longrightarrow\, \theta \le \arctan{(\sqrt{2})}$$
Thus, in polar coordinates, we have ${D_1}' = \{\, (r, \theta) \mid 0 \le r \le \tan{(\theta)} \sec{(\theta)}, \theta \in [0,\arctan{(\sqrt{2})}]\,\}$
For $D_2$, we have $y \le 2\,$:
$$y \le 2 \, \longrightarrow \, r \sin \theta \le 2 \, \longrightarrow \, r \le 2 \csc \theta$$
Thus, in polar coordinates, we have ${D_2}' = \{\,(r, \theta) \mid 0 \le r \le 2\csc \theta, \theta \in [\arctan{(\sqrt{2})}, \frac{\pi}{2}]\,\}$.
And so, finally,
$$\begin{aligned}\int_{0}^{2}{\int_{0}^{\sqrt{y}}{4xy^{2} \, \mathrm{d}x} \, \mathrm{d}y} &= 4\int_0^{\arctan{(\sqrt{2})}}{\int_0^{\tan{(\theta)} \sec{(\theta)}}{r^4 \cos{(\theta)}\sin^2{(\theta)} \, \mathrm{d}r} \, \mathrm{d}\theta}\\ &+4\int_{\arctan{(\sqrt{2})}}^{\frac{\pi}{2}}{\int_0^{2 \csc\theta}{r^4 \cos{(\theta)}\sin^2{(\theta)} \, \mathrm{d}r} \, \mathrm{d}\theta} \end{aligned}$$
Alternatively, you could define a function $\rho \colon [0,\frac{\pi}{2}] \to \mathbb{R}$ by
$$\rho(\theta) = \begin{cases}\tan{(\theta)}\sec{(\theta)}, &\theta \in [0, \arctan{(\sqrt{2})}[\\
2\csc \theta, & \theta \in [\arctan{(\sqrt{2})}, \frac{\pi}{2}] \end{cases}$$
And then rewrite the integral as
$$=4\int_0^{\frac{\pi}{2}}{\int_0^{\rho{(\theta)}}{r^4 \cos{(\theta)}\sin^2{(\theta)} \, \mathrm{d}r} \, \mathrm{d}\theta}$$
