" It can probably be solved using beta/gamma function but I can't figure it out."
Well here's the beta function approach,
$$I=\int_0^\pi \sin ^2 \theta(1 + \cos \theta)^4\, d\theta$$
Use $\theta=2x,$
$$I=2\int_0^\frac{\pi}{2} \sin ^2 2x(1 + \cos 2x)^4 \,dx$$
$$I=2\int_0^\frac{\pi}{2} (4\sin^2 x \cos^2 x)(2\cos^2x)^4 dx\implies 128\int_0^\frac{\pi}{2} \sin^2 x \cos^{10} x\,dx$$
$$\frac{1}{128}I=\int_0^\frac{\pi}{2} \sin^2 x \cos^{10} x\,dx$$
Use the below formula,
$$\color{red}{\frac{1}{2}\beta(p, q) = \int_0^{\frac{\pi}{2}} \sin^{2p-1}x \, \cos^{2q-1}x \, dx}$$
On comparing,
$$p=\frac{3}{2}, q=\frac{11}{2}$$
$$\frac{1}{128}I=\frac{1}{2}\beta\left(\frac{3}{2}, \frac{11}{2}\right)$$
$$\boxed{I=64\beta\left(\frac{3}{2}, \frac{11}{2}\right)}$$
We have utilised one of the integral forms from the definition of beta function,
$$\boxed{I=\frac{21\pi}{16}}$$