How to evaluate these integrals?
$$\int_0^{\pi/2}{x^2}\ln^2(\sin x)\ln(\cos x)dx \tag{1}$$ $$\int_0^{\pi/2}x\ln(\sin x)\ln^2(\cos x)dx \tag{2}$$
How to evaluate these integrals?
$$\int_0^{\pi/2}{x^2}\ln^2(\sin x)\ln(\cos x)dx \tag{1}$$ $$\int_0^{\pi/2}x\ln(\sin x)\ln^2(\cos x)dx \tag{2}$$
Put $$ I(a,b,c)=\int_0^{\frac{\pi }{2}} x^a \log ^b(\sin (x)) \log ^c(\cos (x)) \, dx $$ Our goal is to find $I(1,1,2)$ and $I(2,2,1)$. One may notice that they are log-integrals of weight 5 and 6.
Tools
Differentiating Beta function provides $I$ whenever $a=0$ $$ I(0,b,c)=\frac1{2^{b+c}}\left.\frac{\partial ^{b+c}B(p,q)}{\partial p^b\, \partial q^c}\right|_{b=c=1/2} $$
It is also used for other log-integrals.
Differentiating the formula $$ \int_{0}^{\pi/2} \cos (k x) \cos ^{s}( x)~dx=\frac{\pi \Gamma(s+1)}{2^{s}\Gamma(1+\frac{s+k}{2})\Gamma(1+\frac{s-k}{2})} $$ provides $I$ for $b=0$ and even $a$. $$ I(2n,0,c)=\pi(-1)^n\left.\frac{\partial ^{2n+c}}{\partial k^{2n}\, \partial s^c}\frac{\Gamma(s+1)}{2^{s}\Gamma(1+\frac{s+k}{2})\Gamma(1+\frac{s-k}{2})}\right|_{s=k=0} $$ Consider $$ \log(\sin x)=-\log2+\Re \text{Li}_1(e^{2ix}) \\ \log(\cos x)=-\log2+\Re \text{Li}_1(-e^{2ix}) $$ and IBP yields $$ I(a,1,0)=-\frac{ \log (2)}{a+1}\left(\frac{\pi }{2}\right)^{a+1}+a!~\Re\left(\frac{i}{2}\right)^{a+1} \left(-\zeta (a)+\sum _{k=0}^a \frac{(-\pi i)^k }{k!}\left(2^{-a+k-1}-1\right) \zeta (a-k+2)\right)\\ I(a,1,0)=-\frac{ \log (2)}{a+1}\left(\frac{\pi }{2}\right)^{a+1}+a!~\Re\left(\frac{i}{2}\right)^{a+1} \left((1-2^{1-a})\zeta (a)+\sum _{k=0}^a \frac{(-\pi i)^k }{k!} \zeta (a-k+2)\right) $$
The integral formula representation $$ \text{Li}_{n+1}(z)=\frac{(-1)^n}{n!}\int_0^1\frac{\log^n(x)}{z^{-1}-x}dx $$ can be extended to the whole complex plane, interpreted as analytic continuation. Values at $z>1$ are converted into ordinary values by the identity (straight forward through differentiating) $$ 2 \sum _{r=0}^{\left\lfloor \frac{n}{2}\right\rfloor } \frac{\eta (2 r) }{(n-2 k)!}\log ^{n-2 k}(z)+\frac{\log ^n(z)}{n!}+(-1)^n \text{Li}_n\left(-\frac{1}{z}\right)+\text{Li}_n(-z)=0 $$ with proper branch chosen, whose real part can be obtained by taking principal value integrals. Actually the only one needed here is $$ \int_0^1\frac{\log^4(x)-\log^4(2)}{1/2-x}dx=\int_0^1\frac{\log^4(x)-\log^4(2)}{1/2-x}dx=24\Re\text{Li}_5(2)\\=24 \text{Li}_5\left(\frac{1}{2}\right)-\frac{1}{5} \log ^5(2)+\frac{4}{3} \pi ^2 \log ^3(2)+\frac{8}{15} \pi ^4 \log (2) $$
Evaluation of $I(1,1,2)$
Notice that $$ \Im \left(\log ^3\left(\frac{e^{2 i x}+1}{2}\right) \log \left(\frac{e^{2 i x}-1}{2 i}\right)-\frac14\log ^4\left(\frac{e^{2 i x}+1}{2}\right)\right)\\ =-x^3 \log (\sin (x))-2 x^3 \log (\cos (x))+3 x \log (\sin (x)) \log ^2(\cos (x)) $$ Integrate from $0$ to $\dfrac\pi2$ $$ A=\Im\int_0^{\pi/2}\left(\log ^3\left(\frac{e^{2 i x}+1}{2}\right) \log \left(\frac{e^{2 i x}-1}{2 i}\right)-\frac14\log ^4\left(\frac{e^{2 i x}+1}{2}\right)\right)dx\\ =-I(3,1,0)-2I(3,0,1)+3I(1,1,2) $$ Let $z=e^{2ix}$, $A$ can be seen as a contour integral on the upper semi unit circle (with perturbations at the branch points $z=\pm1$), which can be safely deformed to the segment $(-1,1)$ (with small semi circle to bypass the origin). Hence $$ A=\Im \int_C\left(\log ^3\left(\frac{z+1}{2}\right) \log \left(\frac{z-1}{2 i}\right)-\frac14\log ^4\left(\frac{z+1}{2}\right)\right)\frac{dz}{2iz}\\ =\Im \left(\text{ PV} \int_{1}^{-1}\log ^3\left(\frac{x+1}{2}\right) \left(\log \left(\frac{1-x}{2 }\right)+i\frac\pi2-\frac14\log \left(\frac{x+1}{2}\right)\right)\frac{dx}{2ix}+R\right) $$ $R$ is the integral around the origin and residue theorem implies $$ R=\pi\log^4(2)-i\frac{\pi^2}4\log^3(2) $$ Now it suffice to find $$ A'=\text{ PV} \int_{-1}^{1} \left(\log ^3\left(\frac{x+1}{2}\right)\log \left(\frac{1-x}{2 }\right)-\frac14\log ^4\left(\frac{x+1}{2}\right)\right)\frac{dx}{2x}\\ =\int_{0}^{1} \left(\log ^3\left(\frac{x+1}{2}\right)\log \left(\frac{1-x}{2 }\right)-\frac14\log ^4\left(\frac{x+1}{2}\right)-\log ^3\left(\frac{1-x}{2}\right)\log \left(\frac{1+x}{2 }\right)+\frac14\log ^4\left(\frac{1-x}{2}\right)\right)\frac{dx}{2x} $$ Let $x\to \dfrac{1-x}{1+x}$ and ultilize $$ \log ^4\left(1+1/x\right)=\log ^4(x)-4 \log (x+1) \log ^3(x)+6 \log ^2(x+1) \log ^2(x)-4 \log ^3(x+1) \log (x)+\log ^4(x+1) $$ one has $$ A'=\int_0^1\left(\frac{\log ^4(x)}{4}-\frac{3}{2} \log ^2(x+1) \log ^2(x)+\log ^3(x+1) \log (x)\right)\frac{dx}{1-x^2}\\ =\frac14\underbrace{\left(\int_0^1\frac{\log^4(2)-\log ^4\left(1+1/x\right)}{1-x^2}dx+\int_0^1\frac{\log ^4(x+1)-\log^4(2)}{1-x^2}dx\right)}_{A_1}-\underbrace{\int_0^1\frac{\log ^3(x) \log (x+1)}{1-x^2}dx}_{A_2}+\frac12\underbrace{\int_0^1\frac{\log ^4(x)}{1-x^2}}_{93 \zeta (5)/4} $$ Firstly, $$ A_1=\underbrace{\int_0^1\frac{\log^4(2)-\log ^4\left(1+1/x\right)}{1-x^2}dx}_{x=y/(2-y)}+\underbrace{\int_0^1\frac{\log ^4(x+1)-\log^4(2)}{1-x^2}dx}_{x=1/y-1}\\ =-\frac12\int_0^1\frac{\log^4(y)-\log^4(2)}{1/2-y}dx=-12\Re\text{Li}_5(2)=-12 \text{Li}_5\left(\frac{1}{2}\right)+\frac{\log ^5(2)}{10}-\frac{2}{3} \pi ^2 \log ^3(2)-\frac{4}{15} \pi ^4 \log (2) $$
Employing common techniques, $A_2$ can be rewritten as combinations of Beta derivatives, thus $$ A_2=-\frac{7 \pi ^2 \zeta (3)}{16}+\frac{279 \zeta (5)}{32}-\frac{1}{16} \pi ^4 \log (2) $$ So $$ A=-3 \text{Li}_5\left(\frac{1}{2}\right)+\frac{7 \pi ^2 \zeta (3)}{16}+\frac{93 \zeta (5)}{32}+\frac{\log ^5(2)}{40}-\frac{5}{12} \pi ^2 \log ^3(2)-\frac{1}{240} \pi ^4 \log (2) $$ and $$ I(1,1,2)=\frac{A+2I(3,0,1)+I(3,1,0)}3\\ =-\text{Li}_5\left(\frac{1}{2}\right)+\frac{13 \pi ^2 \zeta (3)}{192} +\frac{155 \zeta (5)}{128}+\frac{\log ^5(2)}{120}-\frac{5}{36} \pi ^2 \log ^3(2)-\frac{49 \pi ^4 \log (2)}{2880} $$ Evaluation of $I(2,2,1) $
Simple substitution yields $$ I(2,2,1)=\frac{\pi^2}4I(0,1,2)-\pi I(1,1,2)+I(2,1,2) $$ The first is trivial, second evaluated above, and the third appears to be easier to handle.
Notice that $$ \Re ~\log ^4\left(\frac{e^{2 i x}+1}{2}\right) \log \left(\frac{e^{2 i x}-1}{2 i}\right)\\ =x^4 \log (\sin (x))-4 x^2 \log ^3(\cos (x))+4 x^4 \log (\cos (x))-6 x^2 \log (\sin (x)) \log ^2(\cos (x))+\log (\sin (x)) \log ^4(\cos (x)) $$ so $$ B=\Re\int_0^{\pi/2}\log ^4\left(\frac{e^{2 i x}+1}{2}\right) \log \left(\frac{e^{2 i x}-1}{2 i}\right)dx\\ =I(4,1,0)-3I(2,0,3)+4I(4,0,1)-6I(2,1,2)+I(0,1,4) $$ Since all but $I(2,1,2)$ can be evalutated from the tools above, it suffice to find $B$.
The contour method is the same as above, and $$ B=\Re \left(\text{ PV} \int_{1}^{-1}\log ^4\left(\frac{x+1}{2}\right) \left(\log \left(\frac{1-x}{2 }\right)+i\frac\pi2\right)\frac{dx}{2ix}+R'\right) $$
$$ R'=-\frac{1}{2} \pi \log ^5(2)+\frac{1}{4} i \pi ^2 \log ^4(2) $$
Substituve $x=2t-1$, one has $$ B'=-\frac\pi2\text{ PV} \int_{-1}^{1}\log ^4\left(\frac{x+1}{2}\right) \frac{dx}{2x}=\frac\pi4\text{ PV} \int_{0}^{1}\frac{\log^4(t)}{1/2-t}dt=\frac\pi4\cdot24\Re\text{Li}_5(2)\\ =6 \pi \text{Li}_5\left(\frac{1}{2}\right)-\frac{1}{20} \pi \log ^5(2)+\frac{1}{3} \pi ^3 \log ^3(2)+\frac{2}{15} \pi ^5 \log (2) $$ Now $B$ is known, and at last we have $$ I(2,2,1)=\frac{\pi ^3 \zeta (3)}{96}-\frac{17 \pi \zeta (5)}{64}+\frac{1}{8} \pi \zeta (3) \log ^2(2)-\frac{1}{24} \pi ^3 \log ^3(2)+\frac{1}{320} \pi ^5 \log (2) $$ which, surprisingly, only involves zeta values.