So, we're starting at this integral:
$$I = \int_0^{\pi/2}\ln\lvert\sin(mx)\rvert\cdot \ln\lvert\sin(nx)\rvert\, dx$$
Now,
here is to use a special way of writing $\ln|\sin\theta|$ as an infinite sum:
$$\ln|\sin\theta| = -\ln 2 - \sum_{k=1}^{\infty} \frac{\cos(2k\theta)}{k}$$
This neat little formula works as long as $\theta$ isn't a multiple of $\pi$ (like $0, \pi, 2\pi$, etc.), which is fine for our integral since we're going from $0$ to $\pi/2$.
Let's plug this into our integral. We'll have:
$$\ln|\sin(mx)| = -\ln 2 - \sum_{j=1}^{\infty} \frac{\cos(2jmx)}{j}$$
$$\ln|\sin(nx)| = -\ln 2 - \sum_{k=1}^{\infty} \frac{\cos(2knx)}{k}$$
Substituting these into our integral $I$:
$$I = \int_0^{\pi/2} \left(-\ln 2 - \sum_{j=1}^{\infty} \frac{\cos(2jmx)}{j}\right) \left(-\ln 2 - \sum_{k=1}^{\infty} \frac{\cos(2knx)}{k}\right) dx$$
Now, we need to multiply these two big expressions. It's going to give us a few terms:
$$I = \int_0^{\pi/2} \left[ (\ln 2)^2 + (\ln 2)\sum_{k=1}^{\infty} \frac{\cos(2knx)}{k} + (\ln 2)\sum_{j=1}^{\infty} \frac{\cos(2jmx)}{j} + \left(\sum_{j=1}^{\infty} \frac{\cos(2jmx)}{j}\right)\left(\sum_{k=1}^{\infty} \frac{\cos(2knx)}{k}\right) \right] dx$$
We can handle this integral piece by piece.
First up, we have $\int_0^{\pi/2} (\ln 2)^2 dx$. Since $(\ln 2)^2$ is just a constant, this integral is easy:
$$\int_0^{\pi/2} (\ln 2)^2 dx = (\ln 2)^2 \int_0^{\pi/2} dx = (\ln 2)^2 \left[x\right]_0^{\pi/2} = \frac{\pi}{2}(\ln 2)^2$$
Next, let's look at the second term: $(\ln 2)\sum_{k=1}^{\infty} \frac{1}{k} \int_0^{\pi/2} \cos(2knx) dx$.
We need to evaluate $\int_0^{\pi/2} \cos(2knx) dx$. The antiderivative of $\cos(2knx)$ is $\frac{\sin(2knx)}{2kn}$. So,
$$\int_0^{\pi/2} \cos(2knx) dx = \left[\frac{\sin(2knx)}{2kn}\right]_0^{\pi/2} = \frac{\sin(kn\pi)}{2kn} - \frac{\sin(0)}{2kn} = \frac{\sin(kn\pi)}{2kn}$$
Since $k$ and $n$ are positive integers, $kn$ is also an integer. And we know that $\sin(\text{integer} \times \pi) = 0$. So, this whole second term becomes zero!
Similarly, the third term $(\ln 2)\sum_{j=1}^{\infty} \frac{1}{j} \int_0^{\pi/2} \cos(2jmx) dx$ will also be zero for the same reason.
Now for the last, and most interesting, part:
$$\int_0^{\pi/2} \left(\sum_{j=1}^{\infty} \frac{\cos(2jmx)}{j}\right)\left(\sum_{k=1}^{\infty} \frac{\cos(2knx)}{k}\right) dx = \sum_{j=1}^{\infty}\sum_{k=1}^{\infty} \frac{1}{jk} \int_0^{\pi/2} \cos(2jmx)\cos(2knx) dx$$To handle the product of cosines, we can use the identity $\cos A \cos B = \frac{1}{2}(\cos(A-B) + \cos(A+B))$. So,$$\cos(2jmx)\cos(2knx) = \frac{1}{2}(\cos(2(jm-kn)x) + \cos(2(jm+kn)x))$$Now we need to integrate this from $0$ to $\pi/2$:$$\frac{1}{2} \int_0^{\pi/2} [\cos(2(jm-kn)x) + \cos(2(jm+kn)x)] dx$$
Let's consider two scenarios:
If $jm - kn \neq 0$ and $jm + kn \neq 0$:
The integral becomes:
$$\frac{1}{2} \left[\frac{\sin(2(jm-kn)x)}{2(jm-kn)} + \frac{\sin(2(jm+kn)x)}{2(jm+kn)}\right]_0^{\pi/2} = \frac{1}{2} \left(\frac{\sin((jm-kn)\pi)}{2(jm-kn)} + \frac{\sin((jm+kn)\pi)}{2(jm+kn)}\right)$$
Since $j, m, k, n$ are all integers, both $jm-kn$ and $jm+kn$ are also integers. Therefore, their sines at $\pi$ will be zero, making the entire integral zero in this case.
Now, what if $jm - kn = 0$? This means $jm = kn$. Since $j, m, k, n \ge 1$, this can definitely happen. In this case, $jm + kn = 2jm \neq 0$. Our integral simplifies to:
$$\frac{1}{2} \int_0^{\pi/2} [\cos(0) + \cos(2(2jm)x)] dx = \frac{1}{2} \int_0^{\pi/2} [1 + \cos(4jmx)] dx$$Integrating this gives:$$\frac{1}{2} \left[x + \frac{\sin(4jmx)}{4jm}\right]_0^{\pi/2} = \frac{1}{2} \left(\frac{\pi}{2} + \frac{\sin(2jm\pi)}{4jm} - (0 + 0)\right)$$
Again, since $jm$ is an integer, $\sin(2jm\pi) = 0$. So, the integral in this special case is $\frac{1}{2} \cdot \frac{\pi}{2} = \frac{\pi}{4}$.
So, the sum becomes $\sum_{j=1}^{\infty}\sum_{k=1}^{\infty} \frac{1}{jk} \cdot \delta_{jm,kn} \cdot \frac{\pi}{4}$, where $\delta_{jm,kn}$ is the Kronecker delta, which is 1 if $jm = kn$ and 0 otherwise.
This simplifies to $\frac{\pi}{4} \sum_{j,k \ge 1 \text{ such that } jm=kn} \frac{1}{jk}$.
To figure out when $jm = kn$, let's bring in the greatest common divisor of $m$ and $n$. Let $d = \gcd(m,n)$. Then we can write $m = dm'$ and $n = dn'$, where $m'$ and $n'$ are coprime positive integers (they don't share any common factors other than 1).
The condition $jm = kn$ now becomes $j(dm') = k(dn')$, which simplifies to $jm' = kn'$.
Since $m'$ and $n'$ have no common factors, for this equality to hold, $j$ must be a multiple of $n'$, say $j = \lambda n'$ for some positive integer $\lambda$, and similarly, $k$ must be a multiple of $m'$, say $k = \lambda m'$ for the same $\lambda$.
Substituting these back into our sum:
$$\sum_{\lambda=1}^{\infty} \frac{1}{(\lambda n')(\lambda m')} = \sum_{\lambda=1}^{\infty} \frac{1}{\lambda^2 m'n'} = \frac{1}{m'n'} \sum_{\lambda=1}^{\infty} \frac{1}{\lambda^2}$$
We know a famous result that $\sum_{\lambda=1}^{\infty} \frac{1}{\lambda^2} = \zeta(2) = \frac{\pi^2}{6}$ (this is the Basel problem!).
So our sum becomes $\frac{1}{m'n'} \frac{\pi^2}{6}$. Now, let's put back $m' = m/d$ and $n' = n/d$:
$$\frac{1}{(m/d)(n/d)} \frac{\pi^2}{6} = \frac{d^2}{mn} \frac{\pi^2}{6}$$
Finally, the contribution of the fourth term to our integral $I$ is $\frac{\pi}{4} \left(\frac{d^2}{mn} \frac{\pi^2}{6}\right) = \frac{\pi^3 d^2}{24mn}$.
Putting it all together, the value of the integral $I$ is the sum of the first part and the fourth part (since the second and third parts were zero):
$$I = \frac{\pi}{2}(\ln 2)^2 + \frac{\pi^3 (\gcd(m,n))^2}{24mn}$$
And there you have it!
Let's quickly check the examples they gave:
If $m=n=1$, then $d=\gcd(1,1)=1$.
$$I = \frac{\pi}{2}(\ln 2)^2 + \frac{\pi^3 (1)^2}{24(1)(1)} = \frac{\pi}{2}(\ln 2)^2 + \frac{\pi^3}{24}$$
This matches the known result.
If $m=1, n=2$, then $d=\gcd(1,2)=1$.
$$I = \frac{\pi}{2}(\ln 2)^2 + \frac{\pi^3 (1)^2}{24(1)(2)} = \frac{\pi}{2}(\ln 2)^2 + \frac{\pi^3}{48}$$
This also matches the known result.