$\displaystyle I=\int\frac{\sqrt{2t^2-1}}{t^2-1}dt$, so $\displaystyle t=\frac{1}{\sqrt{2}}\sec\theta, dt=\frac{1}{\sqrt{2}}\sec\theta\tan\theta d\theta$ gives
$\displaystyle I=\frac{1}{\sqrt{2}}\int\frac{\sec\theta\tan^2\theta}{\frac{1}{2}\sec^2\theta-1}d\theta=\sqrt{2}\int\frac{\sec\theta\tan^2\theta}{\sec^2\theta-2}d\theta=\sqrt{2}\int\frac{\sin^2\theta\cos\theta}{\cos^2\theta(1-2\cos^2\theta)}d\theta$.
Next letting $u=\sin\theta, du=\cos\theta d\theta$ gives
$\displaystyle I=\sqrt{2}\int\frac{u^2}{(1-u^2)(2u^2-1)}du=\sqrt{2}\cdot\frac{1}{2}\int\left(\frac{1}{1-u}+\frac{1}{1+u}-\frac{1}{\sqrt{2}u+1}+\frac{1}{\sqrt{2}u-1}\right)du$
$\displaystyle=\frac{1}{\sqrt{2}}\left[\ln\left(\frac{1+u}{1-u}\right)+\frac{1}{\sqrt{2}}\ln\left(\frac{\sqrt{2}u-1}{\sqrt{2}u+1}\right)\right]+C$
$\displaystyle=\frac{1}{\sqrt{2}}\left[\ln\left(\frac{1+\sin\theta}{1-\sin\theta}\right)+\frac{1}{\sqrt{2}}\ln\left(\frac{\sqrt{2}\sin\theta-1}{\sqrt{2}\sin\theta+1}\right)\right]+C=\frac{1}{\sqrt{2}}\left[2\ln\lvert\sec\theta+\tan\theta\rvert+\frac{1}{\sqrt{2}}\ln\left(\frac{\sqrt{2}\sin\theta-1}{\sqrt{2}\sin\theta+1}\right)\right]+C$
$\displaystyle=\sqrt{2}\ln\big|\sqrt{2}t+\sqrt{2t^2-1}\big|+\frac{1}{2}\ln\bigg|\frac{\sqrt{2t^2-1}-t}{\sqrt{2t^2-1}+t}\bigg|+C$
$=\displaystyle\sqrt{2}\ln\bigg|\sqrt{2}\cos x+\sqrt{\cos 2x}\bigg|+\ln\bigg|\frac{\cos x-\sqrt{\cos 2x}}{\sin x}\bigg|+C$