$x = \sec\theta, \mathrm dx = \sec\theta \tan\theta\mathrm d\theta$
$$\begin{align}\int \frac{\mathrm dx}{(x^2-1)^{\frac{3}{2}}}&=\int \frac{\mathrm dx}{(\tan^2\theta)^{\frac{3}{2}}}\\&=\int \frac{\mathrm dx}{\tan^{\frac{7}{2}}\theta}\\&=\int \frac{\sec\theta \tan\theta}{\tan\theta ^{\frac{7}{2}}}\mathrm d\theta\\&=\int \tan^{\frac{-5}{2}}\theta \sec\theta\mathrm d\theta\end{align}$$
Here is where I get stuck...I tried converting $\tan\theta$ and $\sec\theta$ in terms of $\cos\theta$ and $\sin\theta$, but that didn't seem to get me anywhere...What is my next move from here? Did I even start this problem correctly? I can't tell :(
Update with more work after initial answers:
$$\int \frac{\cos\theta}{\sin^2\theta}\mathrm d\theta$$ $u = \sin\theta, \mathrm du = \cos\theta\,\mathrm d\theta$
I found $\sin\theta = \frac{\sqrt{x^2-1}}{x}$
$$\implies \int \frac{\cos\theta}{\sin^2\theta}\mathrm d\theta=\int \frac{\mathrm du}{u^2} = \frac{1}{ \frac{1}{3}u^3}+C= \frac{1}{3\sin^3\theta}+C= 3 \bigg( \frac{x}{\sqrt{x^2-1}} \bigg)^3+C$$