You would be on the right track had the integral been
$$\int\frac{\mathrm dx}{(a^2x^2\color{red}{+}b^2)^\frac32}$$
Even if the integral was this, you have made a few mistakes:
$1.$ You have forgotten the arbitrary constant $+C$.
$2$. You have forgotten to write $\mathrm d\theta$ halfway through your solution, if you wrote it, you’d realize that after performing the substitution $x=\frac{b}{a}\tan\theta$, the differential $\mathrm dx$ also needs to be rewritten is terms of $\theta$ and $\mathrm d\theta$:
$$x=\frac{b}{a}\tan\theta\implies\mathrm dx=\frac{b}{a}\sec^2\theta\,\mathrm d\theta$$
$3.$ You have not specified the domain of $\theta$. This is important because:
- The substitution $x=x(\theta)$ must be bijective so that there is no ambiguity in back-substitution.
- This will have to be taken into consideration while taking square roots of even powers and back-substituting in the end.
For example, if you let $\theta\in\left(2n\pi-\frac\pi2,2n\pi+\frac\pi2\right)$ which is the most convenient domain in your integral as $\sec\theta>0\implies\sqrt{\sec^2\theta}=\sec\theta$, and the signs of $\sin\theta$ and $x$ are the same($\sin\theta$ is the anti-derivative of the given integral, see below), so we can write $$\sin\theta=\frac{ax}{\sqrt{a^2x^2+b^2}}, \cos\theta=\frac{b}{\sqrt{a^2x^2+b^2}}$$ but if $\theta\in[2n\pi, (2n+1)\pi)-\left\{2n\pi+\frac\pi2\right\}$, we need to bifurcate this into $2$ cases, $\theta\in\left[2n\pi, 2n\pi+\frac\pi2\right)$ and $\theta\in\left(2n\pi+\frac\pi2, (2n+1)\pi\right)$:
$$\theta\in\left[2n\pi, 2n\pi+\frac\pi2\right)\implies\sin\theta= \frac{ax}{\sqrt{a^2x^2+b^2}},\cos\theta=\frac{b}{\sqrt{a^2x^2+b^2}}$$
$$\theta\in\left(2n\pi+\frac\pi2, (2n+1)\pi\right) \implies\sin\theta= \frac{-ax}{\sqrt{a^2x^2+b^2}},\cos\theta=\frac{-b}{\sqrt{a^2x^2+b^2}}$$
which unnecessarily complicates things.
Also, there was no need to complicate the first part of your solution so much, you could’ve simply let the power in the denominator be $\frac32$, which would later get converted to $3$ due to $\sec^2\theta$.
Let me also share the correct solution for your reference:
$$\require{cancel}\begin{align}\int\frac{\mathrm dx}{(a^2x^2+b^2)^\frac32}&\overset{x=\frac{b}{a}\tan\theta, \text{ }\theta\in\left(-\frac\pi2,\frac\pi2\right)}{=}\int\frac{\frac{\cancel{b}}{a}\cancel{\sec^2\theta}\mathrm d\theta}{b^{\cancel{{3}}^{2}}\sec^\cancel{3}\theta}\\&=\frac1{ab^2}\int\cos\theta\,\mathrm d\theta\\&=\frac{\sin\theta}{ab^2}+C\\&=\frac{x}{b^2\sqrt{a^2x^2+b^2}}+C\end{align}$$
Now, coming back to your original integral
$$\int\frac{\mathrm dx}{(a^2x^2-b^2)^\frac32}$$
a possible method is to substitute $x=\frac{b}{a}\sec\theta, \theta\in(0,\pi)-\{\frac\pi2\}$ as we need $\tan\theta>0$ since we end up $\left(b^2\tan^2\theta\right)^\frac32$ upon performing the substitution. Another faster substitution is $x=\frac1v$ since this is a standard technique to integrate functions of the type $\dfrac1{(ax^2+b)\sqrt{cx^2+d}}$:
$$\begin{align}\int\frac{\mathrm dx}{(a^2x^2-b^2)^\frac32}&=-\operatorname{sgn}x\int\frac{v}{(a^2-b^2v^2)^\frac32}\mathrm dv\\&=\frac{\operatorname{sgn}x}{2b^2}\int\frac{\mathrm d(a^2-b^2v^2)}{(a^2-b^2v^2)^\frac32}\\&=\frac{-\operatorname{sgn}x}{b^2\sqrt v}+C\\&=\frac{-x}{b^2\sqrt{a^2x^2-b^2}}+C\end{align}$$
which is the same result obtained on replacing $b$ with $\pm ib$ in the integral of $(a^2x^2+b^2)^\frac{-3}2$.