Here is a step-by-step solution where each step is not too hard.
Step 1: Instead of doing a Weierstrass substitution, substitute $t=\cos x$ and take factors out to get
$$I=2a\int_{-1}^1\frac{\sqrt{(t-1/a)^2+b/a^2}}{\sqrt{1-t^2}}\,dt$$
The roots of the upper quadratic are always complex since $a,b>0$: $c=\frac{1+\sqrt{-b}}a$ and $c^*$.
$$I=2a\int_{-1}^1\frac{\sqrt{(t-c)(t-c^*)}}{\sqrt{1-t^2}}\,dt$$
Step 2: Perform a linear fractional transformation that keeps the integrand's poles at $\pm1$ but shifts the zeros at $c$ and $c^*$ to the imaginary axis, so as to leave only even powers of the polynomial under root. This is the most involved part. Define$\newcommand{Re}{\operatorname{Re}}$
$$A=\frac{1+|c|^2+|c^2-1|}{2\Re(c)}>1$$
$$A_1=A^2-2A\Re(c)+|c|^2>0\qquad A_2=A^2|c|^2-2A\Re(c)+1>0\qquad B=\frac{A_2}{A_1}>0$$
The actual substitution is $u=\frac{At+1}{t+A}$. After taking out more factors
$$I=2a\sqrt{A_1(A^2-1)}\int_{-1}^1\frac1{(u+A)^2}\sqrt{\frac{u^2+B}{1-u^2}}\,du$$
Define $g=2a\sqrt{A_1(A^2-1)}$ and move on.
Step 3: Multiply top and bottom by $(u-A)^2\sqrt{u^2+B}$ to get
$$I=g\int_{-1}^1\frac{(u-A)^2(u^2+B)}{(u^2-A^2)^2}\frac1{\sqrt{(u^2+B)(1-u^2)}}\,du$$
$$=g\left(\int_{-1}^1\frac{(u^2+A^2)(u^2+B)}{(u^2-A^2)^2}\frac1{\sqrt{(u^2+B)(1-u^2)}}\,du-A\int_{-1}^1\frac{(2u)(u^2+B)}{(u^2-A^2)^2}\frac1{\sqrt{(u^2+B)(1-u^2)}}\,du\right)$$
The two integrands are even and odd about zero respectively, so
$$I=2g\int_0^1\frac{(u^2+A^2)(u^2+B)}{(u^2-A^2)^2}\frac1{\sqrt{(u^2+B)(1-u^2)}}\,du$$
We perform a partial fraction decomposition of the rational part of the integrand:
$$I=2g\int_0^1\left(1-\frac{3A^2+B}{A^2-u^2}+\frac{2A^2(A^2+B)}{(A^2-u^2)^2}\right)\frac1{\sqrt{(u^2+B)(1-u^2)}}\,du$$
Step 4: Finally we use elliptic integrals (all arguments here follow Mathematica/mpmath conventions). Byrd and Friedman 213.11 gives$\newcommand{sn}{\operatorname{sn}}$
$$I=\frac{2g}{\sqrt{1+B}}\left(V_0-\frac{3A^2+B}{A^2-1}V_1+\frac{2A^2(A^2+B)}{(A^2-1)^2}V_2\right)$$
where
$$V_k=\int_0^{K(m)}\frac1{(1-n\sn^2u)^k}\,du\qquad m=\frac1{B+1}\qquad n=\frac1{1-A^2}$$
Step 5: B&F 336.00, .01, .02 gives solutions for the $V_k$.
$$V_0=K(m)\qquad V_1=\Pi(n,m)$$
$$V_2=\frac{nE(m) + (m-n)K(m) + (2nm+2n-n^2-3m)\Pi(n,m)}{2(n-1)(m-n)}$$
Putting everything together and simplifying a lot we get the final answer.
$$I=4a\sqrt{-n(A_1+A_2)}(E(m)-K(m)+(1-n)\Pi(n,m))$$