Both your and Matlab's answers are equivalent. The apparent discrepancy arises due to the fact that the argument of the logarithm of Matlab's answer you mentioned is negative for all real $x$ but the other logarithm is positive, since $x+\frac1x\ge2\forall x\in\mathbb R^+$(this can be proved by the A.M.-G.M. inequality or by differentiation), so $$\sqrt{\tan x}+\sqrt{\cot x}\geq2>\sqrt2>-\sqrt2$$
$$\implies\tan x+1>\sqrt{2\tan x}>-\sqrt{2\tan x}$$
To deal with logarithms of negative reals(or complex numbers in general), we need need step out of real analysis(where it is defined only for positive real numbers) and consider complex analysis, namely the concept of complex logarithms.
To extend the concept of logarithms from the reals to complex numbers in general, we define the complex logarithm $\log z$ of a complex number $z$ such that $e^{\log z}=z$, i.e., $\sum_{k=0}^\infty \frac{(\log z)^k}{k!}=z$. Note that there are infinitely many possible values for $\log z$ since the fundamental periods of $\sin x$ and $\cos x$ are $2\pi$: $$z=|z|e^{i\text{parg}z}=|z|(\cos\text{parg}z+i\sin\text{parg}z)=|z|e^{i(2n\pi+\text{parg}z)}$$
Hence, $\log z$ is a multi-valued function:
$$\log z=\ln|z|+i(\text{parg}z+2n\pi)$$
So, if $a\in\mathbb R^+$,
$$\log(-a)=\ln a+(2n+1)\pi i$$
$$\implies \log(\sqrt{2\tan x}-\tan x-1)=\ln(-\sqrt{2\tan x}+\tan x+1)+\underbrace{(2n+1)\pi i}_{\text{absorbed in }+C}$$
A point worth noting is that in real analysis, we consider the arbitrary constant of integration to be any real number, which is why the arguments of logarithms appearing in an anti-derivative are enclosed in moduli; but in the general case of complex analysis, it is an arbitrary complex number, so we write $\int\frac{\mathrm dx}{x}=\log x+C\forall x\in\mathbb C$, because:
$$\frac{\mathrm d}{\mathrm dz}e^{\log z}=1$$
$$\require{cancel}\implies\cancelto{z}{e^{\log z}}\frac{\mathrm d}{\mathrm dz}\log z=1$$
P.S. Perhaps the shortest way to evaluate the integral would be to perform the substitution $t=\sqrt{\tan x}$, then break the numerator strategically as follows:
$$\int\sqrt{\tan x}\mathrm dx=\int\frac{(t^2+1)+(t^2-1)}{t^4+1}\mathrm dt=\int\frac{1+\frac1{t^2}}{\left(t-\frac1t\right)^2+2}\mathrm dt+\int\frac{1-\frac1{t^2}}{\left(t+\frac1t\right)^2-2}\mathrm dt$$
$$\int\sqrt{\tan x},\mathrm{d}x=\Re!\left(F!\left(x\right)\right)$$
– gpmath Dec 08 '24 at 00:05