One approach to simplifying radicals is to get repeated expressions out of the way to see things more clearly.
I. Simplification
So the expression shortens to,
$$
Z =\sqrt{
X^2 -\color{blue}v }
+ \sqrt{2 X^2 + \color{blue}v
+ \frac{2 X^3}{\sqrt{
X^2 - \color{blue}v}} }
- 3 X
$$
where $v$ is the cubic root,
$$\color{blue}v = \sqrt[3]{X^2 Y + \sqrt{\left(X^2 Y\right)^2 + \left(\frac{2}{3} Y\right)^3}}
+ \sqrt[3]{X^2 Y - \sqrt{\left(X^2 Y\right)^2 + \left(\frac{2}{3} Y\right)^3}}$$
In other words, it seems the OP is trying to solve a quartic.
To re-construct the quartic, the 1st step is to multiply the four conjugates and collect the variable $Z$,
$$P(Z)=
\left(Z -\Big(\sqrt{
X^2 -v } + \sqrt{2 X^2 + v + \tfrac{2 X^3}{\sqrt{ X^2 - v}}}
- 3 X\Big)\right)\\
\left(Z -\Big(\sqrt{
X^2 -v } - \sqrt{2 X^2 + v + \tfrac{2 X^3}{\sqrt{ X^2 - v}}}
- 3 X\Big)\right)\\
\left(Z -\Big({-\sqrt{X^2 -v}} + \sqrt{2 X^2 + v - \tfrac{2 X^3}{\sqrt{ X^2 - v}}}
- 3 X\Big)\right)\\
\left(Z -\Big({-\sqrt{X^2 -v}} - \sqrt{2 X^2 + v - \tfrac{2 X^3}{\sqrt{ X^2 - v}}}
- 3 X\Big)\right)=0
$$
hence,
$$P(Z) = Z^4 + 12 X Z^3 + 48 X^2 Z^2 + 64 X^3 Z + \frac{4v^3}{v-X^2} = 0$$
or simply,
$$\color{blue}{P(Z) = Z(Z+4X)^3\,(v - X^2) + 4 v^3 =0}\tag1$$
The 2nd step, re-construct its resolvent cubic,
$$P(v) = \prod_{k=0}^2\left(v-\Big(\sqrt[3]{X^2 Y + \sqrt{D}}\,\zeta_3^k+ \sqrt[3]{X^2 Y - \sqrt{D}}\,\zeta_3^{2k}\Big)\right) = 0$$
with discriminant $D=\left(X^2 Y\right)^2 + \left(\frac{2}{3} Y\right)^3$ and cube root of unity $\zeta_3 = e^{2\pi i/3}$ to get,
$$\color{blue}{P(v) = v^3 + 2 v Y - 2 X^2 Y=0}\tag2$$
The 3rd step, using resultants in Wolfram Alpha to eliminate $v$ between $(1)$ and $(2)$ yields,
$$Z^4 + 12X Z^3 + 48X^2 Z^2 + 64X^3 Z - 8 Y = 0$$
or more elegantly, the final quartic,
$$Z (Z + 4 X)^3 = 8 Y$$
II. Hypergeometric solution
Because of the special properties of the quartic,
$$Z (Z + 4 X)^3 = 8 Y$$
if a simple closed-form is wanted (and not necessarily in radicals), then it has a rather neat "cubed" generalized hypergeometric solution,
$$ Z = \frac{Y}{(2X)^3}\left({_3F_2}\Big(\tfrac14,\tfrac24,\tfrac34;\,\tfrac23,\tfrac43;\,-\tfrac{8Y}{27X^4}\Big)\right)^3$$
The OP prefers $X,Y \geq 0$ which implies the cubic discriminant $27X^4+8Y \geq 0$. So for example, let $X = 2$ and $Y = 3$, then the hypergeometric above evaluates as,
$$Z = 0.0460743\dots$$
which is the only positive real root of the quartic.
III. Euler's solution
If radicals are desired but not in the denominator, then the technique was found by Euler. Given the "depressed" form (which any quartic can be transformed to),
$$w^4+pw^2+qw+r = 0$$
then its four roots are among the 8 sign changes of,
$$w = \frac{\pm\sqrt{u_1}\pm\sqrt{u_2}\pm\sqrt{u_3}}2$$
with the $u_i$ as the three roots of the Euler resolvent cubic,
$$ u^3 + 2p u^2 + (p^2 - 4r)u - q^2 =0$$
IV. Depressing an equation
As requested by the OP, to depress an equation is to get rid of the $x^{n+1}$ term using a linear transformation. For the general quartic,
$$Z^4+aZ^3+bZ^2+cZ+d = 0$$
let $Z=w+t$, and collect the new variable $w,$
$$w^4 + (\color{blue}{a + 4 t}) w^3 + (b + 3 a t + 6 t^2) w^2 + (c + 2 b t + 3 a t^2 + 4 t^3) w +(d + c t + b t^2 + a t^3 + t^4) = 0$$
then solve $\color{blue}{a+4t = 0}$ for the variable $t$. The $w^3$ term vanishes and we get the depressed form,
$$w^4+pw^2+qw+r = 0$$
V. Example
Given,
$$w^4+3w^2+2w+1=0$$
then the Euler resolvent is,
$$u^3+6u^2+5u-4=0$$
with two conjugate roots of the quartic,
$$w_1 = \frac{+\sqrt{u_1}+\sqrt{u_2}+\sqrt{u_3}}2 \approx 0.3497 + 1.7469i$$
$$w_2 = \frac{-\sqrt{u_1}-\sqrt{u_2}+\sqrt{u_3}}2 \approx 0.3497 - 1.7469i$$
and similarly for the other pair.