Euler angles are not good for computing metrics as they have no proper symmetries. I suggest (since the question is tagged under quaternions) you convert your two orientations to quaternions $p$ and $q$. Then:
First, assuming by distance you mean how far apart are two rotations.
Second, knowing that quaternions have a double cover of the rotation manifold, that is, $q$ and $-q$ represent the same rotation.
Third, from second, say that the 'distance' from any quaternion $q=(w,x,y,z)$ to the identity quaternion $(1,0,0,0)$ makes sense if $w>0$. Otherwise, for $w<0$, the distance should be computed to the 'other' identity $(-1,0,0,0)$, which is indeed closer; or equivalently, one would find the distance between $-q$ and the true identity $(1,0,0,0)$.
Fourth, the distance between two quaternions $p$ and $q$ can be found as follows:
- take $r = p^* q$ the quaternion that joins $p$ to $q$. The distance from $p$ to $q$ is now equal to the distance from $r$ to $(1,0,0,0)$ (or to $(-1,0,0,0)$, see Third above).
- take the log, $u\theta = \log(p^* q)$, where $u=[u_x,u_y,u_z]$ is a unit vector defining the axis of rotation, and $\theta$ is half the rotated angle.
- the distance is the rotated angle, $\phi=2\theta$.
Note: having unit quaternions $q=[\cos(\phi/2), u_x \sin(\phi/2), u_y \sin(\phi/2), u_z \sin(\phi/2)]$, the log can be found by inspection, but if only the distance is sought, this distance $\phi$ can be found by just:
- $r = p^* q$ as in 1. above
- if $r_w<0$ do $r \gets -r$ (change sign of the quaternion $r$).
- $\phi=2 \arccos(r_w)$, where $r_w$ is the real part of $r$.
Fifth. You can use a shortcut by considering just double the angle between the two quaternions seen as 4'vectors, ie,
$$
\cos \theta = p^\top q \rightarrow \phi = 2\theta = 2\arccos (p^\top q)
$$
If $\cos\theta<0$, change the sign of one of the quaternions and start over.
All this info can be found here https://arxiv.org/abs/1711.02508