My solution has a few steps, so I would first use G Cab's answer to see if it is worth going further. That is if the distance from G Cab's answer is larger than the small circle radius, then they do not intersect. However, that approach does not tell you if the great circle segment intersects the small circle, and for that a bit more work may be required. This approach has the added benefit of finding the intersection points.
The steps are:
- Identify the planes normal to the great circle and small circle;
- Find the line of intersection of the two planes; and
- Find the points on the line of intersection that lie on the surface of the sphere.
Before we get there, any points are going to need to be transformed to cartesian coordinates. So for example given a latitude $\theta$ and longitude $\phi$, one would need to calculate the $[x,y,z]^T$ equivalent coordinates, e.g.:
$$\mathbf{x} = R \left[\begin{array}{c} \cos\theta\cos\phi \\ \cos\theta\sin\phi \\ \sin\theta\end{array}\right],$$
where $R$ is the radius of the sphere. This coordinate system puts the center of the sphere at the origin.
Step 1: Identify the planes normal to the great circle and small circle
Great Circle
Given two points on the surface of the sphere, $\mathbf{p}_1$ and $\mathbf{p}_2$, the vector normal to the great circle is
$$\mathbf{n}=\mathbf{p}_1\times\mathbf{p}_2.$$
That normal vector along with either point define the plane. (Below we use $\mathbf{p}_1$ but $\mathbf{p}_2$ could be used just as easily.)
Small Circle
The point $\mathbf{c}$ at the center of the small circle gives us the direction normal to the plane containing it. Knowing that the small circle is also defined based on some distance $r$ along the surface of the sphere, then the angle subtended from the line connecting the origin and $\mathbf{c}$ to a point on the small circle will have an arc $\alpha = r/R$ (in radians). Thus, with some simple trigonometry (SOHCAHTOA), we can show that an arbitrary point in the plane containing the small circle would be $\mathbf{p}_c = \mathbf{c}\cos\alpha = \mathbf{c}\cos(r/R)$.
2. Find the line of intersection of the two planes
From here, we can work through the mathematics to find the intersection of the points by following Krumm (2000). (Note that this part of the solution is based on findings in another post that includes a link to Matlab code if one wishes to use it.)
The direction of the line of intersection of the two planes is
$\mathbf{s} = \mathbf{c} \times \mathbf{n}$. This plus a point on the line gives us all we need to define the line of intersection between the planes.
Finding a point on the line is a bit more involved, as one must use Lagrange multipliers. Here we find a point $\mathbf{p}$ that is on the line and closest to an arbitrary point. For simplicity I used the origin, but it is straightforward to choose a different point as described below.
As the point $\mathbf{p}$ is on the line of intersection and thus is also in both planes, we have two constraints:
\begin{array}{ll}
\displaystyle\left(\mathbf{p}-\mathbf{p}_1\right)\cdot\mathbf{n} = 0; & \mathrm{and} \\ \\
\displaystyle\left(\mathbf{p}-\mathbf{p}_c\right)\cdot\mathbf{c} = 0. \\
\end{array}
Following Krumm (2000), looking to find the point on the line that is closest to the origin, we are looking to minimize $||\mathbf{p}||$ subject to these constraints, which we can combine into the function $w$:
$$w = ||\mathbf{p}||^2 + \lambda\left(\mathbf{p}-\mathbf{p}_1\right)\cdot\mathbf{n} + \mu \left(\mathbf{p}-\mathbf{p}_c\right)\cdot\mathbf{c}.$$
We must then take the derivates of $w$ and set them to zero:
$$\begin{array}{l}
\displaystyle
\frac{\partial w}{\partial p_x} = 2p_x + \lambda n_x + \mu c_x = 0 \\ \\
\displaystyle
\frac{\partial w}{\partial p_y} = 2p_y + \lambda n_y + \mu c_y = 0 \\ \\
\displaystyle
\frac{\partial w}{\partial p_z} = 2p_z + \lambda n_z + \mu c_x = 0 \\ \\
\displaystyle
\frac{\partial w}{\partial \lambda} = \mathbf{p} \cdot \mathbf{n} - \mathbf{p}_1 \cdot \mathbf{n} = 0 \\ \\
\displaystyle
\frac{\partial w}{\partial \mu} = \mathbf{p} \cdot \mathbf{c} - \mathbf{p}_c \cdot \mathbf{c} = 0
\end{array}$$
where we have decomposed $\mathbf{p}=\left[p_x, p_y, p_z\right]^T$, $\mathbf{n} = \left[n_x, n_y, n_z\right]$, and $\mathbf{c} = \left[c_x, c_y, c_z\right]^T$.
This gives us five equations and five unknowns which can be represented as,
$$
\left[\begin{array}{ccccc}
2 & 0 & 0 & n_x & c_x \\
0 & 2 & 0 & n_y & c_y \\
0 & 0 & 2 & n_z & c_z \\
n_x & n_y & n_z & 0 & 0 \\
c_x & c_y & c_z & 0 & 0
\end{array}\right]
\left[\begin{array}{c} p_x \\ p_y \\ p_z \\ \lambda \\ \mu \end{array}\right]
=
\left[\begin{array}{c} 0 \\ 0 \\ 0 \\
\mathbf{p}_1 \cdot \hat{\mathbf{n}} \\
\mathbf{p}_c \cdot \hat{\mathbf{c}}
\end{array}\right]
$$
(Note that if a location other than the origin had been picked, say instead some other point $\mathbf{q}$, then we would instead be looking to minimize $||\mathbf{p}-\mathbf{q}||$ and instead of zeros the first three terms on the right hand side of the equation would be $2q_x$, $2q_y$, and $2q_z$, respectively.)
3. Find the points on the line of intersection that lie on the surface of the sphere
Given the point $\mathbf{p}$ and the direction $\mathbf{s}$, we have an equation for the line
$$
\mathbf{g}(t)=\mathbf{p}+\mathbf{s}t,
$$
where $t$ is a parametric variable. From here, to see if the great circle crosses into the small circle, one needs just to find the values for $t$ such that $||g||^2=R^2$. Note that this will be quadratic in $t$:
$$||\mathbf{s}||^2t^2 + 2(p_xs_x + p_ys_y + p_zs_z)t + ||\mathbf{p}||^2 - R^2 = 0.$$
If two (real) values exist, then the great circle crosses through the small circle. If one exists, then the great circle is tangent to a spot on the small circle. If none exists, then they do not intersect at all.
Finally, say you have two points at $t_1$ and $t_2$ that give you solutions. To identify if they are in the path of the great circle segment between $\mathbf{p}_1$ and $\mathbf{p}_2$, we can define a new parameterization along the great circle by defining
$$\mathbf{p}_{1\perp} = \hat{\mathbf{n}}\times\mathbf{p}_1$$
This vector will be in the great circle plane and be perpendicular to $\mathbf{p}_1$, and have the property $||\mathbf{p}_1||=||\mathbf{p}_{1\perp}||=R$.
There exists some angle $0<\beta<2\pi$ such that
$$\mathbf{p}_2 = \mathbf{p}_1\cos\beta_{p_2} + \mathbf{p}_{1\perp}\sin\beta_{p_2},$$
and there are similar angles $\beta_{t_1}$ and $\beta_{t_2}$.
If $\beta_{t_1}<\beta_{p_2}$, then it is along the great circle segment. Of course, the same is true for $\beta_{t_2}$.
