Determining on what side a point lies relative to a pair of vectors

Question

Imagine I have 3 points A,B,C. I need to know what "side" a point P is in relative to A-B, B-C. That is, if acos(dot(AB,BC))=(30 degrees), then is point P in the partition of space defined by the acute angle made by the vectors or the space defined by 330 degrees. All coordinates are in 3D, but P would be projected onto the plane made by A,B,C. Essentially imagine wanting to classify a point into one of the 4 regions made by 2 intersecting vectors on a 2D plane.

Answer 1

You can indeed project P onto the plane made by the triangle ABC to get P'. One should than be able to test if the angle between BA and BP is smaller than the angle between BA and BC. If this is the case, the point P should be within the partition of space defined by the acute angle of the vectors A-B and A-C

Answer 2

This problem, now that it is clearly settled as a 2D issue boils down to a change of coordinates.

Let us consider 2 vectors, say for the sake of being concrete $\binom{2}{1}$ and $\binom{3}{5}$ (the generalization to any pair of non-proportional vectors is straightforward).

These vectors define axes passing through the origin directed by these vectors. We want to be able to situate a point $P'\binom{x}{y}$ in one of the four "quadrats" defined by these axes. It amounts to be able to obtain coordinates $\binom{X}{Y}$ of $P'$ with respect to the new axes. Combining the 2 signs of $X$ and $Y$ will indicate in which case you are :

$$\begin{cases} X \geq 0 \ \& \ Y \geq 0 & \text{quadrat I}\\ X \leq 0 \ \& \ Y \geq 0 & \text{quadrat II}\\ X \leq 0 \ \& \ Y \leq 0 & \text{quadrat III}\\ X \geq 0 \ \& \ Y \leq 0 & \text{quadrat IV}. \end{cases}$$

How to obtain $(X,Y)$ knowing $(x,y)$ ?

By expressing, in a first step, the definition of coordinates $X$ and $Y$ :

$$\binom{x}{y}=X\binom{2}{1}+Y\binom{3}{5}=\underbrace{\begin{pmatrix}2 & 3\\1 & 5\end{pmatrix}}_{M}\binom{X}{Y}$$

Which is equivalent to the final formula :

$$\binom{X}{Y}=\underbrace{\tfrac17\begin{pmatrix} \ \ 5 & -3\\-1 & \ \ 2\end{pmatrix}}_{M^{-1}}\binom{x}{y}$$

That's all...

---

Old solution before the explanation of the OP :

It suffices to test the sign of the following determinant :

$$\det(\vec{AB},\vec{AC},\vec{AP})=\begin{cases}>0 & \ \text{(on one side of the plane)}\\=0 & \ \text{(on the plane)}\\<0 & \ \text{(on the other side)} \end{cases}$$

Indeed, the transition case where the determinant is $0$ corresponds to the case where $\vec{AP}$ is a linear combination of the first two $\vec{AB}$ et $\vec{AC}$.