While watching a video about dot products (https://www.youtube.com/watch?v=WDdR5s0C4cY), the following formula is presented for finding the angle between two vectors:
For vectors $a$, and $b$, $$\cos( \theta ) = \frac{(a, b)}{ \| a \| \| b \| }$$ where $(a,b)$ is the dot product of $a$ and $b$.
How is this formula derived?
There are several derivations of this online. Here's where you can start.
Define two vectors $\textbf{a}$ and $\textbf{b}$. Then $ \textbf{a} - \textbf{b}$ is the vector that connects their endpoints and makes a triangle.
Therefore, we have a triangle with side lengths $|\textbf{a}|$, $|\textbf{b}|$, and $|\textbf{a} - \textbf{b}|$. Let the angle between the two vectors be $\theta$. By the Law of Cosines, we have
$$|\textbf{a} - \textbf{b}|^2 = |\textbf{a}|^2 + |\textbf{b}|^2 - 2 |\textbf{a}| |\textbf{b}| \cos (\theta)$$
Now, use the fact that $$ \begin{align*} |\textbf{a}- \textbf{b}|^2 &= (\textbf{a}- \textbf{b}) \cdot (\textbf{a}- \textbf{b})\\ &= \textbf{a} \cdot \textbf{a} - 2 (\textbf{a} \cdot \textbf{b}) + \textbf{b} \cdot \textbf{b} \\ &= |\textbf{a}|^2 - 2 (\textbf{a} \cdot \textbf{b}) + |\textbf{b}|^2 \end{align*} $$
Simplify this equation, and you will get the desired formula.
Since $(a,b)$ is independent of the basis, just choose a basis where $a$ lies along the $x$ axis: $a=(|a|,0)$, so that: $$ {(a,b)\over|a||b|}={b_x\over|b|}=\cos(\theta), $$ where $\theta$ is the angle between $b$ and $x$ axis (as you may recall from trigonometry), which is the same as the angle between $a$ and $b$.
It's a trig identity, really.
Two vectors determine a plane. Without loss of generality, we'll take that to be the $xy$-plane. Now, let $A$ and $B$ be the angles that $\overrightarrow{a}$ and $\overrightarrow{b}$ make with the (positive) $x$-axis; and let $r$ and $s$ be the respective magnitudes. Then, the $xy$-coordinates of the vectors can be written as
$$\overrightarrow{a}= r\,(\cos A, \sin A) \qquad\text{and}\qquad \overrightarrow{b} = s \,(\cos B, \sin B )$$ so that
$$\overrightarrow{a}\cdot\overrightarrow{b} = r s \cos A \cos B + r s\sin A \sin B = r s \cos(A-B)$$
I think Augustin was answering a much more general question. In two dimensions, where we have "cosine" already defined, from trigonometry, one can show that the "dot product", defined in some other way, is equal to the lengths of the two vectors times the cosine of the angle between them.
But in higher dimension spaces, $\mathbb{R}^n$, we can use that formula to define the angle between two lines.
I use projection to remember this formula. Let $\vec a$ and $\vec b$ be two vectors. Then, projection of $\vec a$ over $\vec b$ is defined as:
$$proj_{\vec b}(\vec a)=\left( \frac{\vec a \cdot \vec b}{\vec b \cdot \vec b} \right) \vec b$$
Then we can form the following equality:
$$\begin{aligned} ||\vec a||\cos \theta &= ||proj_{\vec b}(\vec a)|| \\ &=\left( \frac{\vec a \cdot \vec b}{\vec b \cdot \vec b} \right) ||\vec b|| \\ &=\frac{(\vec a \cdot \vec b)}{||\vec b||^2} ||\vec b|| \end{aligned} $$
Rearranging both sides, we get
$$\vec a \cdot \vec b=||\vec a|| ||\vec b|| \cos \theta$$
The derivation of the formula for projection is here.
This is a part of the geometric definition, but originally comes from the algebraic definition (see here)
In Euclidean space, you have to define a vector. With that vector, following the algebraic definition
$\vec{a} \cdot \vec{b} = \sum_i a_i b_i$
Imagine at the origin you have two vectors, pointing from $O$ to $A = (a_i)_{i=1}^n$ and to $B = (b_i)_{i=1}^n$. You can pick $n=2,3$ for the sake of imagination.
Now drop a perpendicular line from $B = (b_i)_{i=1}^n$ onto $OA$, crossing $OA$ at a point, called $H$. Let $\theta$ be the angle $(OA,OB)$. After a long calculation, the length of $OH$ is $\frac{\sum_i a_i b_i}{||a||} $.
Observe that $\cos \theta = \frac{|OH|}{|OB|}$, you now arrive at the formula
$\cos \theta = \frac{\vec{a}\cdot \vec{b}} {||a|| \cdot ||b||}$
By the law of cosines, $$\tag 1 \left\Vert{\mathbf a - \mathbf b}\right\Vert^2 = \left\Vert{\mathbf a}\right\Vert^2 + \left\Vert{\mathbf b}\right\Vert^2 - 2 \left\Vert{\mathbf a}\right\Vert \left\Vert{\mathbf b}\right\Vert \cos \theta$$ whereas, using the definition of the dot product, $$\tag 2 \left \| \mathbf a-\mathbf b \right \|^{2}=(\mathbf a-\mathbf b)\cdot (\mathbf a-\mathbf b)=\left \| \mathbf a \right \|^{2}+\left \| \mathbf b \right \|^{2}-2(\mathbf a \cdot \mathbf b)$$ Now combine $(1)$ and $(2)$
Dot product of two vectors is defined as the product of the magnitude of the two vectors together with the cosine of the angle between the two vectors. Mathematically,
a.b=|a||b|cos§
