A real inner product on a real vector space $V$ is a symmetric positive definite bilinear form $V \times V \to \mathbb{R}$.
You already accept that $|x| = \sqrt{\langle x, x\rangle}$ captures the notion of length.
Before getting to angles in general we should discuss right angles (orthogonality).
It should be geometrically reasonable that $v \perp w$ if and only if $|v - \alpha w|$ has $\alpha = 0$ as the unique minimizer.
$$
|v - \alpha w|^2 = |v|^2 + \alpha^2 |w|^2 +2\alpha\langle v, w\rangle
$$
This is a quadratic in $\alpha$ which achieves a minimum value at $\alpha = -\frac{\langle v, w \rangle}{|w|^2}$. So it should be geometrically reasonable to define $v \perp w$ if and only if $\langle v, w\rangle = 0$.
Now that we have defined perpendicularity, it is reasonable to define angles through right triangle trigonometry.
The same calculation shows that $v$, $\alpha w$ and $v - \alpha w$ form a right triangle when $\alpha = -\frac{\langle v, w \rangle}{|w|^2}$ with $v$ as the hypotenuse. Assume for the moment that $\langle v, w \rangle > 0$ so that I can ignore sign issues. So it would be reasonable to define
$$\cos(\theta) = \frac{|\alpha w|}{|v|} = \frac{\langle v, w\rangle}{|v||w|}$$
The Cauchy-Schwarz inequality justifies that this equation has a solution (the RHS is between $-1$ and $1$).