The question asks:
a) What is kernel space of linear map defined by $$ M = \begin{bmatrix} 1 & 2 & 3 \\ 2 & 4 & 6 \\ 3 & 6 & 9 \\ \end{bmatrix} $$
b) Give orthonormal basis for map's kernel from a)
My attempt of a)
I attempted to find the kernel space, but I am not sure if I am right, furthermore, I don't know how to approach b). Thank you for any help
You've got the right idea. Put another way, the kernel of the transformation has $\bigl\{[-2,1,0]^\dagger,[-3,0,1]^\dagger\bigr\}$ as a basis. Use Gram-Schmidt method to determine an orthonormal basis for the kernel.
So since ${\rm ker}M=\left\{\begin{bmatrix}-2y-3z\\ y\\ z\end{bmatrix}\in \mathbb{R}^{3}: y,z\in \mathbb{R}\right\}$ hence a basis for ${\rm ker}M$ is given by $\beta=\left\{\begin{bmatrix}-2\\1\\0\end{bmatrix},\begin{bmatrix}-3\\0\\1\end{bmatrix}\right\}$. Define $v_{1}=\begin{bmatrix}-2\\1\\0\end{bmatrix}$ and $v_2=\begin{bmatrix}-3\\0\\1\end{bmatrix}$.
An orthonormal basis for ${\rm ker}M$ is given by $\beta^{*}=\{u_1,u_2\}$ where $u_1$ and $u_2$ is obtained by Gram-Schmidt algorithm:
$\displaystyle u_1 =\frac{v_1}{||v_1||}$.
$\displaystyle u_2=\frac{v_{2}-\langle v_{2},u_{1}\rangle u_{1}}{||v_{2}-\langle v_{2},u_{1}\rangle u_{1}||}$.
Where $||\cdot||$ the norm induced by the inner product $\langle \cdot, \cdot \rangle$. If you haven't seen general inner products, then you may be working with the usual inner product of $\mathbb{R}^n$ which is just the dot product between vectors.
Using the dot product as the inner product we get $$\beta^{*}=\left\{\begin{bmatrix} -\frac{\sqrt{35}}{35}\\ \frac{\sqrt{35}}{7}\\-\frac{3\sqrt{35}}{35} \end{bmatrix}, \begin{bmatrix} -\frac{3\sqrt{10}}{10}\\0\\ \frac{\sqrt{10}}{10}\end{bmatrix} \right\}.$$
Recall the dot product in $\mathbb{R}^{3}$ is given by $$\left\langle \begin{bmatrix} a\\ b\\ c\end{bmatrix},\begin{bmatrix}x\\ y\\ z\end{bmatrix} \right\rangle=ax+by+cz$$ and the norm induced by the dot product is given by $$\left|\left|\begin{bmatrix} a\\b\\c\end{bmatrix}\right|\right|=\sqrt{a^{2}+b^2 + c^2 }.$$
