I need to find the matrix representing reflection in the line $y=2x-2$.
I wanted to change variables to $X=x$ and $Y=y+2$ and then proceed as normal to find the reflection matrix in $Y=2X$. But then I didn't know how to change the variables back to $x, y$ in the matrix. I got the matrix to be: \begin{pmatrix} 0 & 2.4 \\ 1 & -0.2 \\ \end{pmatrix} But I think this is wrong?
Ordinarily, matrices are used for linear maps. Linear maps take $(0,0)$ to itself, so they can't describe this map. There are two ways you could get around this:
(1) You can write the map in the form $$\begin{bmatrix} x\\y \end{bmatrix} \longrightarrow \begin{bmatrix} a&b \\ c&d \end{bmatrix} \begin{bmatrix} x\\y \end{bmatrix} + \begin{bmatrix} e\\f \end{bmatrix}$$ for some constants $a$, $b$, $c$, $d$, $e$, $f$.
I don't want to do a homework problem, so I'll answer a different question: Let's think about rotation of $60^{\circ}$ counterclockwise around the point $(2,3)$. As you suggest, make the change of variables $(X,Y) = (x-2, y-3)$. The matrix of the rotation in $(X,Y)$ coordinates is $\left[ \begin{smallmatrix} 1/2 & -\sqrt{3}/2 \\ \sqrt{3}/2 & 1/2 \\ \end{smallmatrix} \right]$. In $(x,y)$ coordinates, we have $$\begin{bmatrix} x\\y \end{bmatrix} \longrightarrow \begin{bmatrix} 1/2 & - \sqrt{3}/2 \\ \sqrt{3}/2 & 1/2 \\ \end{bmatrix} \begin{bmatrix} x-2 \\ y-3 \end{bmatrix} + \begin{bmatrix} 2\\3 \end{bmatrix} = \begin{bmatrix} 1/2 & - \sqrt{3}/2 \\ \sqrt{3}/2 & 1/2 \\ \end{bmatrix}\begin{bmatrix} x\\y \end{bmatrix} + \begin{bmatrix} 1+3 \sqrt{3}/2 \\ 3/2 - \sqrt{3} \end{bmatrix}.$$
(2) The group of rigid symmetries of the plane can be described with $3 \times 3$ matrices: Describe the symmetry $\vec{x} \mapsto Q \vec{x} + \vec{b}$ by the $3 \times 3$ block matrix $$\left[ \begin{array}{c|c} Q & \vec{b} \\ \hline 0 & 1 \end{array} \right] .$$ This group preserves the set of vectors of the form $(x,y,1)$, and acts on them by rigid motions. So, inside this group, we can use the usual conjugation formula to change coordinates. Repeating my examples of a $60^{\circ}$ rotation around $(2,3)$, we get $$\begin{bmatrix} 1&0&2 \\ 0&1&3 \\ 0&0&1 \\ \end{bmatrix} \begin{bmatrix} 1/2 & - \sqrt{3}/2 & 0 \\ \sqrt{3}/2 & 1/2 & 0 \\ 0&0&1 \end{bmatrix} \begin{bmatrix} 1&0&2 \\ 0&1&3 \\ 0&0&1 \\ \end{bmatrix}^{-1} = \begin{bmatrix} 1/2 & - \sqrt{3}/2 & 1+3 \sqrt{3}/2 \\ \sqrt{3}/2 & 1/2 & 3/2-\sqrt{3} \\ 0&0&1 \end{bmatrix}.$$
Generally speaking:
A reflection can be thought of as a composition of the following transformations:
You can't really model translations using matrices as far as I know and hence you can't directly do this reflection procedure.
On the attempt you've made and how to modify it to get the correct result:
The shifts you used are wrong since in the shifted co-ordinate system the line actually doesn't pass through the origin for the shifts which you have used. Instead, follow this procedure,
(1) use the shift of $Y=y-2$
(2) then use the rotation matrix to transform our 'vector' (*)
(3) switch sign-on $y$ component
(4) undo the rotation of our point using the rotation matrix of the opposite angle
(5) use the inverse transformation of the shift you used on the final point which you got.
Refer this question which I posted where I used complex numbers to achieve the same result
*: The point can be described by vectors by considering the position vector connecting to it from the origin.
The image $(x',y')$ of a point $(x,y)$ in the line $ax+by+c=0$ is given as $$\frac{x'-x}{a}=\frac{y'-y}{b}=-2\frac{ax+by+c}{a^2+b^2}$$ So here in this case we have $$\frac{x'-x}{2}=\frac{y'-y}{-1}=-2\frac{2x-y-2}{5}$$ $$\implies x'=(1/5)x+(2/5)y+4/5, y'=(4/5)x+(3/5)y-(4/5).$$ In the matrix notation $$\begin{bmatrix} x'\\y' \\1 \end{bmatrix}= \begin{bmatrix} 1/5 & 2/5 & 4/5 \\ 4/5 & 3/5 &-4/5\\ 0 & 0& 1 \end{bmatrix}\begin{bmatrix} x\\y \\1 \end{bmatrix}$$
Suppose you have a point $P(X,Y)$
Now, the mirror is a line with direction cosine $(\frac{1}{\sqrt{5}},\frac{2}{\sqrt{5}})$ - hence the normal will have direction - $(\frac{2}{\sqrt 5}, -\frac{1}{\sqrt 5})$
Also, the distance $d$ of the point from the mirror is given as
$$d = \frac{|2X-Y-2|}{\sqrt5}$$
Hence, the reflection is given as
$$P' = \left(X +\frac{4d}{\sqrt 5} , Y - \frac{2d}{\sqrt 5}\right)$$
As you can probably see, this cannot be represented as a matrix multiplication - but also has an extra constant component
