The given system is an algebraic symmetric system in the unknowns $a,b$. (I.e. the polynomials involved in the two equations are symmetric in the unknowns $a,b$.) In such cases, the standard method to solve is to use the substitution $s=a+b$ (for the sum), and $p=ab$ (for the product). The given system becomes then an equivalent algebraic system in $s,p$, and the degree drops by chance:
$$
\left\{
\begin{aligned}
s + p &= 2+3\sqrt 2\ ,\\
s^2 - 2p &= 6\ .
\end{aligned}
\right.
$$
We eliminate $p$ among the two equations (add two times the first to the second to eliminate $p$), and obtain an equation in $s$:
$$
s^2 + 2s - 10-6\sqrt 2=0\ .
$$
Its half-discriminant is $1^2+(10+6\sqrt 2)=11+6\sqrt2$. Is this a "nice square"? For instance in the ring $\Bbb Z[\sqrt 2]$.
If yes, $11+6\sqrt 2=(m+n\sqrt 2)^2$.
We obtain passing to the norm on both sides (where the norm of $m+n\sqrt2$ is $(m+n\sqrt2)(m-n\sqrt 2)=m^2-n^2\in\Bbb Z$) the relation:
$49=11^2-2\cdot 6^2=(m^2-2n^2)^2$. So we search (up to a sign) for $m,n\in\Bbb Z$ with $m^2-2n^2=7$ and $11+ 6\sqrt 2 =(m+n\sqrt2)^2=(m^2+2n^2)+(2mn)\sqrt 2$. We have luck, the equations $m^2+2n^2=11$, $m^2-2n^2=7$ give us $m^2=9$, $n^2=1$, and we need also $2mn=6$. Yes, the luck shows we have a discriminant equal to $11+6\sqrt 2=(3+\sqrt 2)^2$. So $s = -1\pm (3+\sqrt 2)$.
The two possible values for the sum $s$ are: $2+\sqrt 2$ and $-4-\sqrt 2$.
The corresponding values for $p = 2+3\sqrt 2-s$ are respectively: $2\sqrt 2$ and $6+4\sqrt 2$. The last value has to be rejected because of $a,b\in\Bbb R$, which leads to $0\le (a- b)^2=a^2+b^2- 2ab=6-2ab=6-2p$, which is invalid for p=6+4\sqrt 2$.
So the only chance is: $a+b=s=2+\sqrt 2$ and $ab=p=2\sqrt 2$. Then (by Vieta) $a,b$ are the roots of the equation in a (new) variable $T$:
$$T^2 - sT+p=0\ ,\qquad\text{ i.e. }\qquad T^2-(2+\sqrt 2)T+2\sqrt 2=0\ .
$$
We either "see the roots", they are $2$ and $\sqrt 2$, or we compute them.
Conclusion: The given symmetric system has two solutions: $(a,b)=(2,\sqrt 2)$, and $(a,b)=(\sqrt2, 2)$. We cannot "determine" $a,b$, but the set $\{a,b\}$ is determined.
Second solution: (Can be given if we know the solution $a=2$. If not, this is not really a good path.)
The first equation, $a+ab+b=2+3\sqrt 2$ is linear in $b$, so we can extract $b$ linearly from it and plug in into the second equation. This implies successively:
$$
\begin{aligned}
b &=\frac{2+3\sqrt 2-a}{a+1}\ ,\\
0 &= a^2 -6 + \frac{(2+3\sqrt 2-a)^2}{(a+1)^2}\ ,\\
0 &= a^2(a+1)^2 - 6(a+1)^2 + (a-2-3\sqrt 2)^2\ ,\\
0 &= \underbrace{a^4 + 2a^3 - 4a^2 -(16+6\sqrt 2)a + (16+12\sqrt 2)}_{P(a)}\ .
\end{aligned}
$$
We may plot the polynomial (function) $P$ or compute its values in some points trying to
identify some intervals with roots. When computing its values in $0,1,2,3,\dots$ we have the chance to see that $a=2$ is a root, $P(2)=0$. This leads to $b=\sqrt 2$.
But the given system is symmetric, so together with $(a,b)$ then $(b,a)$ is also a solution. So $a=\sqrt 2$ must be a root, and we easily check this.
Then $b=2$ is the corresponding value.
We have two $a$-roots. The other roots are obtained after a division of $P$
by $(a-2)(a-\sqrt 2)$. Let us denote by $u$ the value $u=\sqrt 2$ for easy typing. Then the Horner scheme for this division is:
$$
\begin{array}{r|ccccc}
& 1 & 2 & -4 & -16-6u & 16 + 12u\\\hline
2 & 1 & 4 & 4 & -8-6u & \boxed 0\\\hline
u & 1 & 4+u & 6+4u & \boxed 0\\\hline
\end{array}
$$
so the remained roots are roots of the polynomial
with the coefficients from the last line:
$a^2+(4+u)a+(6+4u)$. But its discriminant is negative, $(4+u)^2-4(6+4u)=4u-6<0$, so there are no further real solutions.
