One property that real numbers and rational numbers share is density; for any $x, y$ with $x \lt y$, there's always a $z$ such that $x\lt z\lt y$ - just take $z=\frac{x+y}{2}$. This explains how, if we know $\sqrt{2}$ exists, we can conclude that there are no numbers next to it. But as pointed out in the comments, the argument here is somewhat circular; to run the simple density argument, we need to have a square root of $2$ in the first place.
Suppose instead that the only numbers we knew were rational; how could we conclude that for any number $y$ with $y^2\gt 2$, we can find a $z$ with $z\lt y$ but $z^2\gt 2$ still? One natural approach would be to run the averaging argument on the squares of $y$ and $z$, that is to choose $z$ such that $z^2 = \frac{2+y^2}2$; but if we can't find a square root of $2$ there's no reason to believe that we can find a square root of $\frac{2+y^2}{2}$.
Instead, let's take a different tack; if we know that $y^2$ is close to 2, then we know that $\frac2y$ is pretty close to $y$; what's more, we know that its square $\frac4{y^2}$ is on the 'other side' of $2$; $y^2\gt 2$, so $\frac4{y^2}\lt\frac42=2$. So let's try averaging $y$ and $\frac2y$; take $z=\frac12\left(y+\frac2y\right) = \frac{y^2+2}{2y}$. Since $y^2\gt 2$ then $\frac2y\lt y$, so their average $z$ must be $\lt y$; but $z^2 = \frac{(y^2+2)^2}{4y^2} = \frac{y^4+4y^2+4}{4y^2}$, and so $z^2-2 = \frac{y^4-4y^2+4}{4y^2} = \frac{(y^2-2)^2}{4y^2}\gt 0$.
This construction gives us a rational number $z$ (that is, a number that we know 'exists') that's less than y but whose square is still greater than $2$, and this $z$ disproves y's claim to be a least upper bound for the set $A$ ($=\{x: x^2\lt 2\}$).