For the case where we do not require $\gcd(r,t) = 1$, and for $\sum_{t = 1}^N [\sqrt{rt} \in \mathbb{Z}]$ it is useful to know that every positive integer $n$ can be written in a unique way as $n = a\cdot b^2$ with a squarefree $a$. Indeed, by the fundamental theorem of arithmetic $a$ must be the product of the primes whose exponent in the factorisation of $n$ are odd. Call $a$ the squarefree part of $n$, write it $\operatorname{sf}(n)$. Then it is easy to see that for positive integers $m,n$ we have $\sqrt{mn} \in \mathbb{Z}$, i.e. $mn$ is a square, if and only if $\operatorname{sf}(m) = \operatorname{sf}(n)$.
Also useful is the fact that $\lvert \mu(n)\rvert$ is the characteristic function of the squarefree positive integers, and that the number of squarefree integers not exceeding $x$ is
$$Q(x) = \sum_{n \leqslant x} \lvert \mu(n)\rvert = \frac{6}{\pi^2}x + O(\sqrt{x})\,.$$
(Using the prime number theorem one can show that the remainder term is $o(\sqrt{x})$, but that doesn't help us here.)
For $\sum_{t = 1}^N [\sqrt{rt} \in \mathbb{Z}]$ the matter is easy. Let $a = \operatorname{sf}(r)$. Then $\sqrt{rt} \in \mathbb{Z} \iff \operatorname{sf}(t) = a \iff t = a\cdot b^2$ for some positive integer $b$. And
$$t \leqslant N \iff b^2 \leqslant \frac{N}{a} \iff b \leqslant \sqrt{\frac{N}{a}}\,,$$
i.e.
$$\sum_{t = 1}^N [\sqrt{rt} \in \mathbb{Z}] = \biggl\lfloor \sqrt{\frac{N}{\operatorname{sf}(r)}}\biggr\rfloor\,.$$
This is a nice exact formula, but it requires knowing the prime factorisation of $r$ almost completely, which is generally a hard problem for large $r$. (Not completely; if one knows that the unfactored part has at most two prime factors and is not a square, then one knows that it is squarefree without having to know whether it's a prime or a semiprime, and in the latter case which primes divide it.)
For the quadratic forms without the uniqueness condition we have to calculate a little more. It's a relatively simple calculation to obtain the leading term if we sum over the common squarefree part of $r$ and $t$. We need to know how many $r \leqslant N$ have a given squarefree part $a \leqslant N$. We just computed that above, the number is $\lfloor \sqrt{N/a}\rfloor$. By symmetry, the number of $t$ with squarefree part $a$ is the same, and since we can choose the square parts of $r$ and $t$ independently we have
$$\sum_{r = 1}^N\sum_{t = 1}^N [\sqrt{rt} \in \mathbb{Z}]
= \sum_{a = 1}^N \lvert\mu(a)\rvert\cdot \biggl\lfloor \sqrt{\frac{N}{a}}\biggr\rfloor^2\,.$$
Now for every real $x$ we have $\lfloor x\rfloor^2 = (x - \{x\})^2 = x^2 - 2x\{x\} + \{x\}^2$, where $\{x\} = x - \lfloor x\rfloor$ is the fractional part of $x$, so
$$\sum_{r = 1}^N\sum_{t = 1}^N [\sqrt{rt} \in \mathbb{Z}]
= \sum_{a = 1}^N \lvert\mu(a)\rvert\cdot \frac{N}{a} - 2\sum_{a = 1}^N \lvert\mu(a)\rvert\sqrt{\frac{N}{a}}\biggl\lbrace\sqrt{\frac{N}{a}}\biggr\rbrace + \sum_{a = 1}^N \lvert \mu(a)\rvert \biggl\lbrace\sqrt{\frac{N}{a}}\biggr\rbrace^2\,.$$
The first of the sums on the right can easily be evaluated with satisfactory precision using summation by parts:
\begin{align}
\sum_{a = 1}^N \frac{\lvert \mu(a)\rvert}{a} &= \frac{Q(N)}{N} + \int_1^N \frac{Q(u)}{u^2}\,du \\
&= \frac{\frac{6}{\pi^2}N + O(\sqrt{N})}{N} + \int_1^N \frac{\frac{6}{\pi^2}u + O(\sqrt{u})}{u^2}\,du \\
&= \frac{6}{\pi^2} + O(N^{-1/2}) + \frac{6}{\pi^2}\int_1^N\frac{du}{u} + \int_1^N O(u^{-3/2})\,du \\
&= \frac{6}{\pi^2}\log N + \frac{6}{\pi^2} + \int_1^{\infty} O(u^{-3/2})\,du + O(N^{-1/2}) - \int_{N}^{\infty} O(u^{-3/2})\,du \\
&= \frac{6}{\pi^2}\log N + K + O(N^{-1/2})\,.
\end{align}
Reintroducing the factor $N$ that we ignored for the computation, the first sum is $\frac{6}{\pi^2}N\log N + KN + O(\sqrt{N})$ with an (as yet) unknown constant $K$.
The value of the constant $K$ can be determined by other means, but unfortunately, as it turns out, that won't help us.
We can brutally estimate the second sum using $0 \leqslant \{x\} < 1$:
\begin{align}
\sum_{a = 1}^N \lvert\mu(a)\rvert\sqrt{\frac{N}{a}}\biggl\lbrace\sqrt{\frac{N}{a}}\biggr\rbrace
&\leqslant \sqrt{N} \sum_{a = 1}^N \frac{\lvert \mu(a)\rvert}{\sqrt{a}} \\
&\leqslant \sqrt{N}\sum_{a = 1}^N \frac{1}{\sqrt{a}} \\
&\leqslant 2N\,,
\end{align}
so this is an $O(N)$ term. In fact, with more work one can find that this sum is $C\cdot N + O(N^{\alpha})$ for an $\alpha < 1$ (off the top of my head I'm not sure which $\alpha$, I think $3/4$, or maybe $O(N^{3/4}\log N)$, so $3/4 + \varepsilon$). But again, that isn't worth the effort, because for the third sum I don't know how to do better than
$$\sum_{a = 1}^N \lvert \mu(a)\rvert \biggl\lbrace\sqrt{\frac{N}{a}}\biggr\rbrace^2 \leqslant \sum_{a = 1}^N \lvert \mu(a)\rvert = O(N)$$
(well, we can find an explicit constant factor easily, but it remains an $O(N)$).
Thus altogether
$$\sum_{r = 1}^N\sum_{t = 1}^N [\sqrt{rt} \in \mathbb{Z}] = \frac{6}{\pi^2}N\log N + O(N)\,.$$
So we have found the principal asymptotics, but with an unpleasantly large remainder term.
A different way to evaluate the sum gets us more, also the linear term and an $O(N^{2/3}\log N)$ remainder term. Thus also the third sum $\sum \lvert \mu(a)\rvert \{\sqrt{N/a}\}^2$ behaves well, and it should be possible to show that directly. I'll look at that when I have time. But now to the different evaluation. Let
$$S(N) := \sum_{r = 1}^N\sum_{t = 1}^N [\sqrt{rt} \in \mathbb{Z}]\,.$$
Then $S(N) - S(N-1)$ is the number of pairs $(r,t)$ such that $rt$ is a square where $r = N$ or $t = N$ (or both). By symmetry
$$S(N) - S(N-1) = 2\sum_{t = 1}^{N} [\sqrt{Nt}\in \mathbb{Z}] - 1\,.$$
($r = t = N$ was counted twice in the sum, so we must subtract $1$.) We found the value of the sum above,
$$\sum_{t = 1}^{N} [\sqrt{Nt}\in \mathbb{Z}] = \sqrt{\frac{N}{\operatorname{sf}(N)}}\,.$$
No $\lfloor\,\cdot\,\rfloor$ needed since $N/\operatorname{sf}(N)$ is a perfect square. Thus
$$S(N) = \sum_{k = 1}^{N} \bigl(S(k) - S(k-1)\bigr) = \sum_{k = 1}^{N} \biggl(2\sqrt{\frac{k}{\operatorname{sf}(k)}} - 1\biggr) = 2\sum_{k = 1}^{N}\sqrt{\frac{k}{\operatorname{sf}(k)}} - N\,.$$
If $k = a\cdot b^2$ with squarefree $a$, then $\sqrt{k/\operatorname{sf}(k)} = b$, and to evaluate the sum we can count how often each possible value $b$ occurs. Every $b \leqslant \sqrt{N}$ occurs $Q(N/b^2)$ times, hence
$$\sum_{k = 1}^{N} \sqrt{\frac{k}{\operatorname{sf}(k)}} = \sum_{b \leqslant \sqrt{N}} b\cdot Q\biggl(\frac{N}{b^2}\biggr)\,. \tag{$\ast$}$$
If we plug $Q(x) = \frac{6}{\pi^2} x + O(x^{\rho})$ into this sum, that leads to an $O(N)$ error term, regardless of $\rho \leqslant 1$. But if we use
$$Q(x) = \sum_{k \leqslant \sqrt{x}} \mu(k)\biggl\lfloor \frac{x}{k^2}\biggr\rfloor$$ we get something that helps us:
$$\sum_{b \leqslant \sqrt{N}} bQ\biggl(\frac{N}{b^2}\biggr) = \sum_{b \leqslant \sqrt{N}} \sum_{k \leqslant \sqrt{N}/b} b\mu(k)\biggl\lfloor\frac{N}{(bk)^2}\biggr\rfloor = \sum_{n \leqslant \sqrt{N}} \varphi(n)\biggl\lfloor \frac{N}{n^2}\biggr\rfloor = \sum_{n^2m \leqslant N} \varphi(n)\mathbb{1}(m)\,.$$
This sum can be handled with the hyperbola method (yes, the graph of $x \mapsto c/x^2$ isn't really a hyperbola, but let's not be too pedantic) to yield the above error term. Applying the hyperbola method to $(\ast)$ also gives an $O(N^{1-\delta})$ error term, but if I haven't miscalculated there's a hard lower bound of $N^{7/10}$ because $Q(x) - \frac{6}{\pi^2} \in \Omega_{\pm}(x^{1/4})$ (Liu's $\rho = \frac{11}{35} + \varepsilon$ gives $1 - \delta = \frac{59}{83} + \varepsilon$, while the elementary $\rho = 1/2$ gives $3/4$), so we gain something by the rewriting.
To evaluate the last sum, we need the asymptotics of some sums involving Euler's totient function. The first,
$$\sum_{n \leqslant y} \varphi(n) = \frac{3}{\pi^2}y^2 + O(y\log y)$$
is known well enough. The second is
\begin{align}
\sum_{n \leqslant y} \frac{\varphi(n)}{n^2}
&= \sum_{k\cdot m \leqslant y} \frac{\mu(k)m}{(km)^2} \\
&= \sum_{k \leqslant y} \frac{\mu(k)}{k^2}\sum_{m \leqslant y/k} \frac{1}{m} \\
&= \sum_{k \leqslant y} \frac{\mu(k)}{k^2}\bigl(\gamma + \log y - \log k + O(k/y)\bigr) \\
&= (\gamma + \log y)\biggl(\frac{6}{\pi^2} + O(y^{-1})\biggr) + \sum_{k \leqslant y} \frac{\mu(k)\cdot(-\log k)}{k^2} + O\biggl(\frac{\log y}{y}\biggr) \\
&= \frac{6}{\pi^2}\biggl(\log y + \gamma - \frac{\zeta'(2)}{\zeta(2)}\biggr) + O\biggl(\frac{\log y}{y}\biggr)
\end{align}
since
$$\sum_{k \leqslant y} \frac{\mu(k)(-\log k)}{k^s} = -\frac{\zeta'(s)}{\zeta(s)^2} + O\biggl(\frac{\log y}{y^{\operatorname{Re} s - 1}}\biggr)$$
for $\operatorname{Re} s > 1$.
Then we have
$$\sum_{n^2m\leqslant N} \varphi(n)\mathbb{1}(m) = \sum_{n \leqslant N^{1/3}} \varphi(n)\biggl\lfloor\frac{N}{n^2}\biggr\rfloor + \sum_{m \leqslant N^{1/3}} \sum_{n \leqslant \sqrt{N/m}} \varphi(n) - \bigl\lfloor N^{1/3}\bigr\rfloor \sum_{n \leqslant N^{1/3}} \varphi(n)\,.$$
The first of these sums yields
\begin{align}
\sum_{n \leqslant N^{1/3}} \varphi(n)\biggl(\frac{N}{n^2} + O(1)\biggr)
&= N\sum_{n \leqslant N^{1/3}} \frac{\varphi(n)}{n^2} + O\Biggl(\sum_{n \leqslant N^{1/3}} \varphi(n)\Biggr) \\
&= N\frac{6}{\pi^2}\biggl(\frac{1}{3}\log N + \gamma - \frac{\zeta'(2)}{\zeta(2)} + O\biggl(\frac{\log N}{N^{1/3}}\biggr)\biggr) + O(N^{2/3}) \\
&= \frac{2}{\pi^2}N\log N + \frac{6}{\pi^2}\biggl(\gamma - \frac{\zeta'(2)}{\zeta(2)}\biggr)N + O(N^{2/3}\log N)\,.
\end{align}
The second is
\begin{align}
\sum_{m \leqslant N^{1/3}} \biggl(\frac{3}{\pi^2} \frac{N}{m} + O\biggl(\sqrt{\frac{N}{m}\log \frac{N}{m}}\biggr)
&= \frac{3}{\pi^2}N\biggl(\frac{1}{3}\log N + \gamma + O(N^{-1/3})\biggr) + O\Biggl(\sqrt{N}\log N \sum_{m \leqslant N^{1/3}} \frac{1}{\sqrt{m}}\Biggr) \\
&= \frac{1}{\pi^2}N\log N + \frac{6}{\pi^2}\frac{\gamma}{2} + O(N^{2/3}\log N)
\end{align}
and the third
$$\bigl(N^{1/3} + O(1)\bigr)\biggl(\frac{3}{\pi^2}N^{2/3} + O(N^{1/3}\log N)\biggr)
= \frac{3}{\pi^2}N + O(N^{2/3}\log N)\,.$$
Adding up, we find
$$\sum_{n \leqslant \sqrt{N}} \varphi(n)\biggl\lfloor \frac{N}{n^2}\biggr\rfloor = \frac{3}{\pi^2}N\log N + \frac{6}{\pi^2}\biggl(\frac{3}{2}\gamma - \frac{\zeta'(2)}{\zeta(2)} - \frac{1}{2}\biggr)N + O(N^{2/3}\log N)$$
and therefore
$$S(N) = \frac{6}{\pi^2}N\log N + \frac{6}{\pi^2}\biggl(3\gamma - 2\frac{\zeta'(2)}{\zeta(2)} - 1 - \zeta(2)\biggr)N + O(N^{2/3}\log N)\,.$$
