First answer. This has some problems but now it is fixed.
So you have the result:
\begin{align}\tag{1}
\int^{\infty}_x e^{-t^2}\,dt = \frac{e^{-x^2}}{2x}+o\left(\frac{e^{-x^2}}{x}\right) \ \ \ \text{as} \ \ x\to\infty
\end{align}
In your last step, you had a mistake. It would be:
\begin{align}
\int^b_a e^{-nt^2}\,dt &= \frac{1}{\sqrt[]{n}}\int^{\sqrt[]{n}b}_{\sqrt[]{n}a}e^{-t^2}\,dt\\
& = \frac{1}{\sqrt[]{n}}\left(\int^{\infty}_{\sqrt[]{n}a}e^{-t^2}\,dt - \int^\infty_{\sqrt[]{n}b}e^{-t^2}\,dt \right)\\
\end{align}
Assume $0<a<b$ $(\star)$. First note that $$\frac{e^{-nb^2}}{n}=o\left(\frac{e^{-na^2}}{n}\right)$$ as $n\to\infty$ (we will avoid writing this from now on). So use $(1)$ to get:
\begin{align}\tag{2}
\int^b_a e^{-nt^2}\,dt = \frac{e^{-na^2}}{2na}+o\left(\frac{e^{-na^2}}{n}\right)
\end{align}
For $n$ large enough we can take $n$-th root on both sides of $(2)$ to get:
\begin{align}
\left(\int^b_a e^{-nt^2}\,dt\right)^{1/n}&=\left[\frac{e^{-na^2}}{2na}+o\left(\frac{e^{-na^2}}{n}\right)\right]^{1/n}\\
&=e^{-a^2}\frac{1}{n^{1/n}(2a)^{1/n}}\left[1+o\left(1\right)\right]^{1/n}\\
&\to e^{-a^2}
\end{align}
Where we have used $c_n^{1/n}\to 1$ for $c_n$ strictly positive and bounded away from $0$ and the fact that $\sqrt[n]{n}\to 1$.
$(\star)$: If you allow $a=0$, then something similar can be done which is even easier.
Edit One can also come up with the asymptotics of the integral:
\begin{align}
I_n^n=\int^b_a e^{-nt^2}\,dt
\end{align}
Assume $0<a<b$. Note that $t^2$ is monotonically increasing and using The Laplace Method, we get:
\begin{align}
I_n^n\sim \frac{e^{-na^2}}{2an}
\end{align}
Taking $n$-th root we obtain the result:
\begin{align}
\lim_{n\to\infty} I_n = e^{-a^2}
\end{align}