There are multiple approaches to each of these integrals (which are covered on this page).
For the first integral, I cover it using the Beta function here.
For the second integral, two methods that jump out are (1) Differential Under the Curve (2) Laplace Transforms. Let us consider each
For (1):
We first observe that:
\begin{equation}
I(a,0) = \int_0^\infty e^{-ax^2}\:dx = \frac{1}{2}\sqrt{\frac{\pi}{a}}
\end{equation}
We proceed with $I(a,b)$ by employing Leibniz's Integral rule and taking the derivative with respect to $b$:
\begin{align}
\frac{\partial I}{\partial b} = \int_0^\infty e^{-ax^2} \cdot -x\sin(bx)\:dx = \int_0^\infty -xe^{-ax^2} \sin(bx)\:dx \nonumber
\end{align}
Here we employ integration by parts:
\begin{align}
u(x) &= \sin(bx) & v'(x)&= -xe^{-ax^2} \nonumber \\
u'(x) &= b\cos(bx) & v(x) &= \frac{1}{2a}e^{-ax^2} \nonumber
\end{align}
Thus,
\begin{align}
\frac{\partial I}{\partial b} &= \left[\sin(bx) \cdot \frac{1}{2a}e^{-ax^2} \right]_0^\infty - \int_0^\infty \frac{1}{2a}e^{-ax^2} \cdot b\cos(bx)\:dx \nonumber \\
&= -\frac{b}{2a}\int_0^\infty e^{-ax^2}\cos(bx)\:dx = -\frac{b}{2a}I(a,b)
\end{align}
Thus we have formed a differential equation with respect to $b$:
\begin{equation}
\frac{\partial I}{\partial b} + \frac{b}{2a}I(a,b) = 0
\end{equation}
Here we employ the Integrating Factor:
\begin{equation}
\frac{\partial }{\partial b} \left[e^{\frac{b^2}{4a}}I(a,b) \right] = 0 \Longrightarrow e^{\frac{b^2}{4a}}I(a,b) = C \Longrightarrow I(a,b) = Ce^{-\frac{b^2}{4a}}
\end{equation}
Where $C$ is the constant of integration. We resolve $C$ using the condition $I(a,0)$ as given before:
\begin{equation}
I(a,0) = \frac{1}{2}\sqrt{\frac{\pi}{a}} = C \cdot e^{-\frac{0^2}{4a}} \Longrightarrow C = \frac{1}{2}\sqrt{\frac{\pi}{a}}
\end{equation}
Thus, we arrive at:
\begin{equation}
I(a,b) = \frac{1}{2}\sqrt{\frac{\pi}{a}} e^{-\frac{b^2}{4a}}\nonumber
\end{equation}
For (2):
By using Fubini's Theorem, we first take the Laplace Transform with respect to $a$:
\begin{equation}
\mathscr{L}_{a \rightarrow s}\left[I(a,b)\right] = \int_0^\infty \mathscr{L}_{a \rightarrow s}\left[e^{-ax^2} \right] \cos(bx)\:dx = \int_0^\infty \frac{\cos(bx)}{s + x^2}\:dx
\end{equation}
We again employ Fubini's Theorem by taking the Laplace transform with respect to $b$:
\begin{equation}
\mathscr{L}_{b \rightarrow w}\left[\mathscr{L}_{a \rightarrow s}\left[I(a,b)\right]\right] = \int_0^\infty \frac{\mathscr{L}_{b \rightarrow w}\left[\cos(bx)\right]}{s + x^2}\:dx = \int_0^\infty \frac{w}{w^2 + x^2}\cdot \frac{1}{s + x^2}\:dx
\end{equation}
We proceed by forming a Partial Fraction Decomposition:
\begin{align}
&\mathscr{L}_{b \rightarrow w}\left[\mathscr{L}_{a \rightarrow s}\left[I(a,b)\right]\right] = \frac{w}{w^2 - s}\int_0^\infty\left(\frac{1}{s + x^2} - \frac{1}{w^2 + x^2} \right) \:dx \nonumber \\
&\quad= \frac{w}{w^2 - s}\left[ \frac{1}{\sqrt{s}}\arctan\left(\frac{x}{\sqrt{s}}\ \right) -\frac{1}{w}\arctan\left(\frac{x}{w} \right)\right]_0^\infty = \frac{w}{w^2 - s}\left[ \frac{1}{\sqrt{s}}\frac{\pi}{2} -\frac{1}{w}\frac{\pi}{2}\right] \nonumber \\
&= \frac{w}{w^2 - s}\left[\frac{w - \sqrt{s}}{w\sqrt{s}}\right]\frac{\pi}{2} = \frac{\pi}{2\sqrt{s}\left(w + \sqrt{s}\right)}
\end{align}
We now take the Inverse Laplace Transform with respect to $w$:
\begin{equation}
\mathscr{L}_{a \rightarrow s}\left[I(a,b)\right] = \mathscr{L}_{w \rightarrow b}^{-1}\left[ \frac{\pi}{2\sqrt{s}\left(w + \sqrt{s}\right)}\right] = \frac{\pi}{2\sqrt{s}}e^{-b\sqrt{s}}
\end{equation}
We now take the Inverse Laplace Transform with respect to $s$:
\begin{equation}
I(a,b) = \mathscr{L}_{s \rightarrow a}^{-1}\left[ \frac{\pi}{2\sqrt{s}}e^{-b\sqrt{s}} \right] = \frac{\pi}{2} \cdot \frac{1}{\sqrt{a\pi}}e^{-\frac{b^2}{4a}}= \frac{\sqrt{\pi}}{2\sqrt{a}}e^{-\frac{b^2}{4a}}
\end{equation}
