We're usually taught the oversimplification that you "can't" divide by zero. This isn't true. Just watch me: $\frac 1 0$. In order to get an accurate statement out of this idea, we need to be more precise with what we mean by "you can't do it".
One possible meaning is that if you know $ab=ac$, then you can conclude $b=c$, unless $a=0$, since for example $0\cdot 1=0\cdot 2$. This isn't really about dividing, it's about what deductions can be made about multiplying.
In your case however, the relevant meaning is that dividing by zero is undefined. In other words, when you define $Z=X/Y$, where $Y$ can be zero with non-zero probability, you actually haven't defined $Z$. More specifically, since a random variable is formally a function on a set (the probability space), you need to define $Z(\omega)$ at each $\omega$. You haven't done that, since I have no idea what $Z(\omega)$ should be for values of $\omega$ for which $Y(\omega)=0$. It's as if you defined a function on $\mathbb N$ by saying "$f(n)$ is the position of the fourth $7$ in the decimal expansion of $n$". I have no idea how you want to define $f(372)$.
So what you need to do here is define your random variable more precisely, and then see if your argument applies to the resulting object.
Let's see how we can do this with your problem. I'll write $\hat{p_n}$ for your $\hat{p^*}$ to make it clear it's a sequence. You have the tentative statement
$$Z_n:=\sqrt n \frac {\hat{p_n} - p}{\sqrt{\hat{p_n}(1-\hat{p_n})}}\to \mathcal N(0,1)$$
Of course you expect this to happen because $\hat{p_n}$ has the law of the average of $n$ independent Bernoulli trials, so without the denominator you would have something that converges in law to $X\sim \mathcal N(0, p(1-p))$, and the denominator in $Z_n$ converges in law to $\sqrt{p(1-p)}$, and $X$ divided by that has the law $\mathcal N(0, 1)$, so by Slutsky's theorem you should be away.
As I said, our task is to redefine $Z_n$ more precisely (or indeed define it at all) so that this argument works. Can we do this? We could do something like say "when the denominator is zero, $Z_n=0$". The worry is that then Slutsky's theorem might not apply anymore. Well, let's look at what Slutsky's theorem says exactly:
$X_n/Y_n\to X/c$ when $X_n\to X$, $Y_n\to c$, as long as $c\neq 0$
Here $c$ is definitely non-zero (except when $p=0$ or $1$), so that's fine. Notice that the attempt at patching up the definition of $Z_n$ I mentioned in the last paragraph doesn't work, since then $Z_n$ is no longer of the form $X_n/Y_n$, it's of the form "$X_n/Y_n$ some of the time, but $0$ the rest of the time". Instead, we need to go one level deeper and redefine the denominator itself so it's never zero. Let:
$$\hat{\sigma_n}=\begin{cases}
\sqrt{\hat{p_n}(1-\hat{p_n})} &\text{ when } \hat{p_n}\neq 0, 1 \\
1 &\text{ otherwise}
\end{cases}$$
Note that $1$ could be replaced by any non-zero number. Now we can define $Z_n=\frac{\sqrt n (\hat{p_n} - p)} {\hat{\sigma_n}}$ and see if Slutsky's theorem applies. The numerator still converges to $\mathcal N(0,1)$, and I'm sure you can show that $\hat{\sigma_n}$ converges in law to $\sqrt{p(1-p)}$.
Although it may seem too good to be true that we were able to get out of a mathematical quandary by arbitrarily defining away the thing that was bothering us, the point is that that thing becomes rarer and rarer as $n\to\infty$, so our ad hoc duct-tape solution becomes less and less visible and vanishes in the limit.
