Disintegrating By Parts gave a nice answer already, but here is something useful worth adding.
It is often said that Quantum Mechanics works in a Hilbert space. A Hilbert space is essentially an inner product space where the "vectors" can now be infinite dimensional. Vectors in a Hilbert space can therefore be functions, and an inner product can be defined:
$$\int_{-\infty}^{+\infty} f(x) g(x) dx.$$
Of course, we have to assume here that the vectors, which are functions $f:\mathbb{R} \to \mathbb{R},$ behave nicely enough so that the above inner product above is always defined (they go to zero fast enough; are integrable in the first place, etc.) Also, in Quantum Mechanics, we use complex numbers.
Now, it would be tempting to be able to think of a function $f(x)$ as being the sum of the basis functions which are just defined at a point. Let's call these functions $\Delta_r:$
$$\Delta_r(x) = 0 \text{ for } x \neq r$$
$$\Delta_r(x) = 1 \text{ for } x = r$$
Then we could write
$$f(x) = c_1 \Delta_{r_1}(x) + c_2 \Delta_{r_2}(x) + c_3 \Delta_{r_3}(x) + ...$$
In analogy to finite dimensional vector spaces
$$\vec{v} = c_1 \hat{e}_1 + c_2 \hat{e}_2 + c_3 \hat{e}_3 + ... + c_N \hat{e}_N.$$
or to countably-infinite dimesional vector spaces:
$$\vec{v} = \sum_{n}^{\infty} c_n \hat{e}_n$$
Of course, such a sum does not actually make sense in an uncountably infinite space. We would instead use the integral:
$$f(x) = \int c_r \Delta_r(x) dr.$$
Unfortunately, the $\Delta_r$ we have defined have a "length" of zero by the inner product we have defined, and no matter how you scale them they are always zero. But one of the axioms of an inner product space is if the length of a vector is zero, then that vector must just be the zero vector. So the idea is to define new "functions"
$$\delta(x-r) = \infty \cdot \Delta_{r}(x)$$
where the infinity is of just the right size to make the length of the "vector" $\delta(x-r)$ to be 1 (or $2 \pi)$.
That is the intuition and it should hopefully convince you that the physicists are not just "getting lucky." After all, if you wanted to, instead of looking at the uncountably infinite and continous case, you could just instead look at an extremely large finite system with a countably infinite number of basis vectors. Then, at the end, you could take the limit as the size of the finite system becomes infinite.
But there is still a contradiction here: these $\delta$ functions are not "functions" and so should not be allowed to be in the Hilbert space. The answer to the question of how to make this rigorous then, is that Quantum Mechanics does not actually happen in a Hilbert space, it instead happens in what is called a "rigged Hilbert space."
Historically though, the formalisms and rigorous definitions of rigged Hilbert spaces came after physicists had already been using Dirac-Delta functions. This is not unique in the history of mathematics, calculus was used for many years before a rigorous footing for it was found.