The dual numbers are useful both in physics and mathematics, including developing the theory of Lie algebras. In fields like analysis and differential geometry, the dual numbers are of limited utility because they don't have a number of properties possessed by more advanced number systems such as the hyperreals.
The dual numbers cannot be included in the hyperreals because of the peculiar properties of the dual numbers that make them useful in physics, e.g., $\epsilon^2=0$ which cannot be satisfied in the hyperreals since the latter form a field.
When Riemann originally developed a field today called Riemannian geometry, he defined a metric as a weighted sum of quadratic expressions in the differentials $dx^i$ where each $dx^i$ is an infinitesimal increment of the coordinate $x^i$.
The expressions of the form $dx^i$ were eventually reinterpreted in an infinitesimal-frei manner, as sections of the cotangent bundle on the manifold. This is a more advanced topic than infinitesimal calculus.