Consider the Dieudonné measure on $[0, \omega_1)$ which is defined on the Borel $\sigma$-algebra of $[0, \omega_1)$ by $\mu(A) = 1$ if $A$ contains a closed and unbounded set and $\mu(A) = 0$ otherwise. (The Borel $\sigma$-algebra is the collection of all subsets $A$ of $[0, \omega_1)$ such that either $A$ or its complement $A^c$ contains a closed and unbounded set.) It is known that $\mu$ is purely atomic, $\mu$ has empty support and $\mu$ vanishes on all singletons (in fact $\mu$ vanishes on all compact subsets of $[0, \omega_1)$ - this turns $\mu$ into a standard example of a non-regular Borel measure). Hence, singletons are not atoms.
A natural question would be: Is there an explicit characterization of all the atoms of $\mu$?
I think a possible issue is that if $C \subseteq [0, \omega_1)$ is a closed unbounded set then $\mu(C) > 0$ and if $C$ contains a further closed unbounded set $C' \subsetneq C$ then $\mu(C') > 0$. So this reduction process can be performed ad infinitum (but for how long?). For instance, if $C$ is the set of all countable limit ordinals (which is closed and unbounded) then removing the first countable ordinal $\omega_0$ from $C$ gives a set $C'$ that is also an unbounded closed set in $[0, \omega_1)$.