Building up on Lee Mosher's comments...
Theorem. Let $G$ be an abelian group, written additively. Then the set $\mathscr{H}$ consists precisely of all cosets of finitely generated subgroups of $G$.
Proof. Note that if $\mathscr{A}$ and $\mathscr{B}$ are two collections of subsets of $G$ that contain all singletons such that if they include $H$ then they include $H\bmod g$ for all $g\in G$, then so does $\mathscr{A}\cap\mathscr{B}$. Therefore, the smallest such collection will be the intersection of all collections satisfying these properties.
First, we show that $\mathscr{H}$ has as elements all cyclic subgroups of $G$. We have $\{0\}\in\mathscr{H}$, by assumption. If $g\in G$, then $\mathscr{H}$ contains
$$\{0\}\bmod g = \{ ng\mid n\in\mathbb{Z}\} = \langle g\rangle.$$
So $\mathscr{H}$ contains all cyclic subgroups.
Next we show inductively that $\mathscr{H}$ contains all finitely generated subgroups. Assume that $\mathscr{H}$ contains all subgroups generated by at most $k\geq 1$ elements, and let $K=\langle g_1,\ldots,g_k,g_{k+1}\rangle$ be a subgroup generated by $k+1$ elements. Then $H=\langle g_1,\ldots,g_k\rangle\in\mathscr{H}$, and $K=H+\langle g_{k+1}\rangle$; since $H\bmod g\in\mathscr{H}$, and
$$H\bmod g_{k+1} = \{h+ng\mid n\in\mathbb{Z},h\in H\} = H+\langle g_{k+1}\rangle = K,$$
we conclude that $K\in\mathscr{H}$. Thus, $\mathscr{H}$ contains all finitely generated subgroups.
Next we show that $\mathscr{H}$ contains any coset of a finitely generated subgroup of $G$. Let $K=\langle g_1,\ldots,g_n\rangle$ be a finitely generated subgroup of $G$, and let $x\in G$. Then $\mathscr{H}$ contains $\{h\}\bmod g_1 = h+\langle g_1\rangle$. Therefore, it also contains
$$\begin{align*}
(h+\langle g_1\rangle) \bmod g_2 &= (h+\langle g_1\rangle)+\langle g_2\rangle\\
&= \{h + ng_1 + mg_2\mid n,m\in\mathbb{Z}\} = h+ \langle g_1,g_2\rangle\\
(h+\langle g_1,g_2\rangle) \bmod g_3 &= h+\langle g_1,g_2,g_3\rangle\\
&\vdots\\
(h+\langle g_1,\ldots,g_{n-1}\rangle\bmod g_n &= h+\langle g_1,\ldots,g_n\rangle = h+K.
\end{align*}$$
Thus, $\mathscr{H}$ contains all cosets of finitely generated subgroups of $G$.
Finally, we show that the collection $\mathscr{S}$ of all cosets of finitely generated subgroups of $G$ is a collection that includes all singletons and is closed under the "mod" operation you have defined. Indeed, the singleton $\{g\}$ is the coset of $g$ relative to the subgroup $\{e\}$, so $\mathscr{S}$ contains all singletons. Now let $h+K$ be a coset of a finitely generated subgroup $K$, and let $g\in G$. Then
$$(h+K)\bmod g = \{ h+k+ng\mid k\in K,n\in\mathbb{Z}\} = h+\langle K,g\rangle,$$
and since $K$ is finitely generated so is $\langle K,g\rangle$; therefore, $(h+K)\bmod g\in\mathscr{S}$. Thus, $\mathscr{S}\subseteq \mathscr{H}\subseteq\mathscr{S}$, proving the desired equality. $\Box$
The situation is more complicated for nonabelian groups $G$. It is clear that $\mathscr{H}$ will contain all left cosets of cyclic subgroups of $G$, since (switching to multiplicative notation) it contains $\{x\}$ for all $x\in G$, and $\{x\}\bmod g = \{xg^n\mid n\in\mathbb{Z}\}=x\langle g\rangle$. But it will contains sets that are not cosets in general, as well. For example, consider the Heisenberg group of order $27$,
$$\langle x,y,z\mid x^3=y^3=z^3=1, yx=xyz, xz=zx, yz=zy\rangle$$
which can be realized as the multiplicative group of all unitriangular matrices over $\mathbb{Z}/3\mathbb{Z}$:
$$\left(\begin{array}{ccc}
1 & a & c\\
0 & 1 & b\\
0 & 0 & 1
\end{array}\right).$$
Then $\mathscr{H}$ contains the singletons (27 of them); also the subgroup $\langle x\rangle = \{1,x,x^2\}$. Then it also contains
$$\langle x\rangle\bmod y = \{1, y, y^2, x, xy, xy^2, x^2, x^2y, x^2y^2\} = \langle y\rangle \cup x\langle y \rangle \cup x^2\langle y\rangle,$$
the union of three cosets of $\langle y\rangle$. However, this is not a coset of a subgroup of $G$, since it is not a subgroup: it contains $y$ and $x$, but does not contain $yx = xyz$; and if it were a coset, it would have to be a subgroup since it contains $1$. It will then get more complicated to describe further iterations, so I agree that there might not be much that can be said for arbitrary nonabelian $G$.
\;either side of your $\rm\LaTeX$ code. The spacing is just odd. – Shaun May 23 '23 at 19:43