A somewhat general setting for this is valuation theory. If $K$ is a field, then a valuation ring is a subring $\mathcal{O} \subseteq K$ with $a \in \mathcal{O}$ or $a^{-1} \in \mathcal{O}$ for all $a \in K^{\times}$. In particular $\mathcal{O}$ is a local ring with maximal ideal $\mathcal{o}=\mathcal{O} \setminus {\mathcal{O}}^{\times}$.
Then one can define $\mathcal{O}(a):= a \mathcal{O}$ and $\mathcal{o}(a) := a \mathcal{o}$ for all $a \in K$.
A particular case of that is when $K$ is an ordered field, and $\mathcal{O}$ is a convex subring (containing $1$), e.g. the ring of finite elements in $K$, which are elements $a$ with $-n<a<n$ for some $n \in \mathbb{N}$.
A subcase of this last one is if $K$ is what is called a Hausdorff field. Take $\mathcal{C}$ to be the ring of continuous functions $\mathbb{R} \rightarrow \mathbb{R}$ under pointwise sum and product. Take the quotient $\mathcal{G}$ of $\mathcal{C}$ for the relation $f \equiv g$ if $f(t) =g(t)$ for all sufficiently large $t \in \mathbb{R}$. The equivalence classes are called germs at $+\infty$. Then the pointwise operations factor through the quotient map (the sum/product of germs is the germ of the sum/product), and the resulting ring $\mathcal{G}$ is partially ordered by saying that the germ of $f$ is strictly smaller than that of $g$ if $f(t)<g(t)$ for all sufficiently large $t \in \mathbb{R}$.
A Hausdorff field is a subfield of $\mathcal{G}$, and as a consequence of the IVP for continuous functions, such fields are linearly ordered by the ordering described above. Thus the second paragraph gives you an example of $\mathcal{O}$-$\mathcal{o}$ notion which is close to the classical Landau notion.
An example of Hausdorff field is the set $L$ of germs of functions that can be obtained as combinations of $\exp$, $\log$, and algebraic operations. Another one is or the closure of $L$ under solutions of all linear differential equations of order $1$. This contains many of the (germs of) non-oscillating functions that appear in elementary mathematics.
Note that in these examples, given $a,b \in K^{\times}$, one has $a \in \mathcal{O}(b)$ or $b \in \mathcal{O}(a)$, contrary to what is the case for say arbitrary real-valued functions.