It sounds like you're thinking of category theory. More accurately, it sounds like you're thinking of foundational systems which approach the mathematical universe as a "big category" as opposed to the $\mathsf{ZFC}$-picture of a "big set".
Categories are basically things consisting of objects with no internal structure at all and arrows connecting them. Sufficiently rich categories can provide frameworks within which we can implement mathematics, analogously to sufficiently rich collections of sets (or set-like things).
If you just want a couple search terms re: foundational theories based on categories, skip to the end of this answer; the middle tries to explain what sort of thing a category is, and why we might care.
Somewhat informally, a category is just a collection of things called objects together with a collection of things called arrows (or morphisms) connecting those objects. There are only a couple properties these need to satisfy, basically saying that the arrows are "function-y": we have to have a notion of "composition" of arrows which is associative (when defined), and we have to have "identity" arrows.
The usual motivating examples of categories have as their objects sets with structure and as arrows structure-preserving functions. For example, sets-and-functions, groups-and-homomorphisms, and topological-spaces-and-continuous-maps each provide very natural examples of categories (note that in the first of these there is no structure at all). We also have very silly examples, like the category whose single object is the set $\{1,2,3,4,5\}$ and whose arrows are those functions from the set to itself which don't move $4$.
However this "inside-the-object" data is forgotten when we look at the category alone: in a category $\mathcal{C}$, objects don't have any "internal structure," they're basically just dots. The only things we can see about an object in a category are how it interacts with other objects in that category via the arrows. I think this matches your idea of "units."
For example, and again somewhat informally, consider the category ${\bf Sets}$ whose objects are sets and whose arrows are functions in the usual sense. There is a unique object $i$ in ${\bf Sets}$ such that for all objects $a\in{\bf Sets}$ there is exactly one arrow from $i$ to $a$ (this $i$ is the "initial object" of ${\bf Sets}$):
$i$ is just the emptyset, and the unique arrow $i\rightarrow a$ is the empty function.
Similarly, there are some objects with the opposite property: call an object $e$ in ${\bf Sets}$ final if for every object $c\in{\bf Sets}$ there is a unique arrow from $c$ to $e$. These objects are exactly
the singletons: the sets with a single element.
The real value of category theory comes from its unifying power. Since we don't care about what's going on "inside" an object, we can take concepts which arise naturally in one category and try to understand how they do - or don't - make sense in another category.
In particular, going back to the previous section we can ask about initial and final objects in an arbitrary category. For example, in the category ${\bf Grp}$ whose objects are groups and whose arrows are group homomorphisms, the initial objects are
the one-element groups
and the final objects are
also the one-element groups.
On the other hand, some categories don't have initial or terminal objects. For a neat example, every monoid $M$ can be thought of as a one-object category $\mathcal{C}_M$: arrows in $\mathcal{C}_M$ correspond to elements of $M$, composition of arrows in $\mathcal{C}_M$ corresponds to multiplication in $M$, the identity arrow of the unique object in $\mathcal{C}_M$ corresponds to the identity element of $M$, and the unique object of $\mathcal{C}_M$ is ... nothing whatsoever, it's just a placeholder object. Anyways, if $M$ has more than one element then the unique object of $\mathcal{C}_M$ has lots (namely, $>1$) of arrows from itself to itself and hence is neither terminal nor final.
A more interesting example concerns direct products. Lots of types of mathematical structure have a notion of direct product (e.g. groups, rings, topological spaces, etc.) and in each of these situations we have the same basic behavior. This behavior can be captured categorically via the notion of universal property; this answer is already getting pretty long so I won't go into the details, but see here.
- One key point here is that (unlike with initial/final objects) the direct product of two objects in a category is an object together with some arrows with certain properties; this is a first step towards the key idea in category theory that ARROWS ARE MORE IMPORTANT THAN OBJECTS. (Indeed, we can formulate category theory entirely without objects!)
Of course this isn't the end of the story, but hopefully it gives a tantalizing taste of the notion of categories. I'll end with two final notes: that the generalizing power of category theory applies to theorems as well as definitions, and that category theory does far more than just directly generalizing common concepts. It's a subject well worth your time (to put it mildly!) even ignoring foundational concerns.
So categories fit the "units and relations" idea, and provide a beautiful framework for generalizing ideas across different mathematical concepts. It turns out that we can also use them to provide a foundational system for mathematics! While some categories are rather boring and too simple to implement mathematics, there are others which are extremely rich and can indeed serve as a working framework. For example, it's generally agreed that (elementary) toposes have this property. Their definition is mysterious at first, but really these are just categories with certain nice properties within which we can make sense of high-level questions we normally think of as set-theoretic. Several "category-based" foundational theories have been proposed; if I recall correctly, the first of these was $\mathsf{ETCS}$, and a more recent example is $\mathsf{SEAR}$.
What kind of primitive notions do you have to assume in ETCS? Can you formulate it without sets?
– Daedalus May 06 '20 at 18:29