The main source of confusion is the definition of a set as "a collection of distinct objects". Such definition leads to Russell's paradox and Cantor's paradox, which are usually overcome through adopting an axiomatic set theory.
In order for one to talk about small and large categories, one first needs to specify the adopted foundations of category theory. There are several foundations in which categories can be defined, the most frequently used is Zermelo-Fraenkel set theory with the axiom of choice (ZFC) and the axiom of the universe. In these settings, you can distinguish two different types of set-theoretical objects, namely (small) sets and proper classes, with respect to a given universe. There are some interesting and subtle differences between sets and proper classes, one which is the size difference. All the set-theoretical objects that are bigger than the chosen universe are proper classes.
The category $\mathbf{FSet}$ of finite sets and their maps is small (with respect to an uncountable universe), because the axioms of ZFC and the uncountable universe imply that both $\mathop{\mathrm{Ob}}(\mathbf{FSet})$ and $\mathop{\mathrm{Mor}}(\mathbf{FSet})$ are small sets. However, $\mathop{\mathrm{Ob}}(\mathbf{Grp})$ is bigger than the universe, i.e. it is a proper class, and hence $\mathbf{Grp}$ is not small.
One reason to care about having small or large categories, is that not all constructions that work for small categories work for large categories. For instance, in some foundations, the functor category $\mathbf{Fun}(\mathcal{C},\mathcal{D})$ exists when $\mathcal{C}$ is small, but not necessary when it is large.
To read more about this subject, you may start with the foundations, and refer to the other links.
P.S. Some sources refer to what is called here set-theoretical object simply by set, and emphasise the word small in the name of small sets, to make the distinction.