On one side, there is some irritation in having to deal with the empty structure as a special case.
There are various useful identities that are only valid in a non-empty domain, for example:
$$
\big ( [ (\forall x) \phi ] \to \psi \big ) \equiv (\exists x) [\phi \to \psi]
$$
when $\psi$ does not mention $x$. These identities are used to put formulas in prenex normal form.
Even the definition of "truth in a structure" requires the domain to be nonempty. A common way of defining truth in a structure requires a function assigning the variables in the language to objects in the domain; there is no such function when the domain is empty, and the entire definition fails in that case. By contrast, the definition of truth for an empty structure is a different, ad hoc definition.
On the other side, one of the main goals of first-order logic is to formalize mathematical objects such as groups, equivalence relations, etc. In many cases, these objects must have a nonempty domain (e.g. groups) or are uninteresting when it is empty (e.g. equivalence relations, posets). So there is little motivation to include the empty domain as a possible model, because it will either not be a model of the theories being studied, or will not be a model of mathematical interest.
It is certainly possible to study logic in which models may be empty; more generally, one studies "free logic" in which terms may not have referents. The reason that this is not done in most texts is that the authors don't see the effort as justifying the benefit.