You are right, $\eta_X(x) = \{x\}$ is forced, since $P(\eta_X)$ must be injective by the monad axioms, so $\eta_X(x) = \emptyset$ is not allowed. But for the multiplication there is a different choice, namely
$$\mu_X(\mathcal{T}) := \begin{cases} \emptyset & \emptyset \in \mathcal{T} \\ \bigcup \mathcal{T} & \emptyset \notin \mathcal{T} \end{cases}$$
for a family of subsets $\mathcal{T} \subseteq P(X)$. In words: we take the usual union if all the sets in the family are non-empty, otherwise just refuse to do anything and return the empty set. I suggest to call this the stubborn union.
Proof of the monad axioms
Naturality is obvious. The equations $\mu_X(P(\eta_X)(A)) = \mu_X(\{\{x\} : x \in A\}) = A$ and $\mu_X(\eta_{P(X)}(A)) = \mu_X(\{A\}) = A$ for $A \subseteq X$ are clear.
To prove associativity, let $\mathcal{R} \subseteq P^2(X)$ (this is a family of families of subsets). We compute
$$\begin{align*}
\mu_X(\mu_{P(X)}(\mathcal{R})) & = \begin{cases} \mu_X(\emptyset) & \emptyset \in \mathcal{R} \\ \mu_X(\bigcup \mathcal{R}) & \emptyset \notin \mathcal{R} \end{cases} \\
& = \begin{cases} \emptyset & \emptyset \in \mathcal{R} \\ \emptyset & \emptyset \notin \mathcal{R} \text{ and } \emptyset \in \bigcup \mathcal{R} \\ \bigcup \bigcup \mathcal{R} & \emptyset \notin \mathcal{R} \text{ and } \emptyset \notin \bigcup \mathcal{R} \end{cases}
\\
& = \begin{cases} \emptyset & \emptyset \in \mathcal{R} \text{ or } \emptyset \in \bigcup \mathcal{R} \\ \bigcup \bigcup \mathcal{R} & \emptyset \notin \mathcal{R} \text{ and } \emptyset \notin \bigcup \mathcal{R} \end{cases}
\end{align*}$$
and
$$\begin{align*}
\mu_X(P(\mu_{X})(\mathcal{R})) &= \mu_X(\{\mu_X(\mathcal{T}) : \mathcal{T} \in \mathcal{R}\}) \\
& = \begin{cases} \emptyset & \emptyset \in \{\mu_X(\mathcal{T}) : \mathcal{T} \in \mathcal{R}\}\\ \bigcup \{\mu_X(\mathcal{T}) : \mathcal{T} \in \mathcal{R}\} & \text{else} \end{cases} \\
& = \begin{cases} \emptyset & \emptyset \in \mathcal{R} \text{ or } \emptyset \in \bigcup \mathcal{R} \\ \bigcup \{\bigcup \mathcal{T} : \mathcal{T} \in \mathcal{R}\} & \emptyset \notin \mathcal{R} \text{ and } \emptyset \notin \bigcup \mathcal{R} \end{cases} \\
& = \begin{cases} \emptyset & \emptyset \in \mathcal{R} \text{ or } \emptyset \in \bigcup \mathcal{R} \\ \bigcup \bigcup \mathcal{R} & \emptyset \notin \mathcal{R} \text{ and } \emptyset \notin \bigcup \mathcal{R}.\end{cases}
\end{align*}$$
Any hints if this monad already appears in the literature and if it has a name are appreciated.
Algebras for the monad
This raises of course the question how to describe the algebras for this monad. One can mimic the proof for the usual monad structure, where the algebras are sup-lattices, and arrive at those partially ordered sets where
- every non-empty subset has a supremum,
- there is a largest element.
For the lack of a better name (is there one?), let's call these twisted sup-lattices. For example, $]0,\infty]$ with the usual ordering is a twisted sup-lattice, but not a sup-lattice , since we don't have a smallest element (the supremum of the empty subset).
The free twisted sup-lattice on a set $X$ has underlying set $ P(X)$ but not with the usual ordering! For non-empty subsets it's the usual one, but we have $A \leq \emptyset$ for all $A$.
Classification of monad structures on $P$
Actually, the normal union and the "stubborn union" provide the only monad structures.
Theorem. There are exactly two monad structures on the functor $P : \mathbf{Set} \to \mathbf{Set}$.
The proof is similar to the one in MO/479643 (where the domain of the natural transformation was $P^{\times n} = P \times \cdots \times P$, not $P^2 = P \circ P$). In fact, I have also used the name "stubborn union" for a certain operation $P \times P \to P$ there.
Let $\mu : P^2 \to P$ be a natural transformation such that $(P,\eta,\mu)$ is a monad. We already know $\eta_X(x) = \{x\}$. The monad axioms are $\mu_X(\{A\}) = A$ for $A \subseteq X$, $\mu_X(\{\{x\} : x \in A\}) = A$ for $A \subseteq X$, and associativitiy of $\mu$. Actually, we will not use associativity, it will follow!
Let $\mathcal{T} \subseteq P(X)$ be a family of sets. Consider the set $Y := \coprod_{A \in \mathcal{T}} A$ with the obvious map $p : Y \to X$ defined by extending the inclusions $A \hookrightarrow X$. We can also regard $\mathcal{T}$ as a family of disjoint subsets of $Y$, and naturality of $\mu$ with respect to $p$ yields $\mu_X(\mathcal{T}) = \mu_X(p(\mathcal{T})) = p( \mu_Y(\mathcal{T}))$. Therefore, in order to prove that $\mu$ is the union or the "stubborn union", it suffices to look at $Y$. Hence, we may assume $X = \coprod_{A \in \mathcal{T}} A$.
Then $\mu_X(\mathcal{T})$ is a subset of $X = \coprod_{A \in \mathcal{T}} A$, hence is of the form $\coprod_{A \in \mathcal{T}} A'$ for subsets $A' \subseteq A$. If $\sigma : A \to A$ is a permutation, we can extend it to $\sigma : X \to X$, and then naturality of $\mu$ with respect to $\sigma$ shows that $A'$ is invariant under $\sigma$. Hence, $A' = \emptyset$ or $A' = A$. Let us denote by $\mathcal{T}' \subseteq \mathcal{T}$ the set of those non-empty $A \in \mathcal{T}$ such that $A'=A$. Hence,
$$\mu_X(\mathcal{T}) = \coprod_{A \in \mathcal{T}'} A = \bigcup_{A \in \mathcal{T}'} A.$$
Notice that there is a canonical map
$$q : X \to \mathcal{T}, ~ x \mapsto A \text{ if } x \in A \in \mathcal{T}.$$
On the one hand, we have
$$q(\mu_X(\mathcal{T})) = \bigcup_{A \in \mathcal{T}'} q(A) = \bigcup_{A \in \mathcal{T}'} \{A\} = \mathcal{T}'.$$
On the other hand, by naturality we conclude
$$q(\mu_X(\mathcal{T})) = \mu_{\mathcal{T}}(q(\mathcal{T})) = \mu_{\mathcal{T}}(\{q(A) : A \in \mathcal{T}\}).$$
We have $q(A) = \{A\}$ for every non-empty $A \in \mathcal{T}$. Therefore, if $\emptyset \notin \mathcal{T}$, we conclude from the above calculations that $\mathcal{T}' = \mu_{\mathcal{T}}(\{\{A\} : A \in \mathcal{T}\}) = \mathcal{T}$, and hence $\mu_{X}(\mathcal{T}) = \bigcup_{A \in \mathcal{T}} A$ is the normal union.
Now consider the "problematic" case $\emptyset \in \mathcal{T}$, and let us write $\mathcal{T}^+ := \mathcal{T} \setminus \{\emptyset\}$. Our calculation above shows that
$$\mathcal{T}' = \mu_{\mathcal{T}}(\{\emptyset\} \cup \{\{A\} : A \in \mathcal{T}^+\}).$$
If $\mathcal{T} = \{\emptyset\}$, we get $\mathcal{T}' = \mu_{\mathcal{T}}(\{\emptyset\}) = \emptyset$ and we are done. So assume that $\mathcal{T} \neq \{\emptyset\}$, i.e. $\mathcal{T}^+ \neq \emptyset$.
Now, there are two possible values for
$$W := \mu_{\{\star\}}(\{\emptyset, \{\star\}\}),$$
which is a subset of $\{\star\}$.
Case 1. We have $W = \emptyset$. We use the unique map $f : \mathcal{T} \to \{\star\}$. Then naturality of $\mu$ with respect to $f$ shows
$$f(\mathcal{T}') = \mu_{\{\star\}}(\{\emptyset\} \cup \{\{\star\} : A \in \mathcal{T}^+\}) = \mu_{\{\star\}}(\{\emptyset,\{\star\}\}) = W = \emptyset,$$
hence $\mathcal{T}' = \emptyset$. This means $\mu_X(\mathcal{T}) = \emptyset$. Hence, $\mu$ is the "stubborn union".
Case 2. We have $W = \{\star\}$. Notice that $\mathcal{T}' = \mu_{\mathcal{T}}(\{\emptyset\} \cup \{\{A\} : A \in \mathcal{T}^+\})$ is a subset of $\mathcal{T}^+$ that is invariant under all permutations of $\mathcal{T}$ that fix the empty set, by naturality of $\mu$. Hence, it must be $\emptyset$ or $\mathcal{T}^+$. But if it was $\emptyset$, with $f$ as above we get $\emptyset = f(\mathcal{T}') = W = \{\star\}$, a contradiction. Hence, $\mathcal{T}' = \mathcal{T}^+$, meaning that $\mu$ is the normal union. $\checkmark$
For $U = {{x} | x \in A} $ with $A \subset X$, $$ \mu'X(U) = \mu'_X({{x} | x \in A}) = \bigcup{x \in A} \mu'X({{x}}) = \bigcup{x \in A} {x} = A. $$
Lastly, for a general $U \in \mathcal{P}^2(X)$, we can express $U$ as a union of singletons and apply the same reasoning to conclude.
– Mechap Oct 31 '24 at 17:05