Upon reading this question, I found myself of thinking of a Paillier with a custom group order of size $p$, where $p$ is a prime number. As a result, I came up with the following approach:
$n' = ab$
$n'$ is a normal RSA modulus with primes $a$ and $b$.
$n = n'\cdot p$.
$n$ is the actual modulus we will use for our Paillier. $p$ is public.
$g = (n+1)^{n'}\mod n^2$
$g$ is now a generator of a subgroup of size $p$ given a modulus of $n^2$.
$g^m \cdot r^n \mod n^2$
This is now an encryption scheme with additive homomorphism, where the message space is $\mod p$. $m \in \mathbb{Z}_p$ and $r \in \mathbb{Z}_n$.
While this may be inefficient due to the large modulus, it seems like it could be a useful tool in some scenarios. It even has the benefit that you do not need to own the private key to make this adjustment to standard Paillier.
I can't imagine that this weakens the original $n'$, as anyone can do this operation given $n'$, and I can't personally see any problems with it on the math level, but I don't consider myself an expert in RSA or Paillier.
It seems too simple for it to not have been discovered before.
I have two questions:
Does this currently exist in the literature? I've done some searches, but I haven't seen this.
Are there any vulnerabilities on the math level that I've overlooked? I'm considering any release of information related to $m$ that doesn't exist in standard Paillier to be a vulnerability.