$S$ can have a lot of elements, but in this case, its cardinality is always a multiple of $4$.
Claim : The cardinality of $S$ is always a power of $2$. Therefore, if $|S| \geqslant 4$, then $4||S|$.
First of all, $S$ is a finite abelian group. Let us write $x \oplus y = xy \in S$ when $(x,y) \in S^2$ and,
$$
\otimes : \left\{
\begin{array}{rcl}
\mathbb{Z}/2\mathbb{Z} \times S & \rightarrow & S\\
(c,x) & \mapsto & x^c
\end{array}
\right.
$$
Since elements of $S$ verify $x^2 = 1$, this operation is well-defined and $(S,\oplus,\otimes)$ is a $\mathbb{Z}/2\mathbb{Z}$-vector space. If we call $d$ its dimension ($S$ is finite, thus has a finite dimension), then,
$$
|S| = 2^d.
$$
Moreover, as a group, $S \cong (\mathbb{Z}/2\mathbb{Z})^d$.
Using the Chinese lemma and some modular arithmetic, we can show more precisely that if,
$$
n = 2^{a_0}p_1^{a_1}p_2^{a_2} \cdots p_k^{a_k},
$$
with $a_0 \geqslant 0$ and $a_i \geqslant 1$ when $i \geqslant 1$. Then,
$$
a_0 \leqslant 1 \Rightarrow |S| = 2^k, \qquad a_0 = 2 \Rightarrow |S| = 2^{k + 1}, \qquad a_0 \geqslant 3 \Rightarrow |S| = 2^{k + 2}.
$$
Notice that in particular, $|S| \geqslant 2$ unless $n = 2$. It is due to the fact that $-1$ and $1$ always belong to $S$, and when $n \neq 2$, $-1 \neq 1$ in $\mathbb{Z}/n\mathbb{Z}$. Notice also that if gives the following caracterisation,
We have equivalence between $4||S|$, $|S| \geqslant 4$, $S \neq \{-1,1\}$ and $n$ is neither a prime number, nor twice a prime number.
It explains why your solution works. If $4$ does not divide the cardinality of $S$, then $S = \{-1,1\}$ and we can prove that the optimal solution is actually the set of elements of $(\mathbb{Z}/n\mathbb{Z})^\times$ except $-1$, or in other words, all elements of $(\mathbb{Z}/n\mathbb{Z})^\times\backslash S$ plus $1$. Here is a proof below.
In general, set $T$ to be the set of all $x \in (\mathbb{Z}/n\mathbb{Z})^\times$ such that $x^2 \neq 1$. As you wrote it,
$$
\prod_{t \in T} t = 1,
$$
because elements of $T$ go by pairs of distinct classes which are inverse the one with each other in $\mathbb{Z}/n\mathbb{Z}$.
Then, notice that the equation $x = -x$ has for only solutions $x = 0$, and $x = \frac{n}{2}$ if $n$ is even. Except in the particular case $n = 2$, $\frac{n}{2}$ is not coprime with $n$ so it can't be in $A$. If $n = 2$, $A = \{1\}$ is the only optimal solution. Notice also that $n = 2$ is the only case where $|S| = 1$. Let us get rid of this case first.
If $n = 2$ (equivalently, $|S| = 1$), the only optimal solution is $A = \{1\}$, which contains $1$ element.
From now on, assume that $n \neq 2$, so all elements of $S$ are different from their opposite. In particular, $S$ contains an even number of elements and,
$$
\prod_{s \in S} s = (-1)^{|S|/2},
$$
because elements of $S$ go by pair $\{x,-x\}$. In particular, as you noticed it, when $4||S|$, we can put all elements of $S$ (thus all elements of $(\mathbb{Z}/n\mathbb{Z})^\times$) in $A$. In other words,
If $4||S|$ (equivalently, $|S| \geqslant 4$), the only optimal solution is $A = (\mathbb{Z}/n\mathbb{Z})^\times$, which contains $\varphi(n)$ elements.
Assume now that $4\!\!\!\not||S|$. In this case, the product of all elements of $(\mathbb{Z}/n\mathbb{Z})^\times$ equals $-1$ so $(\mathbb{Z}/n\mathbb{Z})^\times$ is not a solution but $(\mathbb{Z}/n\mathbb{Z})^\times\backslash\{-1\}$ is a solution. Clearly, it is optimal and all optimal solutions are of the form $(\mathbb{Z}/n\mathbb{Z})^\times\backslash\{x\}$ for some $x \in (\mathbb{Z}/n\mathbb{Z})^\times$. But, for all $x$,
$$
\prod_{y \in (\mathbb{Z}/n\mathbb{Z})^\times\backslash\{x\}} y = x^{-1}\prod_{y \in (\mathbb{Z}/n\mathbb{Z})^\times} y = -x^{-1}.
$$
We deduce that $(\mathbb{Z}/n\mathbb{Z})^\times\backslash\{x\}$ is a solution if and only if $x = -1$. It lead to the following result.
If $n \neq 2$ and $4\!\!\!\not||S|$ (equivalently, $|S| = 2$), the only optimal solution is $A = (\mathbb{Z}/n\mathbb{Z})^\times\backslash\{-1\}$, which contains $\varphi(n) - 1$ elements.
