To start things off, let's whip up a simple-to-understand example showing that the sort of switching you have in mind can't possibly be legal in general (it sounds like you already know this, but other readers might benefit from its explicit inclusion here):
Imagine a large dining room with lots of tables, where all but one of the tables have multiple chairs and a single table has a single chair. Let $\varphi(a,b)$ be the formula "$a$ is a chair at table $b$." It may well be the case that exactly one table has exactly one chair at it. However, barring some very surprising chair-related shenanigans, it should not be the case that exactly one chair belongs to exactly one table: presumably every chair belongs to exactly one table. In any reasonable instance of the above situation, $\exists!x\exists!y\varphi(x,y)$ should be false but $\exists!y\exists!x\varphi(x,y)$ should be true.
OK, now we know that there's got to be something wonky going on; let's find it.
As is often the case, the culprit lies in the set of our human-linguistic-efficiency-oriented foibles. Despite the notation we use for it, "$\exists!$" should really be thought of as a mixed-quantifier expression: when we write out $$\exists !x\varphi(x)$$ in terms of the basic quantifiers $\exists$ and $\forall$, what we get is $$\exists x\forall x'[\varphi(x)\wedge(\varphi(x')\rightarrow x=x')]$$ (or something similar). In particular, this means that a quantifier prefix like $$\exists!x\exists!y$$ should really be thought of as $$\exists x\forall x'\exists y\forall y',$$ and the switching you have in mind would require passing the $\forall x'$ through the $\exists y$ and passing the $\exists x$ through the $\forall y'$. Each of these two maneuvers is of course illegal in general. Incidentally, the discussion here is relevant.
Finally, as a fun exercise let's dive into the parenthetical remark above. We can also interpret "$\exists!$" as having the form "$\forall\forall\exists$" (phrasing this precisely takes a bit of work, but the idea should be intuitively clear):
Regardless of what $\varphi$ is, the formula $$\exists!x\varphi(x)$$ is equivalent to $$\forall x\forall x'\exists x''[\varphi(x'')\wedge (x\not=x'\rightarrow\neg(\varphi(x)\wedge\varphi(x')))].$$ Note that the "matrix" (= inside-the-quantifiers part) of this latter formula is a Boolean combination of instances of $\varphi$ itself and quantifier-free formulas, so this really is "just" a $\forall\forall\exists$-interpretation of $\exists!$. Note also that the existential bit doesn't interact with the universal bits at all so this also gives a $\forall\exists\forall$ and $\exists\forall\forall$ interpretation; however, each of these is strictly worse than the $\exists\forall$ interpretation above.
Meanwhile it's not hard to show that we cannot interpret $\exists!$ in a purely-existential, purely-universal, or $\forall\exists$ way. So the pair of interpretations we've seen above - $\exists\forall$ and $\forall\forall\exists$ - is in a precise sense optimal.