Turning my comment into an answer:
It's hard to rigorously state, let alone prove, a negative answer to a question like this; but here's an example which strongly indicates that the answer is indeed negative.
Let $X=[0,1]$, let $A_1=\{0\}$, and let $A_2=\{{1\over 2}\}$. Then $X\setminus A_1$ is connected but $X\setminus A_2$ is not. Lots of analogous things can be done; e.g. if $X$ is a "lollipop" (= a line segment attached to a circle) then removing one point can leave it connected but not simply connected while removing another point can make it contractible. Note that all of the spaces involved here are "nice" (e.g. Hausdorff and compact).
In particular, the OP's guess is incorrect.
To try to turn a negative into a positive, we can ask the following:
Let $X$ be a fixed "nice" space, and define an equivalence relation $\sim$ on the (closed, for simplicity) subsets of $X$ as the smallest equivalence relation with the following two properties:
If $A,B$ are homeomorphic subspaces of $X$, then $A\sim B$.
If $X\setminus A, X\setminus B$ are homeomorphic subspaces of $X$, then $A\sim B$.
The strong failure of a positive answer to your question might(!) be re-interpretable as a frequent coarseness of $\sim$: it may be the case that $\mathcal{P}_{closed}(X)/\sim$ is small in some sense (= "easily classifiable") even when $X$ has lots of closed subsets. You might try to compute $\mathcal{P}_{closed}/\sim$ in a couple simple cases, e.g. $X=[0,1]$ or $X=S^1$ (although even the case of $X=[0,1]^2$ might be too hard to compute by hand, for reasons related to this old answer of mine).