This is true if $X$ is a connected topological manifold. Indeed, according to this post, for any two points $x,y\in X$ there exists a homeomorphism $\phi$ satisfying $\phi(x) =y$. This homeomorphism restricts to a homeomorphism from $X\setminus \{x\}$ to $X\setminus\{ y\}$.
This is not true in general for connected topological spaces and even for manifolds with boundary. The space $X=[0,1]$ is a counterexample. Removing any interior point disconnects $X$ while removing a boundary point does not.
Added later:
More generally, the property you mentioned holds for topological spaces that are homogeneous (with respect to their homeomorphism group). A topological space $X$ is said to be homogeneous if for any $x,y \in X$ there exists a homeomorphism of $X$ sending $x$ to $y$. As the argument given above shows, this implies that $X \setminus \{x \}$ and $X \setminus\{ y\}$ are homeomorphic for any $x,y$. As remarked above, connected topological manifolds are important examples of homogeneous spaces. There are a lot of other examples, such as topological groups, the Cantor set and any set with the discrete topology or the trivial topology.