The proof in the other direction does not work, but not for the reason you stated. For a clone $\mathcal{C}$, the set $\mathcal{C}_k$ of all $k$-ary operations in $\mathcal{C}$ can be viewed as a subset of $A^{A^k}$ and it can be shown that
$$
\mathcal{C}=\mathrm{Pol}(\{\mathcal{C}_1, \mathcal{C}_2, \mathcal{C}_3, \dots\}).
$$
So when $A$ is finite, $\mathcal{C}$ is determined by $\mathrm{Inv}(\mathcal{C})$ and when $A$ is infinite, it is determined by its "infinitary invariant relations".
A clone for which $\mathcal{C}=\mathrm{Pol}(\mathrm{Inv}(\mathcal{C}))$ is called locally closed. The reason for this naming is because being locally closed is equivalent to the following property: if for an operation $f:A^k\to A$ we have that for every finite subset $B$ of $A$ there is $g\in \mathcal{C}_k$ such that $f|_{B^k}=g|_{B^k}$, then $f\in\mathcal{C}$.
Let $A=\mathbb{Z}$. Let $f:A\to A$ be the permutation that switches 0 and 1, let $g:A\to A$ be the the map $g(x)=x+1$ and $h:A\to A$ be the map $h(x)=x-1$. Let $\mathcal{C}=\mathrm{Clo}(\{f,g,h\})$. Note that $g^ifh^i$ is the permutation that switches $i$ and $i+1$. So every permutation on any finite subset of $A$ can be generated from $f,g$, and $h$. If $\mathcal{C}$ were locally closed, then every bijection $A\to A$ would belong to $\mathcal{C}_1$. But there are uncountably many bijections $A\to A$ and $\mathcal{C}_1$ must be countable since it is generated by $f,g$ and $h$. Hence $\mathcal{C}$ is not locally closed.