The classical reference for this is Halmos' Lectures on Ergodic Theory; he also proved one of the two initial famous theorems (the other being proved by Rohlin). A more modern reference is Parry's Entropy and Generators in Ergodic Theory.
While both Walters' and Petersen's books are valuable classics and do indeed mention results around these, they don't seem to contain proofs of the statements. Although they do refer to other, more modern works (e.g. Katok-Stepin results).
Succinctly, there are two popular metrizable topologies in this context. Let $G$ be the group of all bimeasurable, invertible maps $f:[0,1]\to[0,1]$ preserving Lebesgue measure (without loss of generality, this is the standard object whose automorphism group one would look at). Then the two topologies are given by the following:
- $d_u(f,g)=\operatorname{leb}\{x\in[0,1]| f(x)\neq g(x)\}$; this is "uniform topology" and it comes from the (isometry group of the) measure algebra (see e.g. Equivalent definition of weakly mixing or Difference between "measure" and "metric").
- $d_w(f,g) = \sum_{n\in\mathbb{Z}_{\geq1}} \dfrac{1}{2^n}\left[\operatorname{leb}(f(B_n)\triangle g(B_n))+ \operatorname{leb}(f^{-1}(B_n)\triangle g^{-1}(B_n))\right]$, where $B_\bullet$ is a dense sequence of sets in the measure algebra, and $\triangle$ denotes the symmetric difference. This "weak topology" comes from the (unitary group of the) space of square integrable functions on the interval.