Consider the absolute value of the first projection $|\pi_1| : \Bbb S^1 \to [0, 1]$ that maps $(s, t) \mapsto |s|$.
Notice that $\Bbb S^1 / \sim_{|x| = |a|, |y| = |b|}$ is precisely the set of all fibers above $|\pi_1|$. You can check this yourself and it is tedious but not too hard. You can use the following fact to get started: if $(x, y), (a, b) \in \Bbb S^1$ such that $|x| = |a|$, then $$|y| = \sqrt{1 - |x|^2} = \sqrt{1 - |a|^2} = |b|$$
Also, notice that $|\pi_1|$ is an open surjective continuous map. This is because the projection $\pi_1 : \Bbb S^1 \to [-1, 1]$ is an open surjective continuous map and the restricted absolute value map $| \cdot | : [-1, 1] \to [0, 1]$ is also an open surjective continuous map (the absolute value map is not open in general but in this restricted case it is).
Hence, $|\pi_1|$ is a quotient map since open surjective continuous maps are quotient maps.
Thus, by a standard result about quotient maps (the "definition" you talk about in your last paragraph, cf. Corollary $22.3$ in Munkres), $|\pi_1|$ induces a homeomorphism $|\pi_1|^\ast : \Bbb S^1 / \sim_{|x| = |a|, |y| = |b|} \to [0, 1]$ via $\pi : \Bbb S^1 \to \Bbb S^1 / \sim_{|x| = |a|, |y| = |b|}$ such that $|\pi_1| = |\pi_1|^\ast \circ \pi$.
Note: By symmetry, you could have equally used $|\pi_2| : \Bbb S^1 \to [0, 1]$ instead of $|\pi_1| : \Bbb S^1 \to [0, 1]$ and gotten a different homeomorphism from $\Bbb S^1 / \sim_{|x| = |a|, |y| = |b|}$ to $[0, 1]$.