In this definition of total variation of a stochastic process $A$, I don't understand what the author means by "the measure $dD_t(\omega)$ is the total variation of the signed measure signed measure $dA_t(\omega)$."
The definition 1.7.7 is the usual $V_{\mu}(A)= \sup_{\{A_i\}}\sum_i |\mu(A_i)|$, where $\{A_i\}\in \{ \text{partitions of a measurable set}\}$.
One can prove that a function $t \mapsto A_t$ of bounded variation in the sense of Definition 1.7.7 is a function that can be written as the difference of two nondecreasing functions.
Let's now use this characterisation of functions of bounded variation. Note that for $\omega$-almost surely, the path $t \mapsto A_t(\omega )$ is a path of bounded variation, so we may write this as the difference of two non-decreasing functions: $A_t(\omega) = A_t^{(1)}(\omega) - A_t^{(2)}(\omega)$. Now, a nondecreasing function $f$ is a associated to a Borel measure in the following way: $$\mu_f (0, t] = f(t) - f(0)$$ We can then associate $A_t(\omega )$ to the measure $$\mu = \mu_{A^{(1)}} - \mu_{A^{(2)}}$$ This is called the Jordan decomposition of $A$. Thus, the measure $dD(\omega)$ provided in your text is the measure $$d \left(\mu_{A^{(1)}(\omega)} - \mu_{A^{(2)} (\omega)}\right)$$ exhibited above.