Let $\mathcal{S}$ be the state space, and the transition kernel to be $K_i(s,ds') = \sum_{a} P^k(ds'|s,a)\pi_{{\theta_i}}(a|s)$.
How do I upper bound $\|K_1-K_2\|$. What I have tried is to follow the definition, that is $$\|K_1-K_2\| = \sup_{\|q\|=1}\|q(K_1-K_2)\|,$$ where $q$ is a measure on $\mathcal{S}$. I don't know how to expand it so that I can use the Lipschitz assumption of $\pi$. If it is the discrete case, it is trivial. $q$ is just a probability vector and $K$ is the transition matrix. When it comes to continuous case, I feel like I lack some knowledge about operators, and I cannot find relevant materials.
Edit: Based on intuition, I found a solution.
$$\|K_1-K_2\| = \sup_{x\in \mathcal{S}}\sup_{A\in \mathcal{B}(\mathcal{S})}\|K_1(x,A)-K_2(x,A)\|\\=\sup_{x\in \mathcal{S}}\sup_{A\in \mathcal{B}(\mathcal{S})} |\int_{\mathcal{S}}\sum_{a}P(ds|x,a)\pi_{\theta_1}(a|x)-\int_{\mathcal{S}}\sum_{a}P(ds|x,a)\pi_{\theta_2}(a|x) | \\=\sup_{x\in \mathcal{S}}\sup_{A\in \mathcal{B}(\mathcal{S})} |\int_{\mathcal{S}}\sum_{a}P(ds|x,a)(\pi_{\theta_1}(a|x)-\pi_{\theta_2}(a|x)) | \\ \leq\sup_{x\in \mathcal{S}}\sup_{A\in \mathcal{B}(\mathcal{S})} \int_{\mathcal{S}}\sum_{a}P(ds|x,a)|\pi_{\theta_1}(a|x)-\pi_{\theta_2}(a|x) | .$$ Next, we can use the Lipschitz continuity of the policy and the fact the $\int_{A}P(ds|x,a) \leq 1$ to get an upper bound.
