Markov chain expected steps

Question

Consider the numbers 1, 2, ..., 12 written around a ring as they usually are on a clock. Consider a Markov chain that at any point jumps with equal probability to the two adjacent numbers. (a) What is the expected number of steps that $X_n$ will take to return to its starting position? (b) What is the probability $X_n$ will visit all the other states before returning to its starting position?

I can see that we simply have a doubly stochastic 12x12 matrix, with $P_x(x, x+1) = P_x(x, x-1) = 1/2$ (here assuming 12 + 1 = 1). Thus, the stationary distribution $\pi[i] = 1/12$ for all $i \in [1:12]$. Then, the expected number of steps from any $i$ to $i$ is simply $1 / \pi[i] = 12$, right?

(b) is a little trickier - we know that all states here are null recurrent. However, I'm not sure how to think about this probability. Any tips?

Answer 1

For part (b), assume (without loss of generality) that from state 1 we went to state 2 and not state 12 (it does not matter, because the ring is symmetric).

Then we want to know the probability, starting at state 2, that we reach state 12 before getting to state 1.

Let $p_k$ be the probability, starting at state $k$, that we reach state 12 before getting to state 1. Then $p_1 = 0$, $p_{12} = 1$, and $p_k = \frac{p_{k+1} + p_{k-1}}{2}$ for $2 \le k \le 11$.

We can just solve this system of equations directly, getting $p_k = \frac{k-1}{11}$ for all $k$, or we can use some trickery.

Here's one "trickery" way to do it. Imagine a betting game in which you start with 2 dollars, and every time you bet you either win or lose a dollar with equal probability. If you keep playing until you get to 12 dollars or have 1 dollar left what is the expected amount of money you have at the end?

1. Because the game is fair, it must still be 2 dollars.
2. The game is clearly isomorphic to the random walk we're taking here. Starting with 2 dollars, we end with 12 dollars with probability $p_2$ and 1 dollars otherwise, so the expected amount of money at the end is $12p_2 + 1(1-p_2)$.

Setting these equal, we get $12p_2 + (1-p_2) = 2$, or $p_2 =\frac1{11}$.

Answer 2

You already have an answer by Misha, but maybe someone would like a more concrete description of how to calculate it with matrices.

For b) we start at $2$ and use the idea in Misha's answer above : We want to find the probability to get to 1 through 12 instead of from 2. Now we can put the leak in state 2's transfer to state 1, just null that transfer probability, as well as the hop from 1 to 12 and from 1 to 2. Then we let state $1$ become a trapping state, keeping 100% of whatever ends up there. The modified P matrix cropped down to a 6 state clock would be:

$${\bf P} = \frac 1 2 \left[\begin{array}{cccccc}\bf 2&\bf 0&0&0&0&1\\\bf 0&0&1&0&0&0\\0&1&0&1&0&0\\0&0&1&0&1&0\\0&0&0&1&0&1\\ \bf 0&0&0&0&1&0\end{array}\right]$$

All the modifications done to the matrix are in bold.

Now what we want to calculate is $$\lim_{n\to \infty}\left\{[1,0,0,\cdots]({\bf P})^n[0,1,0,\cdots,0]^T\right\}$$

For $n=256$ we get 0.83301 which is rather close to $1/12 \approx 0.83333$

---

Hmm, I see now that I slightly misinterpreted the question. The question was not to return to the first from the last state, but just to reach the last state before reaching the first one for the second time. That explains why we got $1/12$ instead of $1/11$. The fix would be to change so that the last state becomes the trapping state instead of the first one. The necessary matrix modifications could be a good exercise for the eager student.

Answer 3

My attempt at the a) would be to build the matrix, call it $\bf P$, then build a diagonal matrix, call it $\bf D$, where $${\bf D}_{ii}= \begin{cases}0, & i = \text{initial state}\\1,& i \neq \text{initial state}\end{cases}$$ And an initial state column vector $\bf s$ : $${\bf s}_{i}= \begin{cases}1, & i = \text{initial state}\\0,& i \neq \text{initial state}\end{cases}$$

---

EDIT, to explain D matrix : We see that the $\bf D$, if multiplying from the left will zero the same row as the initial state encoded in $\bf s$. If we do not do it, then when counting the contribution to the expected value after hop $k$, the probability that we have just counted that has reached back to the initial state will flow back out of the initial state and get counted once again. We don't want that, because then we would include second and third and fourth et.c. returns to the initial state as well.

---

Then the expected value should be: $$E={\bf s}^T\left(\sum_{k=1}^{\infty}k({\bf PD})^k\right){\bf Ps}$$

This gives me $E=11.000$ if I run for $>8000$ hops. But it might be that I have an index off in my calculation so that it would be $12$ because I miss count the first one or something like that.

---

To show you that it actually works here is a plot :