7

I'm currently trying to understand (intuitively) what a stationary distribution of a Markov Chain is? In our lecture notes, we're given the following definition:

def

This was of little benefit to my understanding, so I've tried searching online for a more useful explanation. I then found the following video, which improved my understanding to the extent that I now understand that stationary distributions are to do with looking at what happens to the probabilities at each state within a Markov Chain when time becomes infinitely large. This is still not a sufficient enough understanding of the concept though.

For example, I've been asked to show that $$ \pi_{a} = \left( \frac{2}{5}, \frac{3}{5}, 0, 0, 0 \right) \\ \pi_{b} = \left( 0, 0, 1, 0, 0 \right) \\ \pi_{c} = \left( 0, 0, 0, \frac{3}{5}, \frac{2}{5} \right) $$ are stationary distributions with respect to the Markov Chain with one-step transition martix $$ \mathbf{P} = \left( \begin{array}{ccccc} \frac{1}{2} & \frac{1}{2} & 0 & 0 & 0 \\ \frac{1}{3} & \frac{2}{3} & 0 & 0 & 0 \\ 0 & 0 & 1 & 0 & 0 \\ 0 & 0 & 0 & \frac{2}{3} & \frac{1}{3} \\ 0 & 0 & 0 & \frac{1}{2} & \frac{1}{2} \end{array} \right) $$

How would you do this? What is a stationary distribution, with respect to this example?

Also, could someone please confirm that I'm correct in thinking that the notation $p_{ij}$ denotes the probability of the process moving from the state $i$ to the state $j$?

M Smith
  • 2,757
  • To answer the problem they gave you, you need to show that each of $\pi_a,\pi_b, \pi_c$ satisfy all three conditions $S1,S2,S3$ to be a stationary distribution. – Mick A Mar 07 '16 at 00:08
  • A stationary distribution is one which, if your system starts with that distribution, it stays at that distribution. It is closely related to, but not the same as, a limiting distribution. More precisely, there is a theorem which says that a certain class of Markov chains have a unique stationary distribution and always converge to this stationary distribution. But other chains have a unique stationary distribution and don't approach it, while still others approach one of their non-unique stationary distributions. – Ian Mar 07 '16 at 00:15
  • @Ian where can I find what class of chains belong to the different possible situations you have described? – PleaseHelp Nov 08 '16 at 18:46
  • @atulgangwar I don't really know a name for the result but irreducible chains have a unique stationary distribution while aperiodic chains approach some stationary distribution. (Both of these assume a finite state space.) – Ian Nov 08 '16 at 18:50

1 Answers1

5

"Also, could someone please confirm that I'm correct in thinking that the notation $p_{ij}$ denotes the probability of the process moving from the state $i$ to the state $j$?" $(*)$

Correct

"How would you do this? What is a stationary distribution, with respect to this example?"

If the chain starts in state $3$ it stays there forever because according to $(*)$ there is zero probability to move to another state.

Therefore

$\pi_{b} = ( 0, 0, 1, 0, 0)$

is an obvious stationary distribution.

If the chain starts in state $1$ or $2$ it stays there forever because according to $(*)$ there is zero probability to move to another state.

If the chain starts in state $4$ or $5$ it stays there forever because according to $(*)$ there is zero probability to move to another state.

Now you can treat these as two $2 \times 2$ matrices and use the result that a vector which fulfills:

$\mathbf{\hat{\pi}} \mathbf{P} = \mathbb{\hat{\pi}}$ $\:\:(**)$

is a stationary distribution.

So you solve these two sets of systems of equations to get the remaining stationary distributions. Here you also need to use that $\hat{\pi}$ is a probability vector; that is, its components sum to one.

"I now understand that stationary distributions are to do with looking at what happens to the probabilities at each state within a Markov Chain when time becomes infinitely large"

You also have this theorem that can be good to know:

If the Markov chain is irreducible and aperiodic then

$\lim \limits_{n \to \infty} P^n = \hat{P}$

where $\hat{P}$ is a matrix whose rows are identical and equal to the stationary distribution $\mathbb{\hat{\pi}}$ for the Markov chain defined by equation $(**)$.

JKnecht
  • 6,637