30

It seems that similar questions have come up a few times regarding this, but I'm struggling to understand the answers.

My question is a bit more basic, can the difference between the strong Markov property and the ordinary Markov property be intuited by saying:

"the Markov property implies that a Markov chain restarts after every iteration of the transition matrix. By contrast, the strong Markov property just says that the Markov chain restarts after a certain number of iterations given by a hitting time $T$"?

Moreover, would this imply that with a normal Markov property a single transition matrix will be enough to specify the chain, whereas if we only have the strong property we may need $T$ different transition matrices?

Thanks everyone!

Davide Giraudo
  • 181,608
Debreu
  • 873
  • Hmmm... The strong Markov property is stronger, in the sense that if a process satisfies it then it satisfies the simple Markov property. Your intuition seems to go the other way, does it? – Did Aug 13 '16 at 17:13
  • It is hard to give an intuitive difference between the two since the strong Markov process is defined using some rather abstract machinery from measure theory. What is your background? – Wavelet Aug 13 '16 at 17:33
  • Thanks, I was under the impression that the strong markov property was more general. Wikipedia states:

    "The strong Markov property implies the ordinary Markov property, since by taking the stopping time T=t, the ordinary Markov property can be deduced."

    Is that correct?

    I think the problem with this is my background, at my university we take stochastic processes first, followed by measure theory. I think it's so actuaries can learn some useful stuff but it means a lot of the depth is lost on me

     – Debreu Aug 13 '16 at 20:35

3 Answers3

34

A stochastic process has the Markov property if the probabilistic behaviour of the chain in the future depends only on its present value and discards its past behaviour.

The strong Markov property is based on the same concept except that the time, say $T$, that the present refers to is a random quantity with some special properties.

$T$ is called stopping time and it is a random variable taking values in $\{0,1,2,\ldots\}$ such that any value $T=n$ can be determined completely by the values of the chain, $X_0,X_1,\ldots ,X_n$, up to time $n$.

A very simple example is when you throw a coin and you want to stop when you reach $T=n$ heads. $T=n$ is completely determined by the values of the sequence of the previous tosses. Of course, $T$ is random.

The strong Markov property goes as follows. If $T$ is a stopping time, for $m\geq 1$

$$P(X_{T+m}=j\mid X_k=x_k,\;0\leq k <T;\;X_T=i)=P(X_{T+m}=j\mid X_T=i)$$

So conditionally on $X_T=i$ the chain again discards whatever happened previously to time $T$.

In order to determine the(unconditional) probabilistic behaviour of a(homogeneous) Markov chain at time $n$ one needs to know the one step transition matrix and the marginal behaviour of $X$ at a previous time point, call it $t=0$ without loss of generality. ie one should know $P(X_1=j\mid X_0=i)$ and $P(X_0)$.

theoGR
  • 719
  • 21
    ...Except that in discrete time - discrete space, simple Markov property and strong Markov property are the same. So this might not be the most adequate setting to explain their differences. – Did Aug 13 '16 at 18:44
  • That was an awesome answer thanks! I don't know why it hasn't been put that intuitively before.

    I would upvote but I don't yet have the reputation :(

     – Debreu Aug 13 '16 at 20:38
  • 1
    @Did What do you mean by "the same"? – theoGR Aug 14 '16 at 01:42
  • That one property holds if and only if the other holds. – Did Aug 14 '16 at 08:18
  • There is a subtle difference. $X$ Markov and $T$ any random time does not always imply strong Markov property. $X$ Markov + $T$ stopping time implies strong Markov. Only the Markov property is not sufficient. – theoGR Aug 14 '16 at 12:02
18

Here's an intuitive explanation of the strong Markov property, without the formalism:

If you define a random variable describing some aspect of a Markov chain at a given time, it is possible that your definition encodes information about the future of the chain over and above that specified by the transition matrix and previous values. That is, looking into the future is necessary to determine if your random variable's definition is being met. Random variables with the strong Markov property are those which don't do this.

For example, if you have a random walk on the integers, with a bias towards taking positive steps, you can define a random variable as the last time an integer is ever visited by the chain. This encodes information about the future over and above that given by the previous values of the chain and the transition probabilities, namely that you never get back to the integer given in the definition. Such a random variable does not have the strong Markov property.

So with regards to your two questions:

1) The strong Markov property states that after a random variable has been observed which was defined without encoding information about the future, the chain effectively restarts at the observed state. (I'm not sure if this was exactly what you were getting at.)

2) If you define a variable for which the strong Markov property does not hold, and you know it's value, you could use this knowledge to get a better estimate of the future state visitations than if you relied on the transition matrix and current observations alone. The chain is still governed by its transition matrix and you don't need more of them to describe it, but you would have more information than if your variable was strongly Markovian.

user7898912
  • 199
  • You've picked a rather challenging topic to address for your first Answer. When a Question already has an Accepted and upvoted Answer (here for the last two years or more), it is more likely to help if you can highlight what new information is being contributed. The pressure to post quickly is off at this point, and more attention should be given to details of examples, citations for definitions, and the use of mathematical notation to support your reasoning. – hardmath Sep 13 '18 at 19:37
  • 1
    Still not a bad answer I'd say. It puts the measure theory in even simpler terms to the above answer. – Debreu Sep 15 '18 at 14:43
  • @hardmath thanks for the feedback etc. I edited the answer to hopefully make it slightly clearer in a few ways, including why I think this is a reasonable addition (upfront lack-of-formalism claim). – user7898912 Sep 16 '18 at 23:23
  • Thanks for the update. I agree it's a worthwhile addition and I upvoted. – hardmath Sep 17 '18 at 15:48
  • 1
    I am not quite understand your point (2). Do you mean that if T is not a stopping time, we can still get more information from the Markov chain even thought there is not Markov property? Any examples (e.g. how does the coin throwing example applies in this point)? – Raven Cheuk Feb 27 '19 at 09:52
  • This explains the strong Markov property, but how does it differ from the Markov property? Or is it effectively the same (the one property implies the other)? – Sextus Empiricus Nov 17 '20 at 08:28
0

When a random time τ appears, the future development may no longer satisfy the Markov property because the occurrence time of this event is uncertain, so the state after that time may depend on previous states and the realization of the random time. In other words, if we do not consider the correlation between the random time τ and the state, the future development no longer satisfies the Markov property.

SimpleMiant
  • 1