3

I am taking a course on Communication Systems (from an engineering point of view). While I'm usually very interested in the formal mathematics, this time I would like to avoid it, since I don't have a good background on probability theory; also, many of my colleagues have little to no interest in formal mathematics at all, and I would like to understand this concept in a way that would be easy to propagate to them.

So, apparently to understand the meaning of ergodicity, one needs to know what is the ensemble average and what is the time average of a random process. After reading this answer on Math.SE, and three related entries on Wikipedia (Ergodic Process, Ergodicity, Stationary ergodic process), this is what I understand:

  • A random process is like a random variable, but it's outcomes are "waveforms" (a.k.a. functions) instead of numbers.

  • The ensemble average is the average of the outcomes of the random process, and therefore is another function (waveform) by itself. A given random variable will have one ensemble average (one function).

  • As opposed to the ensemble average, a random process can have many (possibly infinitely many) time averages, since every outcome of the random process (i.e., every waveform) has its own time average, which is the average value of the waveform. That is, given an outcome $x(t)$, it's time average will be given by

$$\lim_{T \to \infty} \dfrac{1}{T} \int_{-\frac{T}{2}}^{\frac{T}{2}}x(t)dt$$

After reading those wikipedia pages, it seems that an ergodic process is a process that satisfies "the time average is equal to the ensemble average". But, which time average? To my understanding there are many time averages. Which one? All of them? Their mean? At least one of them? Or what?

Also, I would like to take a better look on the following examples, found in the linked wikipedia pages:

Example 1.

Suppose that we have two coins: one coin is fair and the other has two heads. We choose (at random) one of the coins, and then perform a sequence of independent tosses of our selected coin. Let X[n] denote the outcome of the nth toss, with 1 for heads and 0 for tails. Then the ensemble average is ½  (½ +  1) = ¾; yet the long-term average is ½ for the fair coin and 1 for the two-headed coin. Hence, this random process is not ergodic in mean.

Is this correct? (of course my doubt here is a consequence of the fact that I don't know the answer for my bold question above).

Example 2.

Ergodicity is where the ensemble average equals the time average. Each resistor has thermal noise associated with it and it depends on the temperature. Take N resistors (N should be very large) and plot the voltage across those resistors for a long period. For each resistor you will have a waveform. Calculate the average value of that waveform. This gives you the time average. You should also note that you have N waveforms as we have N resistors. These N plots are known as an ensemble. Now take a particular instant of time in all those plots and find the average value of the voltage. That gives you the ensemble average for each plot. If both ensemble average and time average are the same then it is ergodic.

It says "take a particular instant of time in all those plots". Does any instant works? Or rather, do I have to take "all of them" (one at a time)? Taking only one instant of time doesn't seem right... Rather, shouldn't I be taking some sort of limit to infinity? Also, it refers to "the time average" as if it was only one, but to my understanding there are N different time averages here, since there are N waveforms.

Community
  • 1
Pedro A
  • 2,846
  • 1
  • 18
  • 39

1 Answers1

5

Your question is intuitive, so I will try to answer through one very intuitive example.

Example:

Dynamics: You have some money, say 100$, and we're playing a coin game with a fair coin. Each time the coin lands heads (H) we take your money and multiply them by 1.5 and every time the coin lands tails (T) we take your money and multiply them by 0.6.

Averages: Let W(t) denote your wealth at time t, and let this process run for a finite time and take the average over your wealth for that time. This is the finite time-average:

$\left\langle W(t)\right\rangle _{T} = \frac{1}{T}\sum_{t=0}^{T}W(t) $

In contrast, assume that we have N of these processes that we let run until t and then take the average over N. This is the finite ensemble-average of the observable N wealth processes.

$\left\langle W(t)\right\rangle _{N} = \frac{1}{N}\sum_{i=1}^{N}W_i(t)$

Where i denotes the ith of N processes and the average is taken at time t. Letting $T\rightarrow\infty$ and $N\rightarrow\infty$ you get the ensemble average (or expectation operator) and the time average.

Now that we have introduced the relevant dynamics (the coin game) and the averages, lets restate your definition of ergodicity - namely that the time average equals the ensemble average.

1) You are correct in asserting that each of the N coin tossing sequences have a time-average. However, in the limit, all of those N time averages are equal exactly because they are governed by the same dynamic. And they will all converge to the absorbing boundry (zero) with probability one. If you find that unintuitive, try to simulate a bunch of those processes, and plot their distribution. What you will get is a log-normal with diverging moments. E.g. there will be one-in-a-million who's wealth grows exponentially. The bulk of the ensemble will be close to zero.

2) Is the above wealth generating process ergodic? No. The ensemble average will predict infinite positive growth$^*$ while the time-average will converge to zero.$^{**}$ This is commonly known as the St. Petersburg paradox. Unrelated to your question, but both interesting and important: It is possible to create an ergodic observable using the logarithm which solves the St. Petersburg paradox.

Hope you can use this. If you want to see the formal proofs and simulations, I can recommend:

$^{*}$ Ensemble average: $\frac12\times0.6+\frac12\times1.5=1.05$ a number larger than one, reflecting positive growth of the ensemble.

$^{**}$ Time average: $x(t)=r_1^{n_1}r_2^{n_2}$ where $r_1$ and $r_2$ are the two rates and $n_1$ and $n_2$ are the frequences that the wealth process gets subjected to the rates. The limit of $x(t)$ is then for $t\rightarrow\infty$ $x(t)^{1/t} = (r_1r_2)^{1/2}$ or $\sqrt(0.9)\approx0.95$ e.g. a number less than one, which is decay in the long time limit.

Video

Article

tmo
  • 221
  • Thanks, but I think you're making some sort of mistake here. First. Quoting you: "for a given dynamic or process all realisations will have the same time average in the limit". I still think this is wrong. Clearly we can fabricate random processes that will converge to different values in different outcomes. I simulated your own coin toss example 1000 times, each with 20000 coin tosses, and computed those 1000 time averages, getting results as low as 0.008333105152477038 and as high as 42.99348110477372. – Pedro A Nov 22 '16 at 16:24
  • Second. Quoting you: "The ensemble average will predict infinite positive growth" - my simulation does not confirm this. Rather, the ensemble average from the 10000th coin toss onwards yield 0 (the limit of computer precision was reached), so it is far from infinity (as far as you could get). – Pedro A Nov 22 '16 at 16:26
  • Oh, and the St. Petersburg paradox is very interesting, thanks for showing it to me; but I don't really see how it is related, because it can be seen simply as a random event, not a process, i.e., every time you attempt the game, you have one specific result; if you count each coin toss as a 'step', still the game ends in a specific moment (as soon as you get tails); while in a random process (such as your example), we choose when to stop looking, (as I did in my simulations, choosing to make 20000 coin tosses). – Pedro A Nov 22 '16 at 16:38
  • First: I can't seem to find the sentence "for a given dynamic or process all realisations will have the same time average in the limit"? You are right in saying that we can fabricate dynamical processes where the time-average is different. To be more precise, for this specific purely multiplicative dynamic, all the N processes will converge to the absorbing boundary (zero) with probability one in the limit. This is easy to show analytically. – tmo Nov 23 '16 at 09:01
  • Second: Your simulating is misleading you a little bit. It is too small. It is easy to show analytically that the ensemble average will predict infinite growth: Simply take the expectation to the specified dynamics and you will obtain a growth rate larger than one. Importantly, this is wrong. This is a result of non-ergodicity. I.e. the time-average does not equal the ensemble average. If you're keen on letting simulation guide you, which I think is a good idea, increase the ensemble (I did it on a million) and plot the distribution as it changes in time. – tmo Nov 23 '16 at 09:06
  • St. Petersburg: Is valid here because it asks the question: I am offering you a gamble with expected* infinite value for price P. Will you take it?* Anyone using the expectation operator would see that this gamble does indeed predict infinite positive value, and should pay the price P if it is finite. However, anyone sane would never take it, because they realise the very large probability of entering the absorbing boundary. Essentially, two limits are competing here. The exponential growth of wealth and the polynomial decrease of a conditional probability. – tmo Nov 23 '16 at 09:11
  • Thanks for the replies. You said it is easy to show analytically that the ensemble average will predict infinite growth. Can you add this proof to your answer please? It did not appear easily to me. No matter what I do, I find a growth rate lower than one, not higher than one, as you said... EDIT: whoops I just saw that you added some links to your answer. Sorry about that. I will look at them later today, and will also review my simulations. I'll reply again later today/tomorrow, thank you very much for being willing to help! – Pedro A Nov 23 '16 at 10:01
  • No problem. Also notice that I corrected the gamble a while back. From 0.4 to 0.6 in the 'loss' condition. That would def. change your simulations! Feel free to accept as answer if you feel satisfied. – tmo Nov 24 '16 at 09:04
  • @tmo The link to the article no longer works. Could you Kindly fix? Thanks :) – Tom Oct 23 '18 at 09:58
  • @Tom. Done! Apologies for the delay, only just stumbled over your comment now. – tmo Dec 20 '18 at 08:08
  • @tmo One bit I'm stuck on: the time average gives you a distribution. What measure / statistic on that distribution defines the "time average"? Obviously not the mean / expectation, because that gives you the ensemble average. It's something like "the value around which the mass concentrates arbitrarily close to one" or something, but that's never explicitly stated. Is it because it's obvious? Or is that given as part of the standard definition somewhere? – A_P Dec 10 '24 at 17:05
  • @A_P the time average is defined above, at the end of the post. The reason it's useful to think about the time average as a distribution is that its the average value the process takes over time, instead of the average value of the process at a given time. – tmo Dec 12 '24 at 08:59
  • @tmo Thanks. I see how to calculate the time average, just not how to define it. It seems to be something like "the value to which a single process converges almost surely"? Is that ever explicitly stated? Otherwise, "time average" could easily be interpreted as "expectation." – A_P Dec 13 '24 at 17:25