4

How can one use memorylessness and the law of total expectation and the law of total variance to find the expectation and variance of the geometric distribution?

I will post my own answer, but as always, that shouldn't stop anyone else from posting theirs.

Rodrigo de Azevedo
  • 23,223
Michael Hardy
  • 1

2 Answers2

4

$\newcommand{\var}{\operatorname{var}}$ $\newcommand{\E}{\mathbb E}$

I will consider the geometric distribution supported on the set $\{0,1,2,3,\ldots\}$. This is the distribution of the number $X$ of failures before the first success in a sequence of independent Bernoulli trials. Call the probability of success on each trial $p$.

Then $$ X = \begin{cases} 0 & \text{with probability }p \\ 1 & \text{with probability }p(1-p) \\ 2 & \text{with probability }p(1-p)^2 \\ 3 & \text{with probability }p(1-p)^3 \\ \vdots & {}\qquad \vdots \end{cases} $$

Memorylessness of this distribution means that $\Pr(X\ge w+x\mid X\ge w)=\Pr(X\ge x)$, i.e. the probability distribution of the number of remaining trials, given the number of failures so far, does not depend on the number of failures so far.

Let $\displaystyle A=\begin{cases} 1, & \text{if }X\ge 1 \\[6pt] 0, & \text{if }X=0. \end{cases}$

Then $$ \E(X) = E(E(X\mid A)) = E\left.\begin{cases} 0 & \text{if }A=0 \\ 1+\E(X) & \text{if }A=1 \end{cases}\right\} = p\cdot0+(1-p)(1+\E(X)). $$ Thus we have $$ \E(X) = 1-p+(1-p)\E(X). $$ Therefore $$ \E(X) = \frac{1-p}{p}. $$

Now the variance: $$ \var(X) = \var(\E(X\mid A)) + \E(\var(X\mid A)) $$ $$ = \var\left.\begin{cases} 0 & \text{if }A=0 \\ 1 + \E(X) & \text{if }A=1 \end{cases}\right\} + \E\left.\begin{cases} 0 & \text{if }A=0 \\ \var(X) & \text{if }A=1 \end{cases}\right\} $$ $$ = \var\left.\begin{cases} 0 & \text{if }A=0 \\ 1/p & \text{if }A=1 \end{cases}\right\} + p\cdot0 + (1-p)\var(X) $$ $$ = \frac{1-p}{p} + p\cdot0 + (1-p)\var(X) = (1-p)\left(\frac1p+\var(X)\right). $$ So we get $$ \var(X) = (1-p)\left(\frac1p+\var(X)\right). $$ Therefore $$ \var(X) = \frac{1-p}{p^2}. $$

Michael Hardy
  • 1
  • How did you combine the fact that variance = 0 if A = 0 and variance = 1/p if A = 1 to get $\frac{(1-p)}{p}$? – 1110101001 Jun 11 '15 at 08:33
  • 1
    @1110101001 : Just after the word "Then", we have $\operatorname{E}(X) = \cdots = \cdots$, and there appears a sum of two terms, but one of those two terms is $0$. So we get $\operatorname{E}(X) = (1-p)(1+\operatorname{E}(X))$. That expands to $(1-p)+(1-p)\operatorname{E}(X)$. We now have $\operatorname{E}(X) = (1-p)+(1-p)\operatorname{E}(X)$, Subtracting that last term from both sides, we get $\operatorname{E}(X) - (1-p)\operatorname{E}(X) = (1-p)$. The left side of that equation simplifies to $p\operatorname{E}(X)$. Then we have $p\operatorname{E}(X)=1-p$. ${}\qquad{}$ – Michael Hardy Jun 11 '15 at 14:11
  • I was referring to the derivation for the variance, not the expected value -- you wrote $$ = \var\left.\begin{cases} 0 & \text{if }A=0 \ 1/p & \text{if }A=1 \end{cases}\right} + p\cdot0 + (1-p)\var(X) $$

    How did you combine the two cases of variance into $\frac{1-p}{p}$

     – 1110101001 Jun 11 '15 at 18:34
  • 1
    @1110101001 : Suppose $\displaystyle W=\left.\begin{cases} a & \text{with probability }p, \ b & \text{with probability }1-p. \end{cases}\right}$ Then $\operatorname{var}(W)=p(1-p)(a-b)^2$. Apply that to this instance. To see that that's what it is, do it first in the case where $a$ and $b$ are $0$ and $1$ (it doesn't matter which is which). Then recall that variance doesn't change if you add a constant to a random variable, and recall why happens to the variance when you multiply the random variable by a constant. ${}\qquad{}$ – Michael Hardy Jun 11 '15 at 19:39
3

Memorylessness means that either $X=0$, which happens with probability $p$, or that, with probability $1-p$, $X=1+X'$ where $X'$ has the same distribution as $X$. That is,

$$ X=U\cdot(1+X'),\qquad U\sim\mathrm{Ber}(1-p),\qquad U\ \text{independent of}\ X'. $$

This yields every moment of $X$, for example, $E[U]=1-p$ hence $$ E[X]=E[U]\cdot(1+E[X])\implies E[X]=\frac{E[U]}{1-E[U]}=\frac{1-p}p. $$ Likewise, $$ E[X^2]=E[U^2]\cdot E[(1+X)^2]=E[U]\cdot(1+2E[X]+E[X^2]), $$ which implies $$ E[X^2]=\frac{E[U]}{1-E[U]}\cdot\frac{1+E[U]}{1-E[U]}, $$ and $$ \mathrm{var}(X)=\frac{E[U]}{(1-E[U])^2}=\frac{1-p}{p^2}. $$ More generally, for every $x$ in $(0,1]$, $$ E[s^X]=E[s^{U(1+X)}]=p+(1-p)E[s^{1+X}]\implies E[s^X]=\frac{p}{1-(1-p)s}, $$ hence, differentiating $n$ times, $$ E[X(X-1)\cdots(X-n+1)s^X]=n!\,\frac{p(1-p)^n}{(1-(1-p)s)^{n+1}}, $$ and in particular, for $s=1$, $$ E[X(X-1)\cdots(X-n+1)]=n!\,\frac{(1-p)^n}{p^n}, $$ from which every moment of $X$ can be deduced.

Did
  • 284,245