How many tosses for 95% centainty that coin is not fair

Question

Given a bag of 10 coins, 9 are ordinary coins and one is a double headed coin. You select one coin at random and toss it three times. It comes up heads each time what is the probability its the double header?

This can be solved using bayes rule the answer is $\frac{8}{17}$.

However the follow up question asks how many tosses would you need to be 95% sure that the coin is double headed?

Based on Mark Galek's answer here

If I flip a coin 1000 times in a row and it lands on heads all 1000 times, what is the probability that it's an unfair coin?

The probability of getting a head n times is:

$(\frac{1}{2})^n$

and I need to find the value of n that gives us 95% confidence that the coin is not fair.

$(\frac{1}{2})^n=0.05$

$n=ln(0.05)/ln(\frac{1}{2})$

n=5

Is this correct my gut instinct is that it is a little low?

Here I have used the CI for the fair coin to try and find how many heads would be required for us to be 95% sure that it isn't fair. Given that the only other choice here is for the coin to be double headed I believe this is correct.

Is there any other way this calculation could be done without using the CI for the fair coin?

Answer 1

As pointed out at Number of tosses to ensure $95\%$ that the coin selected is double-headed, the existing answer to this question is incorrect, as it doesn't take into account the given information on how the coin was chosen.

The existing answer answers the question how many times you'd have to throw heads in a row so that the probability for a fair coin to produce that result would be less than $5\%$. I understand the question to ask instead how many times you'd have to throw heads in a row so that the conditional probability for the coin to be fair, given how it was chosen, would be less than $5\%$.

As calculated by @Idonknow in the question linked to above, the answer is $8$. The conditional probability for the coin to be double-headed after $n$ heads in a row is

\begin{eqnarray*} P(\text{double}\mid\text{$n$ heads}) &=& \frac{P(\text{double}\land\text{$n$ heads})}{P(\text{$n$ heads})} \\ &=& \frac{P(\text{$n$ heads}\mid\text{double})P(\text{double})}{P(\text{$n$ heads})}\;. \\ &=& \frac{1\cdot\frac1{10}}{1\cdot\frac1{10}+\left(\frac12\right)^n\cdot\frac9{10}} \\ &=& \frac1{1+9\cdot2^{-n}}\;. \end{eqnarray*}

For this to be $\ge95\%$, we need $1+9\cdot2^{-n}\le\frac1{0.95}$ and thus

$$ n\ge\log_2\frac9{\frac1{0.95}-1}=\log_2171\approx7.4\;, $$

so we need $8$ consecutive heads to be $95\%$ sure that the selected coin is the double-headed coin.

Note that the number of fair coins appears in the numerator of the argument of the logarithm. That means that every time we double the number of fair coins, we need one more throw of heads to reach the same level of certainty. For instance, if we had $4\cdot9=36$ fair coins and one double-headed one to choose from, we'd need $10$ consecutive throws of heads to reach $95\%$ certainty that we chose the double-headed one.

Answer 2

Let's think of this in very simple terms. Initially operating under the assumption that a coin is fair, if we flip it $n$ times, there is still a probability of $1/2^n$ that it lands all heads, and $1/2^n$ that it lands all tails, purely by random chance. For example, for $n = 5$ flips, you could get such extreme results as often as $1$ in $16$ such experiments!

So with such a few number of flips, there is no test you could perform to give you $95\%$ confidence that the coin is not fair if in fact it is not fair, because the most discriminating rejection criterion that all flips must be heads or all flips must be tails, cannot with such few flips, tell you with better than a $100(1 - \frac{1}{16})\% = 93\%$ confidence that the coin is in fact not fair.

This does, however, suggest that $n = 6$ might work. In this case, your test to determine unfairness is again that all outcomes are the same (all heads, or all tails), but here the probability of this event is $(1/2)^6 + (1/2)^6 = (1/2)^5 = 1/32$, and such a test has at most a probability of $0.03125$ of being wrong about a fair coin: that is to say, you don't know if the coin is fair or not; but if the coin were fair, and you flipped it six times and got all heads, or all tails--purely by random chance--and as a result concluded it was unfair, such a fluke could happen only $3.125\%$ of the time.

Consequently, the confidence level of such a test is $100(1-0.03125)\% = 96.875\%$. So we need to flip a coin at least six times to be at least $95\%$ confident.

---

It is worth mentioning that five flips isn't good enough, but six flips puts us over $95\%$ confidence. This is because of the discrete nature of the binomial distribution. We could get closer to an actual confidence of $95\%$ if we were willing to flip the coin more times and relax the test criteria. For example, suppose you were to flip the coin $n = 17$ times, and you would call the coin unfair if you do not observe at least $5$ heads and $5$ tails. This test would have a confidence level of $100(1-\frac{1607}{32768})\% = 95.0958\%$. But of course, such a test requires more effort--$17$ flips instead of $6$.