1

I'm sure this is basic statistics, but I'm not even sure of the right words for the concepts I'm dealing with. I took statistics in college but it was ... well ... a few years ago.

Let's say there's an event that happens randomly every X% of the time. If I run my test Y times, there's a nice bell curve I can generate that shows the likelihood of the event happening 1, 2, ... Y times during my Y tests.

I'm actually working on the inverse of this - let's say I don't know X%, but I know I ran my test Y times and the event happened W times. I can't say for sure what X% is, but it seems like I should be able to say "there's a 90% likelihood that X is in the range X1 to X2" or something like that.

Betty Crokker
  • 113
  • Look up Chi squared and t tests for background. – David G. Stork Jun 04 '19 at 18:07
  • Welcome to Mathematics Stack Exchange! A quick tour will enhance your experience. Here are helpful tips to write a good question and write a good answer. For equations, please use MathJax. – dantopa Jun 04 '19 at 18:07
  • 1
    Alas, given my lack of knowledge of statistical terminology, there's not much I can do with statements like "The Chi-Square statistic is most commonly used to evaluate Tests of Independence when using a crosstabulation (also known as a bivariate table). " – Betty Crokker Jun 04 '19 at 18:08
  • But I also see "The chi-squared test is used to determine whether there is a significant difference between the expected frequencies and the observed frequencies in one or more categories. " and that's not what I'm doing, since I don't have an "expected frequency" – Betty Crokker Jun 04 '19 at 18:11

1 Answers1

2

Suppose you see your 'event' happen $X = 35$ times in $n = 100$ independent performances of your procedure. (Jargon: 35 "Successes" in $n$ trials.)

Then $\hat p = X/n = 0.35$ is the estimate of the probability of occurrence. A traditional 'Wald 95% confidence interval' for the true (population) value of $p$ is of the form

$$\hat p \pm 1.96\sqrt{\frac{\hat p(1-\hat p)}{n}}.$$

In my example: this computes to $0.35 \pm 0.0935$ or $(0.2565, 0.4435).$

This interval is based on two assumptions (1) that $n$ is large enough for binomial proportion $\hat p$ to be nearly normal (your "nice bell-shaped" distribution) and (2) that $\hat p$ is reasonably close to $p$.

The latter assumption says it is OK to use $\hat p$ instead of $p$ in the term with the square root. Experience has shown that the combination of these two assumptions often results in confidence intervals that do not really cover (include) the true value of $p$ 95% of the time.

An improved style of confidence interval due to Agresti and Coull. Essentially to avoid the second assumption, it uses $\check p = (X+2)/(n+4)$ and $\check n = n+4$ to make the more accurate 95% interval

$$\check p \pm 1.96\sqrt{\frac{\check p(1-\check p)}{\check n}}.$$

If $n$ exceeds a few hundred, this adjustment (in effect, appending two successes and two failures to the data) is usually too small to be of practical consequence.

In my example, the resulting 95% confidence interval is: $0.3558 \pm 0.0920$ or $(0.2638. 0.4478).$

Notes: (1) If you want a confidence level other than 95%, use an appropriate replacement for 1.96 [1.96 cuts 2.5% of the probability from the upper tail of a standard normal distribution.] (2) On this site or on the Internet you can get more background and details by searching for 'binomial confidence interval', 'Wald confidence interval', and 'Agresti-Coull confidence interval'. Here is one possibly interesting link.

BruceET
  • 52,418
  • Awesome, thank you! What if I have 0 successes in n trials (so X = 0)? – Betty Crokker Jun 04 '19 at 23:43
  • Then Wald CI degenerates to a single point at $0$ because $\hat p = 0.$ But, say for $n = 100,$ the A-C CI computes to $(-0.007, 0.046),$ which is read as $(0, 0.46)$ to avoid nonsensical negative values. For such an extreme case, some statisticians would use a Bayesian 'probability interval' . Others would make a one-sided interval trying to guess how large the true $p$ might be. // What you can't do is claim to have proved that 'successes' are impossible. // Situations like this do arise, for example in trying to assess if a huge herd of cattle is totally free of hoof-and-mouth disease. – BruceET Jun 05 '19 at 00:03