3

I have results from an A/B experiment where users could do one of three things: Watch, Interact, or Nothing

My data is like this: 

   Watch  | Nothing | Interact
A: 327445 | 271602  | 744702
B: 376455 | 140737  | 818204

I tried to use the chisquared test bundled with scipy. I'm completely new to data science, but I believe this is the evaluation metric I would want.

scipy.stats.chisquare([ [327445, 271602, 744702], [376455, 140737, 818204] ])
Power_divergenceResult(statistic=array([  3412.38826538,  41532.93339946,   3456.72996585]), pvalue=array([ 0.,  0.,  0.]))

This does not look like a valid result...

I even tried adding an expected frequencies options, but without success. Maybe I'm missing something, either about evaluating this type of data or just not using scipy correctly. Can anyone help me?

Dawny33
  • 8,476
  • 12
  • 49
  • 106
OneChillDude
  • 133
  • 1
  • 5

1 Answers1

2

If what you're trying to answer is if the action taken by a user (watch, interact, nothing) is influenced by the group they are in (A or B) you can use the chi2 test of independence.

For that you can use the scipy.stats.chi2_contingency function:

a = [327445, 271602, 744702]
b = [376455, 140737, 818204]

chi2, pvalue, _, _ = scipy.stats.chi2_contingency([a, b])

In this case it returns a chi2 test statistic of 48376.48 and a p-value of 0.0, so the null hypothesis ("action is independent of group") is rejected.

You can also use the scipy.stats.chisquare function to arrive at the same results, but other than with the chi2_contingency function you'll have to calculate the "expected frequencies" yourself. The data you have recorded are your observed frequencies:

obs = np.array([a, b]).astype(float)

(Note that I converted the numbers to float, as the chisquare function will run into some weird integer overflows otherwise...!?)

The expected frequencies are calculated like that:

exp = np.outer(obs.sum(axis=1), obs.sum(axis=0)) / obs.sum()

Finally, calling

chi2, pvalue = scipy.stats.chisquare(obs.ravel(), exp.ravel(), ddof=sum(obs.shape)-2)

returns the same chi2 test statistic and p-value as before.

stmax
  • 1,647
  • 15
  • 18