Data Science Stack Exchange
Data Science Stack Exchange

Questions tagged [hypothesis-testing]

50 questions
8
votes
2 answers

A/B testing: How to calculate p-value on post test segments?

My question on A/B testing is about doing post test segmentation analysis. For example: I run an A/B test on my website to track bounce rate. On the treatment group, i put a video to explain my company. On the control group i put just plain…
jxn
  • 233
  • 2
  • 5
7
votes
2 answers

When to use which multiple testing correction?

There are a large number multiple testing p-value correction methods. e.g.: bonferroni : one-step correction sidak : one-step correction holm-sidak : step down method using Sidak adjustments holm : step-down method using Bonferroni…
lordy
  • 314
  • 2
  • 12
4
votes
2 answers

Which statistical test should I use to show significance in favour of a system?

I have a dataset of the following form: System A Rating System B Rating 4.5 5 3 4 5 3 etc. etc. I have 155 such data points gathered using a survey. Which statistical test should I use to show statistical significance if one system…
Michael Pulis
  • 103
  • 1
  • 9
4
votes
1 answer

How do I know that model performance improvement is significant?

Say I am running a Machine Learning model that produces a certain result (say accuracy of 80%). I now change a minor detail in my model (say, in a Deep Learning model, increase the kernel size in one convolutional layer) and run the model again,…
emilaz
  • 141
  • 4
4
votes
3 answers

How do you use KS-test in a data science report?

I'm writing a data science report, I want to find an exist distribution to fit the sample. I got a good looking result , but when I use KS-test to test the model, I got a low p-value,1.2e-4, definitely I should reject the model. I mean, whatever…
Carl
  • 51
  • 2
3
votes
1 answer

Applying Hypothesis testing on single sample and non-normal distribution

A basketball game is played for 30 minutes. A coach claims that his team's players commit, on average, no more than 10 fouls per game. Let µ represent the team's average number of fouls per game. Another coach thinks that these players create more…
Kapil
  • 131
  • 3
2
votes
1 answer

Check if distribution per week is the same

I have sales by customer (b2b) and by date. I want to check if the distribution per day inside weeks remains the same from week to week. Initial…
Ismail
  • 121
  • 2
2
votes
1 answer

Why rejection of a true null hypothesis is called type I error?

I’m comparing two confusion matrices: https://en.wikipedia.org/wiki/Confusion_matrix#Table_of_confusion https://en.wikipedia.org/wiki/Type_I_and_type_II_errors The 2nd is rotated, the Decision is on Y-axis. But I assume both reflect the same…
belz
  • 31
  • 2
2
votes
0 answers

Comparing two multi-class machine learning classifiers using Stuart Maxwell Test

I need to compare 2 multi-class classifiers. So, to assess whether the difference between the two are statistically significant I have taken the following steps: obtain prediction on test data using model 1 obtain prediction on test data using…
ILR
  • 31
  • 2
2
votes
0 answers

How to apply Hypothesis test on time series?

I am working on a project that is related to Wikipedia's revision history. I have around 10 time-series and each one represents how the relative frequency of positive words changes over time. I need to test the hypothesis, that the batch of females…
Vangelis
  • 21
  • 2
2
votes
1 answer

Interpreting the results based on Granger Causality test

I am trying to use Granger Causality test: https://www.statsmodels.org/stable/generated/statsmodels.tsa.stattools.grangercausalitytests.html to assess whether "positivity score" affects value. Here is the code I am using: # Applying…
Darcey BM
  • 197
  • 1
  • 6
2
votes
1 answer

Permutation test on two groups

I am trying to use a permutation test to test my hypothesis. I want to make sure I am understanding concept of permutation correctly. I have control and experimental group. Then I combine them and resample from combined dataset randomly calculating…
haneulkim
  • 479
  • 5
  • 17
2
votes
1 answer

Multiple Hypothesis Testing in feature selection process

I am doing feature selection of features which are of binary nature i.e. each feature represents presence or absence of a substructure in a molecule. And I have a target variable of two classes. My first step was to check if my feature is…
SS16
  • 77
  • 4
2
votes
1 answer

Which statistical procedure to use?

I'm new to Hypothesis testing. I can't really think of how to answer this - Five students take a test one year and another test the next year. Their overall grades(in %) are given below for the test-1 and test-2. Which statistical procedure (e.g.…
a_jelly_fish
  • 227
  • 2
  • 10
2
votes
1 answer

What can we do in order to find customers that follow a specific pre-defined pattern?

Let's say I have a few customers buying product A. How can I find other customers that have the same characteristics, or something close to, these A buyers? I was thinking about using clustering methods, but I don't know if it's possible to "lock"…
Aldla E Aoepql
  • 71
  • 3
1
2 3 4