7

For my thesis, I have created a development approach (consisting of schemes, an application template and prototype code) that should make it easier to develop and adapt applications for a specific platform.

I have conducted interviews with developers of that platform to evaluate my approach. There have been 5 interviewees so far and the feedback that I got from them is sufficient for my scope, I would say.

However, in my thesis I would like to justify my decision that 5 evaluators are enough. Is there any scientific research or paper that suggests a certain number of evaluators for such a rather theoretical approach?

I know Nielsen's work about heuristic usability evaluation which says that with 5 participants, you discover around 75% of all problems in software usability / UI testing.

But as I said, my approach is no specific software product that has to be evaluated, but only the model itself, which is why I'm looking for research in that area.

saschoar
  • 171
  • 3

1 Answers1

5

"Is there any scientific research or paper that suggests a certain number of evaluators for such a rather theoretical approach?"

There is a statistical method to calculate the correct sample size for an experiment.
This method is called a "power analysis" [1]. It calculates the minimum sample size needed to verify that an experiment's results are statistically significant. This approach assumes that you have at least two groups. A control group and an experimental group (i.e. control group uses the platform without your modifications, experimental group uses the platform with your modifications).

Thus, while there is a scientific process to calculate the correct sample size this approach is probably overkill for your study. Nielsen [2] states that heuristic evaluation is just a "cheap and quick" method for easy evaluation. As such, I believe you can justify a sample size of 5 by citing Nielsen. However, if you do want to standardize your evaluation process and use a statistical approach I encourage you to look into power analysis.

You may be interested in this online tool for calculating power.

  1. http://en.wikipedia.org/wiki/Statistical_power
  2. http://www.nngroup.com/topic/heuristic-evaluation/
Raphael
  • 73,212
  • 30
  • 182
  • 400
Camille
  • 84
  • 1
  • 3