Efficient traffic allocation

Question

Users can be assigned to one experiment on my site. I have an API that developers use to trigger logic for each experiment. They call ExperimentEngine.run() to trigger the code logic for the experiment.

I would like to allocate users to each experiment, at the point where a user might be exposed to the logic for that experiment. I would like to assign users to experiments such that experiments implemented downstream (on pages that are usually seen last) still get users assigned to them.

For example, if user A is runs into the code logic for experiment A at login and then goes to page B and runs into experiment B, the user A could be assigned to either experiment A or B. That means that they will only see one of the experiments and not both (either A or B) or neither. I would like to figure out the right algorithm so that experiment B (which is downstream and shown to the user after they've seen experiment A) gets users assigned to it. I don't to assign all users to experiment A.

So the flow is as follow

• User visits page A where experiment A is implemented
• We decide whether to assign users to experiment A. If user is assigned to A, user will be able to see experiment A.
• User visits page B where experiment B is implemented, we decide whether to assign users to experiment B
• Users can only see experiments that they are assigned to.
• I want to come up with an algorithm that allows me to assign users to experiments regardless of where on the site they are implemented so that the traffic distribution is efficient and experiments implemented downstream get enough users.

Can someone please point me in the right direction to an algorithm that allows me to do the above?

Each experiment is given X number of people needed per day to reach stats sig. We could factor that into the algorithm to make sure that we assign users to experiments in a way that experiments reach their sample size in two weeks.

A possible algorithm:

• For each experiment, we make a decision of whether to assign based on the experiment's location using coin flip.
• If we get heads, a list of experiments that are implemented for that location are selected.
• An experiment is chosen from that list based on priority system. At every location, a % of users are assigned to one of the experiments implemented at that location.
• When we decide to assign or not to assign to any experiments at that location, that decision is not made again for the user.

The Question: How do I assign users to one experiment at a time so that experiments that they might see last get enough users? Only users assigned to experiments can see them.

Answer 1

I suggest you try a simple solution: randomly decide whether to assign users to an experiment when they visit the corresponding page.

Suppose you want to have 100 users to a particular experiment, call it experiment E. When a user visits the page for experiment E, if that user hasn't been previously assigned to any experiment, flip a biased coin and assign the user to experiment E if that coin comes up heads. How do you decide what the probability of heads should be? If historically you see 2000 visitors per two-week period, and you want 100 users, then the heads probability should be about 1/20. Then after two weeks you should get about the desired number of users.

This assumes you have historical data on the number of visitors to each page, to set the probability appropriately. If you don't have historical data, there are some alternative strategies you could use. For instance, if you want 100 users for experiment E, you could randomly pick 100 different times throughout the two-week period; for each selected time, take the next unassigned user who visits that page (the first one who visits the page after that time) and assign them to experiment E. In this way you are randomly selecting 100 different users.

These techniques treat each experiment independently, which simplifies the assignment process. This is simple and I suspect it might work well enough for running experiments in practice. Only consider something more complicated if you have evidence that this won't work. (How can you tell whether it will work? You could take your historical data and simulate the strategy on historical data and see if it gets you enough users for each experiment.)