Advanced experimentation beyond classic A/B
An experimentation programme combining classic A/B, multivariate and multi-armed bandits to extract statistical signal even when traffic is limited or the incremental is small.
The problem we solve.
Classic A/B is a good hammer, but not every problem is a nail. When there are many variants, when the expected incremental is small, or when traffic won't sustain two-month tests, you need something else.
We design experimentation programmes that apply the right tool to each case: classic A/B, factorial multivariate, multi-armed bandits for continuous optimisation, or interleaving when what you're comparing are rankings.
What is included
- Classic A/B testing with sample calculation and stopping rules
- Factorial multivariate tests (MVT) to evaluate combinations
- Multi-armed bandits for continuous optimisation without waiting
- Experiment design with guardrail and holdout metrics
- Reporting with confidence intervals, not misleading percentages
Our process for Experimentation and A/B testing.
Diagnosis
We assess your traffic volume, base rate and hypothesis backlog to choose the right method for each test.
Design
Experiment setup, primary and secondary metrics, and pre-flight checks before launching.
Analysis
Honest interpretation of results, roll-out or roll-back decision and documentation of the learning.
When MVT or bandits beat A/B
MVT: when you want to evaluate 2-3 independent changes at once (CTA + headline + image) and discover interactions between variables without running 8 consecutive tests.
Bandits: when there's a real opportunity cost in exposing traffic to the losing variant for weeks. Bandits learn and reallocate traffic towards the best-performing variants, at the cost of less statistical power per variant.