Illustration of two overlapping bell curves with a marker showing the smallest difference worth detecting

Minimum Detectable Effect in Optimizely Web Experimentation

First published in October 2023. Restored in October 2026, with notes on anything that has changed since.

Understanding Minimum Detectable Effect

Minimum Detectable Effect (MDE) is a statistical calculation in Optimizely Web Experimentation (opens in a new tab). In simple terms, it represents the smallest relative change in conversion rate that warrants detection. To illustrate, if your baseline conversion rate is 20%, and you set an MDE of 10%, your experiment will identify any conversion rate shifts beyond the absolute range of 18% to 22%. This translates to a 10% relative effect, equivalent to a 2% absolute change in conversion rate.

Optimizely Web Experimentation Minimum Detectable Effect

Minimum Detectable Effect serves various essential purposes within Optimizely Web Experimentation, playing a key role in different areas of experimentation planning and design:

Experiment Sensitivity with Minimum Detectable Effect

Minimum Detectable Effect quantifies an experiment’s sensitivity, determining its ability to detect meaningful changes. By setting an appropriate MDE, you establish a threshold for recognising significant effects and avoid chasing insignificant fluctuations.

Evaluating Costs with Minimum Detectable Effect

MDE plays a pivotal role in cost considerations for experiments. It helps assess whether the expected outcomes justify the investment in terms of traffic volume, time, and resources. Realistic MDE values enable a clear assessment of cost-effectiveness, as it is an input for determining the sample size per variation, thus how long an experiment will need to run.

Experiment Design Considerations with MDE

Experiment design and scope are directly influenced by MDE. It guides decisions about variations, their number, and the extent of differences required for achieving statistical significance (opens in a new tab). Experimenters can use MDE to fine-tune their experiments efficiently. For example, if you want to detect a large effect within an experiment, then you should consider making a large change to a variation to accommodate this.

Using Minimum Detectable Effect for Estimation

MDE aids in estimating experiment duration by combining additional parameters like baseline conversion rate and statistical significance level. Optimizely’s Sample Size Calculator (opens in a new tab) facilitates this calculation, aiding experimenters in planning effectively by knowing how long an experiment will take to run, based on the calculated required number of visitors per variation.

Experiment Prioritisation with MDE

MDE is instrumental in experiment planning decisions. It helps prioritise experiments by assessing their expected impact and sensitivity. Therefore, when planning your experimentation roadmap, you may want to order your experiments from highest MDE to lowest, so that your most impactful experiments are executed first. This strategic approach ensures that high-impact experiments receive due attention.

Minimum Detectable Effect Best Practice

Effective use of Minimum Detectable Effect in Optimizely Web Experimentation involves adhering to best practices:

Consider Business Impact: Evaluate an experiment’s potential business impact when setting the MDE. Revenue-focused experiments may justify a lower MDE, even with higher traffic requirements.

MDE as a Guide: View MDE as a guiding principle rather than an exact prediction. It helps set boundaries on time investment and expected outcomes.

Focus on Impactful Variations: When dealing with traffic limitations, concentrate variation designs on elements directly affecting the primary conversion event. This focused approach maximises the chances of obtaining meaningful results.

In essence, MDE in Optimizely Web Experimentation is a valuable tool that helps organisations set realistic boundaries for their experiments, ensuring that the investment in terms of traffic, time, and resources is justified by the expected impact and sensitivity of the experiment.

Frequently Asked Questions

What is minimum detectable effect in Optimizely Web Experimentation?

Minimum detectable effect (MDE) is the smallest relative change in conversion rate that an experiment is designed to detect. It guides decisions on experiment design, cost and how much traffic a test needs.

How does minimum detectable effect work in practice?

If your baseline conversion rate is 20% and you set an MDE of 10%, the experiment is sized to detect changes outside the range of 18% to 22%. That is a 10% relative change, or 2 percentage points.

How does MDE affect how long an experiment runs?

MDE is an input to the sample size needed for each variation, along with the baseline conversion rate and the significance level. The smaller the MDE, the more visitors you need and the longer the test runs. Optimizely's Sample Size Calculator works this out.

How should MDE shape the design of an experiment?

If you want to detect a large effect, make a large change to the variation. When traffic is limited, focus variations on the elements that most directly affect the primary conversion.

How can I use MDE to prioritise experiments?

Order your roadmap from the highest MDE to the lowest, so your most impactful experiments run first. Revenue-focused experiments may still justify a lower MDE, even though they need more traffic.