Experiment Design Doc
beta · Family experimentation-docs · Phase measure · Sizes lean, full · ~3,500 tokens
The document a team writes before a product experiment launches, while nobody has seen any data, to fix the hypothesis, the variants and allocation, the primary and guardrail metrics, the minimum detectable effect and sample size or duration, and the decision rule for every outcome including a null one. It leads with what a PRD’s Success metrics section does not already carry, not a restatement of it.
The short card. Why the document is shaped this way, and the argument behind every rule here, is in
experiment-design-doc_companion.md. A fully worked instance is
experiment-design-doc_example.md.
One honest limit before anything else. No source read for this bundle measured whether writing this document improves an experiment’s outcome. The case for it rests on practitioner testimony and on the statistics of what goes wrong without a pre-committed stopping rule, not on a controlled comparison of documented against undocumented tests. Read the rest of this card with that limit in mind.
When to use
Section titled “When to use”- A product change is about to be tested against a baseline with real traffic, and the decisions that follow the result are expensive enough to fix in advance rather than argue about after the data arrives.
- More than one person needs to agree, before anyone has seen any data, on what counts as a win, a null result, and a loss.
- The surface is mature, or the blast radius of a mistake on it is large enough that a pre-committed stopping rule and a shutdown threshold are worth the extra page.
- The team runs on a shared experimentation platform whose statistical framework this document can simply name, rather than argue out per test.
When NOT to use
Section titled “When NOT to use”- You will ship the change regardless of what the test shows. Running a controlled test to confirm a decision already made spends real complexity on nothing. Skip the experiment.
- The population available is too small to produce a result anyone could act on. A low-traffic surface cannot reach a trustworthy effect size no matter how carefully this document is written. Pick a cheaper validation method instead.
- A cheaper method, such as a user interview or a smoke test, would already answer the question. Fixating on controlled experiments for every idea sets the bar too high for the easy cases.
- You need a qa-docs test plan. A test plan verifies a product increment against an agreed specification and grades it pass or fail. It carries no hypothesis, no allocation between variants, and no statistical decision rule. This document tests an open hypothesis and estimates an effect against a baseline. See companion section 8 for the boundary in full.
- You need a spike report (decision-docs). A spike reduces uncertainty about a technical or design question, with no control group and no statistical decision rule; a refuted hypothesis there is a successful spike. That is never true of a result reported here. See companion section 8.
- Nothing is built yet, and the question is only whether to build it. A test run before anything exists, to decide whether to build it at all, is excluded from this family by name; it is not this document’s job under any variant.
- You need a standing instrument, not a one-time record. A tracking plan, a data dictionary, or an experiment log is maintained across many tests and is never finished. This document is written once per test and is finished when the test’s own decision is recorded. See companion section 8.
Pick a variant
Section titled “Pick a variant”Lean (five sections) is the default: Hypothesis, Population/Variants/Allocation, Primary and Guardrail Metrics, Minimum Detectable Effect and Sample Size or Duration, and Decision Rule. Use it while a team is running its first tests, or on a small test where those five sections already force every decision that matters.
Full (seven sections) adds Tracking and Instrumentation Note and Validity Pre-Commitments. Use it when at least one of these is true:
- the experimentation program is mature enough to need a pre-committed shutdown threshold and a ramp-up plan, not only a decision rule;
- the surface is high-traffic or high-stakes enough that the blast radius of a mistake is large;
- the context is regulated or public-sector-adjacent, where proportionality to stakes is itself a named expectation rather than a courtesy.
Every lean heading appears in full, unchanged in name and order, so growing from lean to full never means rewriting a section you already filled in.
The rubric (self-grade)
Section titled “The rubric (self-grade)”Score each row 0, 1 or 2. Score a full-variant document under 11 out of 16, and the test launches with no stopping point anyone can act on: the readout that follows will spend its own time reconstructing decisions the design doc should have already made. The lean variant scores against fewer rows; see the scope table below.
| # | Criterion | 0 | 1 | 2 |
|---|---|---|---|---|
| 1 | Hypothesis is falsifiable | States a goal (“increase open rate”) with no mechanism and no way to know it failed | States a mechanism or the losing condition, but not both, or states evidence with no stated connection to the mechanism | States the mechanism, cites the evidence already pointing that way, and names the specific result that would prove it wrong |
| 2 | Population and unit justified | No eligibility rule or no randomization unit is stated | Eligibility, the traffic split and a unit are stated, but the unit carries no reason | Eligibility, any exclusions and the traffic split are stated, and the unit’s reason ties to who or what the change is visible to, ruling out spillover to someone outside their own assignment |
| 3 | Metrics carry a baseline | A metric is named with no baseline and no target or margin | A baseline or a target/margin is given, not both, or the table restates the PRD’s own metrics with no statement of whether this test needs its own numbers | Every row has both a baseline and a target or margin, and the document states in one line whether any row needs nothing beyond what the PRD already carries |
| 4 | Sample size is derived | A duration or sample size appears with no stated effect size and no named statistical framework | An effect size or a framework is named, not both, or a framework is named with no stated commitment against reviewing results early | The effect size, the named framework, and the resulting sample size or duration are all stated, with an explicit commitment not to review results before that point |
| 5 | Decision rule covers outcomes | Only a win outcome has a named action | A win and one of (null result, guardrail breach) have named actions; the other does not | A win, a null result, and an early guardrail breach each have a named action, and the named decision-maker matches the frontmatter |
| 6 | Tracking note is specific (full) | No events are named, or the note repeats the PRD’s instrumentation section with nothing said about what differs | Events are named but the exposure event is missing, or a verification is claimed with no stated date or environment | The exposure event is named, the existing events feeding each metric are named, and a verification is stated with the environment and the date it happened |
| 7 | Validity checks are owned (full) | A row names a worry with no threshold and no owner | Some rows carry both a threshold and a named owner; others carry one or neither | Every row that applies names a threshold or trigger and a named owner, and a check that does not apply says so in one line rather than being dropped silently |
| 8 | Overlap and risk named (full) | Both the overlap line and the risk line are empty, or a bare “N/A” with no stated reason | One of the two names a concrete practice; the other is generic or missing | The overlap line names a specific mechanism or states in one line why none applies, and the risk line names specific people or teams told, not a general claim that “stakeholders know” |
Which rows apply to what.
| Document | Rows | Maximum | Score against |
|---|---|---|---|
| full | all 8 | 16 | 11 |
| lean | 1-5 | 10 | 7 |
Rows 6, 7 and 8 are scored only against the full variant. The lean variant ships no Tracking and Instrumentation Note and no Validity Pre-Commitments section, so grading it on those rows would penalize the choice of variant rather than the quality of the document.
Score a row 2 only when you can point at the specific sentence, number or name its cell asks for; a longer table is not a better one.
Named anti-patterns
Section titled “Named anti-patterns”Restating the PRD’s metrics, renamed. The tell: the Primary and Guardrail Metrics table repeats the PRD’s primary metric, guardrail and measurement window with no new baseline or margin of its own. The fix: state only what this test adds beyond the PRD, or write in one line that this test needs nothing beyond what the PRD already states. Deep dive: companion section 3 (Anatomy > Primary and Guardrail Metrics) and section 7.
Peeking, called monitoring. The tell: someone checks the results before the committed sample size or duration is reached, and treats what they see as informing whether to stop, rather than as informational only. The fix: fix the sample size or duration in this document before the test launches, and treat any early look as informational unless a sequential design was named in the Minimum Detectable Effect section. Deep dive: companion section 3 (Anatomy > Minimum Detectable Effect and Sample Size or Duration) and section 7.
Presenting a borrowed duration as a standard. The tell: a draft states a fixed duration, such as a week, with no effect size, no variance estimate, and no named framework behind the number. The fix: derive the duration from the effect size and variance this specific test needs, using the platform’s own calculator. A reported typical duration from one company’s own program is not a field-wide convention. Deep dive: companion section 6, item 5.
Leaving the null outcome undecided. The tell: the Decision Rule names an action for a win and says nothing about what happens if nothing moves. The fix: name an action for the null outcome with the same specificity as the win, before the test launches; a fully powered test that shows no movement is a result, not a failure to get one. Deep dive: companion section 3 (Anatomy > Decision Rule).
Treating a vendor’s in-tool surface as a substitute for this document. The tell: the team points at a live dashboard, a configured experiment, or an exported results file and calls that the design document. The fix: keep the written rationale here. An in-tool configuration or analysis surface is built for setup and reporting, not for recording why the test exists or what each outcome will mean. Deep dive: companion section 7.
Amending the design silently after launch. The tell: the sample size, a metric, or the decision rule changes mid-test, and the document is edited in place with no record of what changed or why. The fix: record the amendment and the reason in the document itself, rather than quietly editing the original. Deep dive: companion section 7.
Running an experiment a cheaper method would already answer. The tell: the team will ship the change regardless of the result, or the available population is too small to produce a result anyone could act on. The fix: this is a decision for before you open this template; the “When NOT to use” section above names it directly. Deep dive: companion section 7.
Borrowing a frequency claim no source actually makes. The tell: a draft states a frequency for why experiments fail, or calls a practice “universal,” with no specific source behind the claim. The fix: state only the design principle actually supported, and drop the frequency claim entirely rather than hunting for a citation to justify a sentence already written. Deep dive: companion section 7.
When it is good enough
Section titled “When it is good enough”When a reader who was not in the room could tell, from the document alone, what action each possible result triggers and who makes that call. A design document that still requires a meeting after the results arrive to decide what they mean has not finished its own job.
Then self-grade against the rubric above, fill in the frontmatter owner, reviewers, decision-maker, status, planned start date and duration, and delete every HTML comment before sharing it.
The artifacts
Section titled “The artifacts”experiment-design-doc_template-lean.md · ~3,500 tokens
---title: "{{experiment_name}} Experiment Design Doc"experiment_name: "{{experiment_name}}"owner: "{{owner}}"reviewers: "{{reviewers}}"decision_maker: "{{decision_maker}}"status: "{{status}}"planned_start_date: "{{planned_start_date}}"planned_duration: "{{planned_duration}}"last_updated: "{{date}}"doc_type: experiment-design-docsize: leansource_template: experiment-design-docsource_template_version: 0.1.1---
<!--LEAN EXPERIMENT DESIGN DOC. The five decisions a team must fix before any data arrives: the hypothesis, whois eligible and how they are split between variants, what moves and what must not get worse, how big aneffect is worth detecting and how much traffic or time that takes, and what the team will do for everyoutcome, including a null one. Use it for a single test on a surface the team already understands. To growit into the full variant (see experiment-design-doc_template-full.md), ADD sections; never rename or reorderthe ones below, because the full variant is a strict superset of this one.
WHAT AN EXPERIMENT DESIGN DOC IS, AND IS NOTIt is written before the test launches, while nobody has seen any data, to fix the decisions a team wouldotherwise make only after seeing the results. It is NOT the PRD's Success metrics section restated: thatsection already names a primary metric, a guardrail and a measurement window, and this document starts pastthat point, with the variants, the allocation, the minimum detectable effect, and the decision rule. It isNOT a spike report: a spike reduces uncertainty about a technical question and carries no control group orstatistical decision rule. It is NOT a qa-docs test plan: a test plan verifies a product increment againstan agreed specification and grades it pass or fail, while this document tests an open hypothesis andestimates an effect against a baseline. See experiment-design-doc_companion.md section 8.
NO SOURCE MEASURED WHETHER WRITING ONE IMPROVES AN EXPERIMENT'S OUTCOMEThe case for this document rests on practitioner testimony and on the statistics of what goes wrong without apre-committed stopping rule, not on a controlled comparison of documented against undocumented tests. Seeexperiment-design-doc_companion.md section 1.
HOW TO FILL THIS IN1. Read the comment under each heading: WHAT it wants, WHY it matters (with a pointer into experiment-design-doc_companion.md), guiding questions to ASK, a GOOD and a WEAK example, and the TRAP to avoid. For table sections, PRIORITY explains the ordering rule and ROW HINT says what a good row contains.2. Replace each {{placeholder}} with your content. Fill Hypothesis and Primary and Guardrail Metrics first; the Minimum Detectable Effect section and the Decision Rule both depend on what you name there.3. The owner, reviewers, decision-maker, status, planned start date and duration live in the frontmatter above, not in a body section. See experiment-design-doc_companion.md section 3.4. If a section does not apply, write "N/A" and one line of why, rather than deleting it silently.5. Before you share it: self-grade against experiment-design-doc_guide.md, then DELETE every HTML comment. They are guidance, not content.-->
# {{experiment_name}} Experiment Design Doc
## Hypothesis
<!-- WHAT What you believe will happen, why you believe it, the evidence already behind that belief, and what result would prove you wrong. WHY Every convention in use agrees on one thing: the hypothesis must be falsifiable, and the evidence behind it belongs in this same sentence rather than a separate section. No source converges on one sentence form; this field is left free text on purpose, because at least four distinct, named conventions are in circulation and no source argues its own form is correct over the others. Deep dive: experiment-design-doc_companion.md section 3 (Anatomy > Hypothesis). ASK What do you believe will happen, and why, in mechanism terms rather than a wish? What evidence, qualitative or quantitative, already points this way? What result would prove the hypothesis wrong? GOOD "If we send the weekly digest at 9am local time instead of 6am, then open rate will rise, because recipients check email after arriving at work rather than before it. Supporting evidence: a support-ticket review found several users asking to delay the send. We will know we are wrong if open rate among the 9am group is no higher than the 6am control." WEAK "We think the new send time will help engagement." (no mechanism, no stated evidence, no condition that would prove it wrong) TRAP Writing a goal instead of a hypothesis. "Increase open rate" is an outcome you want; it is not a bet that can turn out to be wrong, and a hypothesis that cannot lose is not one. -->
{{hypothesis}}
## Population, Variants, and Allocation
<!-- WHAT Who is eligible for the test and what excludes someone, how many variants it carries and what each one changes, what share of traffic each variant gets, and the unit the test randomizes on. WHY The randomization unit is a real design choice, not a formality: when a change is visible to people connected to each other, such as teammates or a shared account, randomizing by the wrong unit violates the assumption that one person's assignment does not affect another's outcome, and the test's results stop meaning what they appear to mean. Deep dive: experiment-design-doc_companion.md section 3 (Anatomy > Population, Variants, and Allocation). ASK Who is eligible, and what excludes someone (a beta cohort, a region, an existing treatment)? How many variants, and what does each one actually change? What share of traffic does each get? What is the randomization unit, and could the change spill over to someone outside their own assignment? GOOD "Eligible: active weekly-digest subscribers on the standard plan, excluding anyone already in the digest-frequency beta. Two variants: control (6am send, unchanged) and treatment (9am send), 50/50 split. Randomization unit: subscriber account, because the digest is sent once per account regardless of how many people read it, so account-level randomization avoids splitting a shared inbox across two send times." WEAK "Everyone in the test gets either the old time or the new time." (no eligibility rule, no stated split, no randomization unit) TRAP Leaving the randomization unit unstated. A change visible to a user's teammates or household needs a unit bigger than the individual, and discovering that gap after launch is expensive to fix. -->
{{population_variants_and_allocation}}
## Primary and Guardrail Metrics
<!-- WHAT The one or two metrics that decide whether the hypothesis held, each with its baseline; the guardrail metrics that must not get worse, each with the margin that defines a breach. This is not the PRD's Success metrics section restated. WHY A PRD's Success metrics section already names a primary metric, a guardrail and a measurement window; this section's job is to carry the baseline and margin this specific test is sized around, not to repeat what the PRD already states. The more metrics a test analyzes for significance, the greater the chance one of them moves by chance alone rather than because of the change, which is why the sources that warn about this converge on capping primary metrics at one or two even though none agrees on the exact ceiling. Deep dive: experiment-design-doc_companion.md section 3 (Anatomy > Primary and Guardrail Metrics) and section 6, item 6. ASK What is the primary metric, and its current baseline? What guardrails must not get worse, and by how much before that counts as a breach? Does any row duplicate a metric the PRD already names, and if so, does this test need its own baseline or margin for it, or does it need nothing beyond what the PRD already states? PRIORITY List the primary metric or metrics first, then guardrails. Keep primary metrics to one or two and guardrails to a handful; a long list of primary metrics has usually stopped deciding anything and started hoping one of them moves. ROW HINT A good row names the metric, states whether it is primary or guardrail, gives a numeric baseline, and states the target or margin that defines success or breach. A weak row is a metric name with no baseline and no threshold. GOOD | Digest open rate (24h) | primary | 31% | Target: +3 points or more | WEAK | Engagement | primary | | Higher is better | TRAP Copying the PRD's metrics section verbatim into this table. If this test needs its own baseline or margin, state it; if it needs nothing beyond what the PRD already carries, say that in one line instead of restating the PRD's own numbers. A ratio metric's overall value can move opposite to its own numerator and denominator. Check both separately here, not only the ratio. -->
| Metric | Type (primary or guardrail) | Baseline | Target or guardrail margin ||---|---|---|---|| {{metric_name}} | {{metric_type}} | {{metric_baseline}} | {{metric_target_or_margin}} |
## Minimum Detectable Effect and Sample Size or Duration
<!-- WHAT The smallest effect worth detecting, the statistical framework the test actually runs on, and the sample size or duration that follows from both, fixed before the test starts. WHY This is the content no other document in this library carries, and it is the point of the whole section: committing to a sample size or duration in advance, before any data arrives, is what stops a team from deciding the test is done once the numbers happen to look good. Which statistical framework a test uses (fixed-horizon, sequential, or Bayesian) is a live, unresolved question across sources; this field asks you to name the one your platform actually runs rather than choosing one for you, and no power level or significance threshold is treated as a default here. Deep dive: experiment-design-doc_companion.md section 3 (Anatomy > Minimum Detectable Effect and Sample Size or Duration) and section 6, items 3 and 5. ASK What is the smallest effect worth acting on if you saw it? What statistical framework does the team's platform run on, and who chose it for this test? What sample size or duration follows from the effect size, the expected variance, and current traffic? Is the commitment not to look at results before that point written down, not just assumed? GOOD "MDE: 3 percentage points on open rate. Framework: fixed-horizon, the team's platform default. Duration: 14 days, from the platform's own sample-size calculator given current send volume and the stated MDE. The team will not review results before day 14 except to confirm the test is running." WEAK "We will run it for a couple of weeks and check." (no effect size, no named framework, no stated commitment against looking early) TRAP Treating a reported duration from another team's program, such as "about a week," as a rule for this test. A borrowed norm is not a substitute for a number derived from this test's own effect size, variance, and traffic; see experiment-design-doc_companion.md section 6, item 5. -->
{{mde_and_sample_size_or_duration}}
## Decision Rule
<!-- WHAT The action the team commits to for every outcome the test can produce, including a null result and an early guardrail breach, decided before any data arrives, and who makes the call. WHY Every source in this lineage ties a named action to every outcome a test can produce, including the outcome where nothing moves; deciding this only after seeing the data is the exact failure this whole document exists to prevent. "Ship, iterate or stop" is this family's own wording for the three-way choice, not a phrase a cited source uses verbatim. Deep dive: experiment-design-doc_companion.md section 3 (Anatomy > Decision Rule) and section 6, item 8. ASK What happens if the primary metric moves favorably past the MDE with no guardrail breach? What happens if it does not move, and the test was fully powered to detect it if it existed? What happens if a guardrail breaches its margin before the test finishes? Who makes the final call, and is that the same person named as decision-maker in the frontmatter? GOOD "Win (open rate up 3 points or more, no guardrail breach): ship 9am as the new default. Null (no detectable movement, test ran to its full powered duration): keep 6am; do not re-run this exact test without a new hypothesis. Guardrail breach (unsubscribe rate exceeds its margin at any point): stop the test immediately, regardless of the primary metric's reading. Decision-maker: the digest's product owner, named in the frontmatter above." WEAK "We'll ship it if it works out." (no definition of what counts as working, no stated action for a null result, no stop condition, no named decision-maker) TRAP Leaving the null result undecided. A fully powered test that shows no movement is a result, not a failure to get one, and it needs its own named action as much as a win does. -->
{{decision_rule}}experiment-design-doc_template-full.md · ~5,100 tokens
---title: "{{experiment_name}} Experiment Design Doc"experiment_name: "{{experiment_name}}"owner: "{{owner}}"reviewers: "{{reviewers}}"decision_maker: "{{decision_maker}}"status: "{{status}}"planned_start_date: "{{planned_start_date}}"planned_duration: "{{planned_duration}}"last_updated: "{{date}}"doc_type: experiment-design-docsize: fullsource_template: experiment-design-docsource_template_version: 0.1.1---
<!--FULL EXPERIMENT DESIGN DOC. Everything the lean variant carries, plus a note on what feeds each metric andthe exposure event that is new to this test, and a set of checks fixed before the test runs that decidewhether its result will be trustworthy. Use it when the program is mature enough to need a pre-committedshutdown threshold and a ramp-up plan, or when the blast radius of a mistake on this surface is large.
THIS VARIANT IS A STRICT SUPERSET OF THE LEAN ONE. The five lean sections appear here in the same order, withthe same headings and the same placeholders, and two sections are added at the end. If you started lean, youcan grow into this without rewriting anything you already filled in.
WHAT AN EXPERIMENT DESIGN DOC IS, AND IS NOTIt is written before the test launches, while nobody has seen any data, to fix the decisions a team wouldotherwise make only after seeing the results. It is NOT the PRD's Success metrics section restated: thatsection already names a primary metric, a guardrail and a measurement window, and this document starts pastthat point, with the variants, the allocation, the minimum detectable effect, and the decision rule. It isNOT a spike report: a spike reduces uncertainty about a technical question and carries no control group orstatistical decision rule. It is NOT a qa-docs test plan: a test plan verifies a product increment againstan agreed specification and grades it pass or fail, while this document tests an open hypothesis andestimates an effect against a baseline. See experiment-design-doc_companion.md section 8.
NO SOURCE MEASURED WHETHER WRITING ONE IMPROVES AN EXPERIMENT'S OUTCOMEThe case for this document rests on practitioner testimony and on the statistics of what goes wrong without apre-committed stopping rule, not on a controlled comparison of documented against undocumented tests. Seeexperiment-design-doc_companion.md section 1.
HOW TO FILL THIS IN1. Read the comment under each heading: WHAT it wants, WHY it matters (with a pointer into experiment-design-doc_companion.md), guiding questions to ASK, a GOOD and a WEAK example, and the TRAP to avoid. For table sections, PRIORITY explains the ordering rule and ROW HINT says what a good row contains.2. Replace each {{placeholder}} with your content. Fill Hypothesis and Primary and Guardrail Metrics first; every section after them depends on what you name there.3. The owner, reviewers, decision-maker, status, planned start date and duration live in the frontmatter above, not in a body section. See experiment-design-doc_companion.md section 3.4. If a section does not apply, write "N/A" and one line of why, rather than deleting it silently. This applies to individual rows of the Validity Pre-Commitments table as much as to a whole section: a check this test cannot run, such as a ramp-up plan on a test too small to ramp, is still worth one line saying so.5. Before you share it: self-grade against experiment-design-doc_guide.md, then DELETE every HTML comment. They are guidance, not content.-->
# {{experiment_name}} Experiment Design Doc
## Hypothesis
<!-- WHAT What you believe will happen, why you believe it, the evidence already behind that belief, and what result would prove you wrong. WHY Every convention in use agrees on one thing: the hypothesis must be falsifiable, and the evidence behind it belongs in this same sentence rather than a separate section. No source converges on one sentence form; this field is left free text on purpose, because at least four distinct, named conventions are in circulation and no source argues its own form is correct over the others. Deep dive: experiment-design-doc_companion.md section 3 (Anatomy > Hypothesis). ASK What do you believe will happen, and why, in mechanism terms rather than a wish? What evidence, qualitative or quantitative, already points this way? What result would prove the hypothesis wrong? GOOD "If we send the weekly digest at 9am local time instead of 6am, then open rate will rise, because recipients check email after arriving at work rather than before it. Supporting evidence: a support-ticket review found several users asking to delay the send. We will know we are wrong if open rate among the 9am group is no higher than the 6am control." WEAK "We think the new send time will help engagement." (no mechanism, no stated evidence, no condition that would prove it wrong) TRAP Writing a goal instead of a hypothesis. "Increase open rate" is an outcome you want; it is not a bet that can turn out to be wrong, and a hypothesis that cannot lose is not one. -->
{{hypothesis}}
## Population, Variants, and Allocation
<!-- WHAT Who is eligible for the test and what excludes someone, how many variants it carries and what each one changes, what share of traffic each variant gets, and the unit the test randomizes on. WHY The randomization unit is a real design choice, not a formality: when a change is visible to people connected to each other, such as teammates or a shared account, randomizing by the wrong unit violates the assumption that one person's assignment does not affect another's outcome, and the test's results stop meaning what they appear to mean. Deep dive: experiment-design-doc_companion.md section 3 (Anatomy > Population, Variants, and Allocation). ASK Who is eligible, and what excludes someone (a beta cohort, a region, an existing treatment)? How many variants, and what does each one actually change? What share of traffic does each get? What is the randomization unit, and could the change spill over to someone outside their own assignment? GOOD "Eligible: active weekly-digest subscribers on the standard plan, excluding anyone already in the digest-frequency beta. Two variants: control (6am send, unchanged) and treatment (9am send), 50/50 split. Randomization unit: subscriber account, because the digest is sent once per account regardless of how many people read it, so account-level randomization avoids splitting a shared inbox across two send times." WEAK "Everyone in the test gets either the old time or the new time." (no eligibility rule, no stated split, no randomization unit) TRAP Leaving the randomization unit unstated. A change visible to a user's teammates or household needs a unit bigger than the individual, and discovering that gap after launch is expensive to fix. -->
{{population_variants_and_allocation}}
## Primary and Guardrail Metrics
<!-- WHAT The one or two metrics that decide whether the hypothesis held, each with its baseline; the guardrail metrics that must not get worse, each with the margin that defines a breach. This is not the PRD's Success metrics section restated. WHY A PRD's Success metrics section already names a primary metric, a guardrail and a measurement window; this section's job is to carry the baseline and margin this specific test is sized around, not to repeat what the PRD already states. The more metrics a test analyzes for significance, the greater the chance one of them moves by chance alone rather than because of the change, which is why the sources that warn about this converge on capping primary metrics at one or two even though none agrees on the exact ceiling. Deep dive: experiment-design-doc_companion.md section 3 (Anatomy > Primary and Guardrail Metrics) and section 6, item 6. ASK What is the primary metric, and its current baseline? What guardrails must not get worse, and by how much before that counts as a breach? Does any row duplicate a metric the PRD already names, and if so, does this test need its own baseline or margin for it, or does it need nothing beyond what the PRD already states? PRIORITY List the primary metric or metrics first, then guardrails. Keep primary metrics to one or two and guardrails to a handful; a long list of primary metrics has usually stopped deciding anything and started hoping one of them moves. ROW HINT A good row names the metric, states whether it is primary or guardrail, gives a numeric baseline, and states the target or margin that defines success or breach. A weak row is a metric name with no baseline and no threshold. GOOD | Digest open rate (24h) | primary | 31% | Target: +3 points or more | WEAK | Engagement | primary | | Higher is better | TRAP Copying the PRD's metrics section verbatim into this table. If this test needs its own baseline or margin, state it; if it needs nothing beyond what the PRD already carries, say that in one line instead of restating the PRD's own numbers. A ratio metric's overall value can move opposite to its own numerator and denominator. Check both separately here, not only the ratio. -->
| Metric | Type (primary or guardrail) | Baseline | Target or guardrail margin ||---|---|---|---|| {{metric_name}} | {{metric_type}} | {{metric_baseline}} | {{metric_target_or_margin}} |
## Minimum Detectable Effect and Sample Size or Duration
<!-- WHAT The smallest effect worth detecting, the statistical framework the test actually runs on, and the sample size or duration that follows from both, fixed before the test starts. WHY This is the content no other document in this library carries, and it is the point of the whole section: committing to a sample size or duration in advance, before any data arrives, is what stops a team from deciding the test is done once the numbers happen to look good. Which statistical framework a test uses (fixed-horizon, sequential, or Bayesian) is a live, unresolved question across sources; this field asks you to name the one your platform actually runs rather than choosing one for you, and no power level or significance threshold is treated as a default here. Deep dive: experiment-design-doc_companion.md section 3 (Anatomy > Minimum Detectable Effect and Sample Size or Duration) and section 6, items 3 and 5. ASK What is the smallest effect worth acting on if you saw it? What statistical framework does the team's platform run on, and who chose it for this test? What sample size or duration follows from the effect size, the expected variance, and current traffic? Is the commitment not to look at results before that point written down, not just assumed? GOOD "MDE: 3 percentage points on open rate. Framework: fixed-horizon, the team's platform default. Duration: 14 days, from the platform's own sample-size calculator given current send volume and the stated MDE. The team will not review results before day 14 except to confirm the test is running." WEAK "We will run it for a couple of weeks and check." (no effect size, no named framework, no stated commitment against looking early) TRAP Treating a reported duration from another team's program, such as "about a week," as a rule for this test. A borrowed norm is not a substitute for a number derived from this test's own effect size, variance, and traffic; see experiment-design-doc_companion.md section 6, item 5. -->
{{mde_and_sample_size_or_duration}}
## Decision Rule
<!-- WHAT The action the team commits to for every outcome the test can produce, including a null result and an early guardrail breach, decided before any data arrives, and who makes the call. WHY Every source in this lineage ties a named action to every outcome a test can produce, including the outcome where nothing moves; deciding this only after seeing the data is the exact failure this whole document exists to prevent. "Ship, iterate or stop" is this family's own wording for the three-way choice, not a phrase a cited source uses verbatim. Deep dive: experiment-design-doc_companion.md section 3 (Anatomy > Decision Rule) and section 6, item 8. ASK What happens if the primary metric moves favorably past the MDE with no guardrail breach? What happens if it does not move, and the test was fully powered to detect it if it existed? What happens if a guardrail breaches its margin before the test finishes? Who makes the final call, and is that the same person named as decision-maker in the frontmatter? GOOD "Win (open rate up 3 points or more, no guardrail breach): ship 9am as the new default. Null (no detectable movement, test ran to its full powered duration): keep 6am; do not re-run this exact test without a new hypothesis. Guardrail breach (unsubscribe rate exceeds its margin at any point): stop the test immediately, regardless of the primary metric's reading. Decision-maker: the digest's product owner, named in the frontmatter above." WEAK "We'll ship it if it works out." (no definition of what counts as working, no stated action for a null result, no stop condition, no named decision-maker) TRAP Leaving the null result undecided. A fully powered test that shows no movement is a result, not a failure to get one, and it needs its own named action as much as a win does. -->
{{decision_rule}}
## Tracking and Instrumentation Note
<!-- WHAT Which existing events feed each metric named above, any new event this test needs, the test-specific exposure event that marks who saw which variant, and a statement that tracking has been verified before launch rather than assumed. This is a pointer, not a restatement of the PRD's own instrumentation section. WHY The PRD's own Analytics and instrumentation section already lists the events a feature needs; what this note adds is which of those events, plus anything new, actually feed this test's metrics, and the exposure event that has no PRD equivalent because it belongs to the test rather than the feature. A test that launches with its metrics defined but its events unverified discovers the gap only once it is too late to fix cheaply. Deep dive: experiment-design-doc_companion.md section 3 (Anatomy > Tracking and Instrumentation Note). ASK Which existing events feed each metric above? What new event, if any, does this test need that the feature's own instrumentation does not already fire? What is the exposure event, and does it fire correctly for every variant? Has tracking been checked in a lower environment before launch, and on what date? GOOD "Open rate: existing `digest_opened` event, unchanged. Unsubscribe rate: existing `unsubscribed` event, unchanged. New exposure event: `digest_send_variant_assigned`, firing at send time with the variant id as a property. Verified in staging on 2026-07-28 against both variants; both fired correctly." WEAK "Tracking already works for this." (no events named, no exposure event, no verification date) TRAP Treating this section as a restatement of the PRD's instrumentation section. If nothing here is new or test-specific, say that in one line; if something is, name only what the test itself adds. -->
{{tracking_and_instrumentation_note}}
## Validity Pre-Commitments
<!-- WHAT The checks and thresholds, fixed before the test runs, that decide whether its result will be trustworthy: a sample ratio mismatch threshold, an early-stop threshold for harm, a ramp-up plan where one applies, and how novelty effects and segments fixed in advance will be watched for. Plus two items that are not table rows: how this test handles overlap with concurrent experiments, and who else must be told the test is running and what its risks are. WHY This section's name and grouping are this library's own synthesis: no single source names a section with exactly this scope, and it is built by combining several sources' individually sourced guidance rather than copying one template. Each check exists to catch a specific way a result can look real and not be: an uneven traffic split invalidates the comparison itself, an unmonitored guardrail breach does real harm before anyone notices, and an unflagged novelty effect gets read as a stable one. Whether overlap with concurrent experiments needs designing around is contested: two sources treat it as a design check, while two vendors report, from their own experience, that meaningful interaction effects between parallel tests are rare. This bundle does not adopt either position as settled; it only asks you to state how this test handles the question. Deep dive: experiment-design-doc_companion.md section 3 (Anatomy > Validity Pre-Commitments) and section 6, item 4. ASK What sample-ratio check, run how often, would make you distrust the split itself? What guardrail breach, and detected how, stops the test early rather than waiting for the scheduled end? Does this test ramp up gradually, and if so, on what schedule; if not, why not? How will a novelty effect be checked for, and were any segments fixed before launch so their own composition cannot shift mid-test? Does this test run alongside another experiment on the same surface, and if so, how is that handled? Who outside this team needs to know the test is running and what its risks are? PRIORITY List only the checks that genuinely apply to this test; a ramp-up row on a test too small to ramp is not a check, it is padding. Order by how much skipping the check would undermine trust in the result if it went unnoticed. ROW HINT A good row names the check, the threshold or trigger that would act on it, and who owns watching for it. A weak row names a worry with no threshold and no owner. GOOD | Sample ratio mismatch | Flag if the platform's chi-squared check on assigned subscriber counts reports a significant gap from the configured 50/50 on any daily read | Data analyst on the digest team | Overlap line: "The subject-line test on the same weekly digest ends on 14 March and this test starts on 17 March, so no subscriber receives a send from both." Risks line: "A later send time can raise unsubscribes, which the guardrail row above stops on. Support hears two days before launch; the lifecycle marketing lead owns that note." WEAK | Watch for problems | | QA | Overlap line: "Other tests should not interfere." Risks line: "Stakeholders will be informed as needed." Neither names a test, a date, a person, or a consequence. TRAP Listing every check from a generic list regardless of whether it applies to this test. A row with no threshold and no owner is not pre-committed, it is a reminder to worry; write "N/A" with one line of why instead. -->
| Check | Threshold or trigger | Owner ||---|---|---|| {{validity_check}} | {{validity_threshold_or_trigger}} | {{validity_owner}} |
**Overlap with concurrent experiments.** {{overlap_handling}}
**Risks, and who else must be told.** {{risks_and_notification}}experiment-design-doc_example.md
---title: "Saved Views Adoption Nudge Experiment Design Doc"experiment_name: "Saved Views Adoption Nudge"owner: "Priya Nair (PM, Reporting)"reviewers: "Dana Osei (Staff Engineer, Platform), Lee Zhang (Data Eng)"decision_maker: "Priya Nair (PM, Reporting)"status: "approved"planned_start_date: "2026-08-03"planned_duration: "4 weeks (2026-08-03 through 2026-08-31)"last_updated: "2026-07-30"doc_type: experiment-design-docsize: fullsource_template: experiment-design-docsource_template_version: 0.1.0---
> **Worked example.** A filled `experiment-design-doc`, full variant, for the Saved Views adoption-nudge> test at Acme Analytics, chained onto the same [Saved Views for Dashboards PRD](../prd/prd_example.md),> [KPI dashboard](../kpi-dashboard/kpi-dashboard_example.md), [OKRs](../okrs/okrs_example.md) and> [risk register](../risk-register/risk-register_example.md) that the rest of this library's thread> already uses. Dated 2026-07-30, four days before the test itself launches on 2026-08-03. Its second> half, [the `experiment-readout` bundle's own example](../experiment-readout/experiment-readout_example.md),> reports this same test against the commitments fixed below. *(Corrected 2026-10-11: this said the> readout's example was not yet built. It was built on 2026-10-11.)*>> All figures, account counts, and event names are illustrative.
# Saved Views Adoption Nudge Experiment Design Doc
## Hypothesis
If we show a one-time in-app prompt to a Recurring Analyst after their fifth dashboard view since the2.4.0 release, suggesting they save their current filter, date range, and column setup as a view, thenweekly Saved Views adoption will rise against a no-prompt control, because analysts who have not yettried the feature may simply not have noticed the Views menu.Supporting evidence: adoption sits at 41% (the KPI dashboard, last reviewed 2026-07-20), and the riskregister already carries R-04, the risk that Recurring Analysts do notadopt Saved Views inside the program's 60-day launch-success window because they have deep habits in thelegacy report flow, with an in-product nudge named as that risk's own mitigation (risk register, samereview date). The hypothesis fails if, at the end of the four-week window, weekly adoption amongprompted accounts matches or trails the no-prompt control's own figure.
## Population, Variants, and Allocation
Eligible: accounts with at least one Recurring Analyst active in the trailing 7 days, the same segmentthe KPI dashboard already counts (495 analysts as of 2026-07-20), where that analyst reaches their fifthdashboard view on or after 2026-08-03. Excluded: the six analysts already enrolled in the Saved Viewsdesign-partner pilot, and their accounts, so a pilot account cannot land on either side of this test anddouble up the R-04 mitigation on itself. Two variants: control (current behavior, no prompt) andtreatment (the fifth-view prompt), 50/50 split.
Randomization unit: customer account, not individual analyst. FR-4, the PRD's requirement to share aview so that "other permitted users of that dashboard can select it", means a saved view a treatmentanalyst creates can become visible to a colleague on the same account, and the KPI dashboard's ownadoption panel counts a shared view for the viewer, not the creator. Randomizing at the analyst levelwould let a treatment analyst's share count as adoption for a colleague assigned to control on the sameaccount, which breaks the assumption that one unit's assignment does not affect another's outcome. Keeping a whole account onone side of the test, including everyone who shares its dashboards, avoids that spillover.
**The eligible population is small, and the test is sized against that, not against a bigger number wewish we had.** Acme's entitlements database counts roughly 230 accounts behind the 495 weekly activeRecurring Analysts the KPI dashboard tracks (illustrative; this test's own effective sample, oncerandomization moves from analysts to accounts, is nearer that account count than the analyst count). TheMinimum Detectable Effect section below is sized against accounts of that order, not a population thistest does not have.
## Primary and Guardrail Metrics
The primary metric is KR2, the OKRs' own Saved Views adoption key result, not the PRD's stated primarymetric (median time from dashboard open to first meaningful interaction): the nudge is built to moveadoption directly, and time-to-insight is read on the KPI dashboard's own panel instead. Weekly activeanalysts and shared-view permission incidents need no new baseline beyond what the KPI dashboard and thePRD already carry; this test adds only the adoption row's own margin, sized in the next section.
| Metric | Type (primary or guardrail) | Baseline | Target or guardrail margin ||---|---|---|---|| Saved Views adoption (share of Recurring Analysts using a saved view weekly) | primary | 41% (KPI dashboard, last reviewed 2026-07-20) | A statistically significant lift over control by 2026-08-31; the test is sized to detect a lift of about 18 points (this test's own MDE, see below; not KR2's full-quarter target of 60%) || Weekly active analysts | guardrail | 495 (KPI dashboard, same review date) | Must not fall below 480, the dashboard's own guardrail floor, at any point during the test || Dashboard load error rate | guardrail | No new baseline; the PRD's own guardrail already covers this | Must not rise above its pre-test level || Shared-view permission incidents | guardrail | None recorded; the PRD's own guardrail requires zero | Must stay at zero |
## Minimum Detectable Effect and Sample Size or Duration
Statistical approach: a fixed-horizon test at the experimentation platform's standard setting, 80% powerand a 5% significance level (illustrative); this test does not pick its own level. MDE: about 18percentage points of weekly adoption share, treatment against control. The arithmetic uses Evan Miller'srule of thumb, n = 16 σ²/δ² per arm, where σ² for a proportion is p(1 - p). At the 41% baseline, σ² is0.41 x 0.59, about 0.24. With about 115 accounts per arm, δ is the square root of 16 x 0.24 / 115, about0.18. Counting analysts instead of accounts would give about 12.5 points, but that overstates what thetest can resolve, because analysts on one account share its assignment. An 18-point MDE is a large asknext to KR2's own 19-point, full-quarter target (41% to 60%). It is large because the population issmall: this test can detect a big effect and nothing subtler. Sample: every eligibleaccount, roughly 230 (illustrative); there is no larger population to draw a smaller sample from, so the"sample size" this test commits to is the whole eligible population, not a number a calculator trims downto. Duration: four weeks, 2026-08-03 through 2026-08-31, chosen on its own grounds: long enough for anaccount to cross its fifth dashboard view and then be observed across several weekly adoption windows,not derived from the account count the way a duration would be on a higher-traffic surface.
The team will not review adoption numbers before 2026-08-31 except to confirm the prompt is firing (seethe Tracking and Instrumentation Note below); any look taken before then is informational only andchanges no decision. If the true effect on adoption is well below 18 points, this test will most likelyreturn a null result rather than a confirmed loss, and the Decision Rule below, not a second look at thesame data, is what happens next.
## Decision Rule
Win (treatment's adoption share finishes the four weeks above control by a statistically significantmargin at the platform's standard setting, with no guardrail breach): roll the fifth-view prompt out to100% of Recurring Analysts' accounts and retire the control. Null (no significant difference at the end ofthe full four weeks): this makes a lift of about 18 points or more unlikely, and says nothing about asmaller real lift, which may still exist. Shelve the fifth-view prompt rather than rerunning it unchanged, because the same populationcannot resolve a smaller effect the second time either. Bring the null finding to the next ReportingPlatform Modernization steering review and weigh a different mitigation for R-04 against its cost.Guardrail breach (weekly active analysts fall below 480, the dashboard load error rate rises above itspre-test level, or a shared-view permission incident occurs on either arm): stop the test for everyaccount immediately, regardless of the adoption reading, and send any permission incident to Dana Osei,who owns the permissions service. Decision-maker: Priya Nair, PM for Reporting, the same name as thefrontmatter above.
## Tracking and Instrumentation Note
Weekly adoption and the guardrails read from the existing `view_saved`, `view_switched`,`view_set_default`, `view_shared`, and `view_load_error` events; none of them change for this test. Newexposure event: `nudge_variant_assigned`, firing the moment an eligible account crosses its fifthdashboard view, carrying the variant id and the account id as properties. Checked in the stagingenvironment on 2026-07-29: each arm fired as expected, and the control side emits no visible prompt, sothe adoption panel can still attribute control accounts to the right side of the test.
## Validity Pre-Commitments
| Check | Threshold or trigger | Owner ||---|---|---|| Sample ratio mismatch | Run the platform's chi-squared sample-ratio check on assigned account counts every day; if it flags the split as off 50/50, nobody reads the result until the cause is found | Lee Zhang, Data Eng || Weekly active analysts guardrail | Stop the test early if the count falls below 480 on any day | Priya Nair || Dashboard load error rate and permission incidents | Stop the test early if the error rate rises above its pre-test level, or a single shared-view permission incident occurs on either arm | Dana Osei, Platform || Ramp-up | N/A. A 50/50 split across the full eligible account population is already as large as this test gets before the next steering review; there is no larger traffic tier to ramp into first. | N/A || Novelty effect | Compare week-1 adoption lift against weeks 2 through 4; a lift that decays toward zero by week 4 is flagged as novelty rather than counted as the finding | Priya Nair || Segments fixed in advance | None beyond the account-level control and treatment split; the eligible population is too small to slice further without turning each slice into its own underpowered test | N/A |
**Overlap with concurrent experiments.** No other experiment runs on the dashboard surface during thiswindow. The design-partner pilot already excludes its own six analysts and their accounts from theeligible population above, so there is no overlap between the two to design around.
**Risks, and who else must be told.** The Reporting Platform Modernization steering group already watchesR-04 and the KPI dashboard's adoption panel, so this test adds no new escalation line to them. Support istold before launch: a prompt appearing mid-session is the kind of change that generates a ticket ifnobody warns the team that answers it, and Marta Reyes, the program manager, owns that notificationbefore 2026-08-03.Provenance
Section titled “Provenance”The reasoning, the history and every source, in the repository:
- Companion - the long-form argument: why these sections, where the sources disagree, and what the bundle refuses to claim
- History - what changed in this bundle, and when
- Research log - every source consulted, with what each one actually supports
- Catalog metadata - the machine-readable record this page is generated from
Catalog record: 7 sections across 1 format(s), methodology growth/dual-track, typically owned by PM / Data Scientist.