Experiment Readout
beta · Family experimentation-docs · Phase measure · Sizes lean, full · ~5,100 tokens
The document a team writes after a product experiment ends, once the result is known, to report what ran, the results with their uncertainty, whether the validity checks hold, and the decision those results drove, ship, iterate, or stop, against the plan and decision rule an experiment design doc fixed in advance.
The short card. Why the document is shaped this way, and the argument behind every rule here, is in
experiment-readout_companion.md. A fully worked instance is
experiment-readout_example.md.
One honest limit before anything else. No source read for this bundle measured whether writing this document improves a decision. The case for it rests on the failure modes the sources name (peeking, cherry-picking, a loss reported as a win) and on keeping a record a later reader can audit, not on a controlled comparison of documented against undocumented tests. Read the rest of this card with that limit in mind.
When to use
Section titled “When to use”- A test that an experiment design doc planned has ended, and the decision it was meant to inform needs to be recorded before anyone acts on the result.
- The result needs to be checked against the pre-committed decision rule, the guardrail thresholds, and the validity checks the design doc set, rather than judged fresh from whatever the dashboard shows that day.
- More than one person needs the same account of what happened, including someone who was not in the room when the call was made.
- The result is a null or a guardrail breach, where the pull to skip writing anything down is strongest, and where a record matters most: a decision not to ship is still a finding worth keeping.
When NOT to use
Section titled “When NOT to use”- The test has not ended, or its pre-committed duration has not been reached. Reporting a result before that point is the early look that inflates a reported significance level. Wait for the committed duration, or, if monitoring mid-test, treat anything seen as informational only.
- You need a qa-docs test plan or test summary report. That document verifies a product increment against an agreed specification and grades it pass or fail, with no hypothesis, no allocation between variants, and no statistical decision rule. This document estimates an effect against a baseline. See companion section 8 for the boundary in full.
- You need a spike report (decision-docs). A spike reduces uncertainty about a technical question with no control group and no statistical decision rule; a refuted hypothesis there is a successful spike, which is never true of a result reported here. See companion section 8.
- A vendor’s live scorecard or one-time metrics export already exists, and the team is tempted to call that the readout. Statsig’s export carries no decision field, Optimizely’s scorecard is a live monitoring surface, and no source read for this bundle says either one replaces a written readout or sits beside it as a substitute. Use the in-tool surface for monitoring and raw export; keep the decision and its reasoning in this document.
- A test was run before anything was built, to decide whether to build it at all. That is excluded from this family by name. It is not this document’s job under any variant.
- You need a standing instrument. A tracking plan, a KPI dashboard, or an experiment log is maintained across many tests and is never finished. This document is written once per test and is finished when the decision it exists to inform is recorded.
- There is no design doc to report against. Without one, there is no pre-committed decision rule, guardrail threshold, or validity check to check this result against, and the two documents are a sequential pair for exactly that reason. Write the design doc first, even retroactively, stating what the decision rule would have been before reading the results, rather than filling this document in from a blank page.
A team that keeps one living document for the plan and the results instead of two separate files still uses this bundle’s sections; they simply become that document’s second half, after the sections the experiment-design-doc’s own guide covers. The two bundles’ sections do not overlap, so nothing here needs renaming to fit one file instead of two.
Pick a variant
Section titled “Pick a variant”Lean (eight sections) is the default: Executive Summary, Context and Setup, Key Results: Primary Metrics, Key Results: Supporting Metrics, Validity Checks, Decision, Decision Owner, and Next Steps and Owners. Use it when the eight lean sections already carry the whole decision.
Full (eleven sections) adds Deep-Dive, Implementation Details and Analysis Notes, and Further Analysis. Use it when the decision rests on more than the primary metric alone, such as a business question answered through a specific segment, or when readers will want to audit the underlying analysis later. Every lean heading appears in full, unchanged in name and order, so growing from lean to full never means rewriting a section already filled in.
The rubric (self-grade)
Section titled “The rubric (self-grade)”Score each row 0, 1 or 2. Score the full variant under 14 out of 20, or the lean variant under 11 out of 16, and a reader needs to reopen the design doc and the raw numbers to reconstruct what was actually decided and why, which is exactly the meeting this document exists to make unnecessary. The lean variant carries fewer rows; the scope table below states each variant’s own maximum and threshold.
| # | Criterion | 0 | 1 | 2 |
|---|---|---|---|---|
| 1 | Decision stated up front | The summary describes the test or its mechanics but never states a decision; a reader must go to the Decision section to find out what happened | A decision word appears (“null,” “shipped”) with no statement of why it matters or what it changes | The first sentence names the decision and the one finding behind it, so a reader who stops there could still repeat, correctly, what was decided and why |
| 2 | Deviations from plan checked | The design doc is not linked, or the deviations line is missing entirely | The design doc is linked and “no deviations” is asserted, with nothing said about what was actually checked (dates, split, population) against the plan | The design doc is linked, and the deviations line either names the specific change, with what, when and why, or confirms the match by naming the actual dates, split and population run |
| 3 | Uncertainty is legible | A change is reported with no interval at all | An interval appears, but its confidence level is not stated in the row, or only one of the absolute or relative change columns is filled in | Both change columns, the interval, and the interval’s own stated confidence level appear in the same row, so a reader can tell the result from noise without a follow-up question |
| 4 | Guardrails read against thresholds | A guardrail is reported with a value and no threshold to compare it to, or a guardrail the design doc named is missing from the table | Thresholds appear for some guardrails but not all, or a threshold was set at readout time rather than carried from the design doc | Every guardrail the design doc named appears, each against the design doc’s own threshold, with a held-or-breached call a reader could check against the adjacent numbers |
| 5 | Validity checks named | One blanket line, such as “looks fine,” stands in for the whole section, with no individual check named | Some checks are named with a specific result, but at least one check the design doc pre-committed to running is missing, or a result is stated with no specifics | Every check the design doc pre-committed to appears by name with a specific, checkable result, a number, a date range, or an explicit “not applicable” with why, and nothing beyond what was pre-committed is invented |
| 6 | Null scope stated | The outcome carries a label (win, null, loss) with no statement of what that label rules in or out | An effect size or interval is mentioned but not tied explicitly to what the result does and does not rule out | The decision names which of ship, iterate or stop the pre-committed rule produced, and for a null result, states the effect size it rules out and names a smaller one it does not |
| 7 | Owner reconciled with plan | No individual is named; “the team decided” or “leadership agreed” stands in for a person | A specific person is named, with nothing said about whether that person matches the design doc’s own decision-maker field | A specific person and venue are named, and the section states explicitly whether that person matches the design doc’s decision-maker, with a reason if it changed |
| 8 | Next steps owned | An action is listed with no owner, or “the team” stands in for a name | Most rows carry one named owner, but at least one does not, or no row carries a status | Every row names exactly one person and a status, and the action the decision itself requires, such as a rollout, a rollback or a retirement, appears first |
| 9 | Segments name complements (full) | A segment’s result is reported with no statement of whether the design doc fixed that segment in advance, and no complement value shown | The segment is named as pre-registered or exploratory, but its complement’s own number is missing, or vice versa | Every segment claim states whether the design doc fixed it in advance, and shows the complement subgroup’s own number alongside it, so a reader can see whether the effect is isolated or general |
| 10 | Further Analysis kept separate (full) | An item there is also invoked, even implicitly, to support the Decision section above, or the section reads as an unsorted dump with no stated reason the items do not matter to the decision | Items read as plausibly exploratory, but the section never states outright that none of them factored into the decision | The section states explicitly that nothing in it factored into the decision above, and every item is framed as a lead for a future test rather than evidence for this one |
Rows 9 and 10 are scored only against the full variant. The lean variant ships no Deep-Dive and no Further Analysis section, so grading it on those rows would penalize the choice of variant rather than the quality of the document.
Which rows apply to what.
| Variant | Rows | Maximum | Threshold |
|---|---|---|---|
| full | 1-10 | 20 | 14 |
| lean | 1-8 | 16 | 11 |
Score a row 2 only when you can point at the specific sentence, number or name its cell asks for; a longer document is not automatically a better one.
Named anti-patterns
Section titled “Named anti-patterns”Naming the lift-decay pattern a “primacy effect.” The tell: a draft describes an early lift that falls across date segments and labels it with that term. No source read for this bundle uses it. The fix: call it a novelty effect, the term the sourced pattern actually carries, and detect it by segmenting the result by date, the way the Validity Checks section asks for. Deep dive: companion section 7.
Presenting “ship, iterate or stop” as a convention the sources use. The tell: a draft cites a source for the three-way outcome by name. It is this family’s own wording, stated as the contract’s position, not a phrase any source read here uses verbatim. The fix: state it as this library’s own choice, as the anatomy in companion section 3 does, rather than attributing it to the research. Deep dive: companion section 7.
Presenting a significance level, power level or test duration as a field-wide standard. The tell: a draft states “95% confidence” or a fixed test length as a default rather than something the design doc or the platform actually used. The one specific figure in the sources is one company’s own reported practice, not a cross-industry norm. The fix: state whichever level the design doc or the platform actually used, in the row, and do not present a borrowed number as a default. Deep dive: companion section 6, item 8.
Treating a null result as proof the change has no effect. The tell: the Decision section reads a null as “it doesn’t work” with no reference to the interval’s width. A null on a test powered to detect a large effect says nothing about a smaller one. The fix: state the specific effect size the result rules out, and name a smaller one it does not, as rubric row 6 requires. Deep dive: companion section 7.
Treating a guardrail breach as an automatic block on shipping. The tell: a draft states that any guardrail movement past its threshold ends the test, as a universal rule rather than the specific rule the design doc fixed. No source read for this bundle states a standard way to weigh a primary gain against a guardrail regression. The fix: report the guardrail’s movement against the design doc’s own pre-committed rule for that specific trade-off, and let that rule, not a blanket policy, decide what the breach means. Deep dive: companion section 7.
Claiming that tags make an experiment archive findable. The tell: a draft adds a tags field to this document’s frontmatter and describes it as how future readers will find the experiment. One source reports the opposite of its own practice: labelling experiments with tags proved unsustainable at its scale. The fix: rely on the frontmatter fields this bundle already carries, the experiment’s name and its design-doc link, rather than adding a field this bundle has no source for. Deep dive: companion section 6, item 5.
Borrowing a clinical or public-sector element and presenting it as product-management practice. The tell: the deviations-from-plan line, the both-sided effect-size columns, or the complement-subgroup rule appears with no statement of where it comes from. Each of these three pieces is borrowed from a clinical trial-reporting standard, not from any product source. The fix: name the lineage each time a borrowed element is used, as the anatomy in companion section 3 does, rather than letting it read as settled product practice. Deep dive: companion section 5.
Treating a vendor’s in-tool scorecard or export as a substitute for, or a companion to, this document. The tell: a draft links a live scorecard or a one-time PDF export in place of filling in Key Results or Validity Checks. Statsig’s export carries no decision field, and no source read for this bundle claims either surface replaces or sits beside a written readout. The fix: use the in-tool surface for what it is built for, live monitoring and raw metric export, and keep the decision and its reasoning in this document. Deep dive: companion section 8.
When it is good enough
Section titled “When it is good enough”When a reader who was not in the room, and has not seen the raw data, could state the decision, the one finding behind it, and what happens next, from this document alone, without reopening the design doc or the dashboard to check any of it. A readout that still requires a meeting to explain what the result meant has not finished its own job.
Then self-grade against the rubric above, confirm the decision owner matches the design doc’s own decision-maker or states why it changed, and delete every HTML comment before sharing it.
The artifacts
Section titled “The artifacts”experiment-readout_template-lean.md · ~5,100 tokens
---title: "{{experiment_name}} Experiment Readout"experiment_name: "{{experiment_name}}"design_doc: "{{design_doc_link}}"author: "{{author}}"decision_maker: "{{decision_maker}}"start_date: "{{start_date}}"end_date: "{{end_date}}"decision: "{{decision}}"readout_date: "{{readout_date}}"doc_type: experiment-readoutsize: leansource_template: experiment-readoutsource_template_version: 0.1.1---
<!--LEAN EXPERIMENT READOUT. The eight sections a concluded test must report: the decision stated up front,what ran, the primary and supporting results with their uncertainty, whether the result can be trusted,the decision the pre-committed rule actually produced, who owns that call, and what happens next. Use itfor a test whose decision rests on its primary metric alone. To grow it into the full variant (seeexperiment-readout_template-full.md), ADD sections; never rename or reorder the ones below, because thefull variant is a strict superset of this one.
WHAT AN EXPERIMENT READOUT IS, AND IS NOTIt is written after a test ends, reporting what ran, the results with their uncertainty, whether thoseresults can be trusted, and the decision they drove, with its owner. It is NOT the experiment design docrestated: that document fixed the hypothesis, the variants, the metrics and the decision rule before thetest ran, and this document reports against that exact plan rather than repeating it. It is NOT atest-summary-report: that qa-docs document verifies a product increment against an agreed specificationand grades it pass or fail, with no hypothesis, no allocation between variants, and no statisticaldecision rule, while this document estimates an effect against a baseline. It is NOT a spike report:a spike reduces uncertainty about a technical question and carries no control group or statisticaldecision rule. It is NOT a live experiment scorecard or a one-time metrics export: those are tool-nativesurfaces for monitoring and raw data, not a decision-bearing document, and neither replaces this one orsits beside it as a substitute. See experiment-readout_companion.md section 8.
NO SOURCE MEASURED WHETHER WRITING ONE IMPROVES DECISIONSThe case for this document rests on the failure modes the sources name (peeking, cherry-picking, a lossreported as a win) and on keeping a record a later reader can audit, not on a controlled comparison ofdocumented against undocumented tests. See experiment-readout_companion.md section 1.
HOW TO FILL THIS IN1. Read the comment under each heading: WHAT it wants, WHY it matters (with a pointer into experiment-readout_companion.md), guiding questions to ASK, a GOOD and a WEAK example, and the TRAP to avoid. For table sections, PRIORITY explains the ordering rule and ROW HINT says what a good row contains.2. The experiment's name, a link to its design doc, the author, the decision-maker, the test's start and end dates, the decision, and the readout date live in the frontmatter above, not in a body section. Fill the design doc link first: every section below reports against that exact plan.3. Report the same hypothesis, metrics and decision rule the design doc fixed, applied to what actually happened. If anything ran differently from the plan, say so in Context and Setup rather than quietly reporting as if the plan and the run were identical.4. If a section, or a row in a table, does not apply, write "N/A" and one line of why, rather than deleting it silently. A reader cannot tell a deleted section from one nobody considered.5. Before you share it: self-grade against experiment-readout_guide.md, then DELETE every HTML comment. They are guidance, not content.-->
# {{experiment_name}} Experiment Readout
## Executive Summary
<!-- WHAT The decision itself, stated up front, not buried under the evidence that supports it. WHY Every section-order source names this section first, under three different titles: Eppo's "Executive Summary", Tanmayi Sai's "TL;DR", and Optimizely's "Purpose". A separate account of one team's own practice found that leading with what was learned, rather than a win or loss label, was what readers actually remembered. Deep dive: experiment-readout_companion.md section 3 (Anatomy > Executive Summary). ASK What is the decision, in one line? What is the single sentence a reader who stops here needs to walk away with? Does the summary lead with the finding, or does it make a reader hunt through the results sections below to find out what was decided? GOOD "Decision: stop. The one-click reorder button did not raise the 7-day reorder rate past what control already does; the result is a null, not a win or a loss, and the finding is that returning customers already use the existing reorder flow when they want one." WEAK "The test ran for three weeks and here is what we found." (no decision stated, forces the reader to hunt for it below) TRAP Treating the summary as a teaser for the sections below rather than the decision itself. A reader who reads only this paragraph should still know what was decided. -->
{{executive_summary}}
## Context and Setup
<!-- WHAT What ran: the variants, the dates, and the population, with a link to the design doc rather than a restatement of it, plus one line on anything that ran differently from the plan. WHY Sourced to Tanmayi Sai's "Experiment Setup" and Optimizely's "Details" field (number of variations, the dates the test ran, and the visitor counts by variation). The deviations line is new in this bundle's own research and is borrowed from the clinical trial-reporting lineage, not from product practice: it traces to CONSORT's own 2010 addition of an item for identifying "any changes to the primary and secondary outcome (endpoint) measures after the trial started," and is corroborated by Booking.com's pre-registration discipline, which fixes the metrics a hypothesis is tested against before the test begins. Deep dive: experiment-readout_companion.md section 3 (Anatomy > Context and Setup). ASK What variants ran, over what dates, against what population, as the design doc specified? Does this match the design doc exactly, or did anything change, such as the traffic split, the dates, or who was eligible, after the test launched? What is the one-line answer on what ran differently from the plan, if anything? GOOD "Ran 2026-09-01 through 2026-09-21, control and treatment at the planned 50/50 split, on the order-history page for all logged-in customers with at least one prior order, as the design doc specified. Deviations from the design doc: none; the test ran the full 21 days with no early stop." WEAK "We ran the test for about three weeks." (no dates, no population, no statement on deviations) TRAP Restating the design doc's hypothesis and metric baselines in full here. Link to it instead; this section's job is what actually ran, not what was planned. -->
{{context_and_setup}}
**What ran differently from the design doc.** {{deviations_from_design_doc}}
## Key Results: Primary Metrics
<!-- WHAT The primary metric's movement against its baseline, stated with its uncertainty. WHY The column shape is read verbatim from Eppo's own export: "Metric Control Test Change (abs) Change (rel) CI Sig?". Optimizely's narrative version asks for the same content in prose. The two change columns, absolute and relative, are corroborated from the clinical trial-reporting lineage, labeled here rather than presented as product practice: CONSORT's 2010 revision recommends that "for binary outcomes, presentation of both relative and absolute effect sizes is recommended." The interval's confidence level is stated in the row, never assumed as a field default; no source read states one level as a convention. Deep dive: experiment-readout_companion.md section 3 (Anatomy > Key Results: Primary Metrics) and section 6, item 8. ASK What is the primary metric's control value and test value? What is the absolute change, and the relative change? What is the confidence interval, and at what confidence level (state the level in the row)? Is the change significant at that level? PRIORITY List the primary metric row or rows first; the design doc capped these at one or two, and this table should not grow past what the design doc actually named as primary. ROW HINT A good row names the metric, both values, both change columns, an interval with its stated confidence level, and a significance call. A weak row reports a change with no interval, so a reader cannot tell it from noise. GOOD | 7-day reorder rate | 11.0% | 11.3% | +0.3 pts | +2.7% | -0.6 to +1.2 pts (95% CI) | No | WEAK | Reorder rate | up slightly | | | | | | TRAP Reporting the relative change alone. A small relative lift on a tiny base can read as dramatic; report both, per the clinical-lineage recommendation this bundle borrows for binary outcomes. A ratio metric's overall value can move opposite to its own numerator and denominator. Check both separately before trusting the ratio's own direction. -->
| Metric | Control | Test | Change (abs) | Change (rel) | CI | Sig? ||---|---|---|---|---|---|---|| {{metric_name}} | {{metric_control}} | {{metric_test}} | {{metric_change_abs}} | {{metric_change_rel}} | {{metric_ci}} | {{metric_significant}} |
## Key Results: Supporting Metrics
<!-- WHAT Every guardrail's movement, reported against the threshold the design doc fixed. WHY This is Eppo's own second results section. A guardrail metric is one that measures what a team does not want to degrade but does not expect to improve; a separate vendor's in-tool surface draws the same line between decision-making metrics and guardrail metrics. Each row here is read against the number the design doc pre-committed, not re-derived at readout time: that pre-commitment is what makes a breach legible rather than a judgment call made after the fact. Deep dive: experiment-readout_companion.md section 3 (Anatomy > Key Results: Supporting Metrics). ASK What is each guardrail's movement, control against test? What threshold did the design doc set for a breach? Did any guardrail breach its threshold, and if so, when? PRIORITY List every guardrail the design doc named, even one that held with no interesting movement; a guardrail silently dropped from the readout cannot be checked by a reader. ROW HINT A good row names the guardrail, the design doc's own threshold, both values, and a held or breached call. A weak row reports a number with no threshold to compare it to. GOOD | Checkout error rate | Design doc: no increase past 0.4% | 0.31% | 0.33% | Held | WEAK | Checkout errors | about the same | | | TRAP Re-deriving a new threshold here instead of using the one the design doc pre-committed. A threshold decided after seeing the data is not a guardrail, it is a rationalization. -->
| Guardrail | Design-doc threshold | Control | Test | Status ||---|---|---|---|---|| {{guardrail_name}} | {{guardrail_threshold}} | {{guardrail_control}} | {{guardrail_test}} | {{guardrail_status}} |
## Validity Checks
<!-- WHAT Whether the result can be trusted at all, checked before it is read as a finding: a sample ratio mismatch, a mismatch in a metric's own observation units, a novelty effect read by date, an early look before the test finished, a telemetry change mid-test, and interaction with another live experiment on the same surface. WHY This section's name and scope are this library's own POSITION: no single source names one section with exactly this job, and it is built by combining several individually sourced checks, the same way the design doc's own Validity Pre-Commitments section was built. Each check reports against the design doc's pre-commitment where one exists: an uneven traffic split invalidates the comparison itself; a decline in the lift across date segments is a named indicator of a novelty effect; reading results before the pre-committed duration ends inflates the reported significance level; and two sources note that interaction with another concurrently running experiment is worth designing around even though vendors report it is rare in practice. Deep dive: experiment-readout_companion.md section 3 (Anatomy > Validity Checks). ASK Did the sample ratio match the configured split on every scheduled check? Did any metric's own observation unit shift mid-test? Did the lift decay across date segments, which would flag a novelty effect? Was the result read or acted on before the pre-committed duration ended? Did any telemetry change mid-test? Did this test overlap with another live experiment on the same surface, and if so, how was that handled? PRIORITY Report every check the design doc pre-committed to running, ordered so a check whose failure would most undermine trust in the result comes first. A check the design doc did not pre-commit to is reported as "not pre-committed," not invented after the fact. ROW HINT A good row names the check, states the result plainly (passed, flagged, not applicable), and if flagged, what it changes about how the result should be read. A weak row says "looks fine" with no specific check named. GOOD | Sample ratio mismatch | Passed on every daily read | Configured 50/50; the largest daily gap stayed within the design doc's own chi-squared threshold | WEAK | Validity | Looks OK | No major issues noticed | TRAP Treating a passed sample-ratio check as proof the whole result is trustworthy. Each row here catches a different failure mode; a result needs all of them checked, not just the one that is easiest to run. -->
| Check | Result | Notes ||---|---|---|| {{validity_check_name}} | {{validity_check_result}} | {{validity_check_notes}} |
## Decision: Ship, Iterate or Stop
<!-- WHAT The outcome the pre-committed decision rule actually produced, including what a null result does and does not rule out, and whether the decision is reversible. WHY "Ship, iterate or stop" is this family's own wording for the three-way choice, stated as the contract's own POSITION, not a phrase any source read here uses verbatim. The nearest conventions in the sources are a pre-committed "ship/no-ship decision," "Win Lose Flat" outcomes, "roll out to more users, end, iterate, etc.," and "inconclusive" or "hypothesis proved." New in this bundle's own research: a null result states what it rules out and what it does not. One source's own framework counts an adequately powered null as a result in its own right rather than a non-result, with its own default action of iterating, abandoning, or shipping if the change is infrastructure-only; a separate account states the same principle from the opposite direction, that a decision not to ship is still a finding. An ASK on the decision's reversibility is traced to a source outside the experimentation canon entirely: most decisions are changeable and reversible, and only a smaller set deserve the slower process a one-way door warrants. Deep dive: experiment-readout_companion.md section 3 (Anatomy > Decision) and section 6, items 2 and 8. ASK Which of ship, iterate or stop does the design doc's pre-committed decision rule actually produce for this result? If the result is a null, what effect size does it rule out, and what smaller effect does it not rule out (the interval's width, not the point estimate, answers this)? Is this decision reversible, and does that change how fast it needs to be made? GOOD "Null. The design doc's decision rule for a null result is to iterate on a different mechanism rather than re-run this exact test unchanged. This result rules out a lift of 1.2 points or more at 95% confidence; it does not rule out a smaller lift the test was not powered to detect. The decision is reversible: the button can be removed with no migration cost, so this is a two-way door." WEAK "It didn't really work so we're not shipping it." (no named outcome category, no statement of what the null rules out, no reversibility note) TRAP Treating a null result as proof the change has no effect. A null on a test powered to detect a 1.2-point lift says nothing about a 0.3-point one; state the interval's width, not just its center. -->
{{decision_narrative}}
**What this result rules out, and what it does not.** {{null_result_scope}}
**Reversibility.** {{decision_reversibility}}
## Decision Owner
<!-- WHAT Who is named as having made the call, written as a field rather than a signature buried in prose. WHY This section is this library's own POSITION, and it is unsourced: no source read names a field for the decision owner. The nearest is Optimizely's "Decision maker," which names a role among the people involved in a test, not a field this document itself carries. This section exists so that whoever that person turns out to be, their name is written down rather than implied. Deep dive: experiment-readout_companion.md section 3 (Anatomy > Decision Owner) and section 6, item 3. ASK Who made this call? Is it the same person the design doc named as decision-maker in its own frontmatter? If not, who, and in what forum was the call made? GOOD "Jordan Alvarez (PM, Checkout), the decision-maker named in the design doc's frontmatter, made this call at the 2026-09-22 experiment review." WEAK "The team decided to stop." (no named person, no venue, cannot be traced back to the pre-committed decision-maker) TRAP Leaving this section's named person different from the frontmatter's decision-maker field with no note of why. If the decision-maker changed, say so; if not, confirm the match rather than letting the two fields silently disagree. -->
{{decision_owner_statement}}
## Next Steps and Owners
<!-- WHAT The concrete actions the decision requires, each with a named owner. WHY Eppo's own filled example assigns an owner to its very first next step. A separate practitioner frames the same section more broadly, as the plan for the project going forward, which is why this section carries not only the immediate action the decision requires but anything else it implies, each with its own owner. Deep dive: experiment-readout_companion.md section 3 (Anatomy > Next Steps and Owners). ASK What does the decision itself require (a rollout, a rollback, a retirement of a tracking field)? What does it imply beyond that immediate action? Who, by name, owns each one? PRIORITY List the action the decision itself requires first, then anything downstream of it. ROW HINT A good row names a concrete action, one owner, and a status. A weak row names a hope with no owner. GOOD | Remove the reorder-button flag and the control path from the codebase | Sam Ruiz (Eng) | Not started | WEAK | Think about next steps | | | TRAP Naming "the team" as the owner. One name per row, the same discipline the Decision Owner section above asks for. -->
| Action | Owner | Status ||---|---|---|| {{next_step_action}} | {{next_step_owner}} | {{next_step_status}} |experiment-readout_template-full.md · ~6,650 tokens
---title: "{{experiment_name}} Experiment Readout"experiment_name: "{{experiment_name}}"design_doc: "{{design_doc_link}}"author: "{{author}}"decision_maker: "{{decision_maker}}"start_date: "{{start_date}}"end_date: "{{end_date}}"decision: "{{decision}}"readout_date: "{{readout_date}}"doc_type: experiment-readoutsize: fullsource_template: experiment-readoutsource_template_version: 0.1.1---
<!--FULL EXPERIMENT READOUT. Everything the lean variant carries, plus a Deep-Dive for the business questionsthe decision actually depends on, a technical Implementation Details and Analysis Notes section, and aFurther Analysis section for findings that do not bear on the decision. Use it for a test whose decisionrests on more than its primary metric, or whose readers will want to audit the analysis later.
THIS VARIANT IS A STRICT SUPERSET OF THE LEAN ONE. The eight lean sections appear here in the same order,with the same headings and the same placeholders, and three sections are added: Deep-Dive, ImplementationDetails and Analysis Notes, and Further Analysis. If you started lean, you can grow into this withoutrewriting anything you already filled in.
WHAT AN EXPERIMENT READOUT IS, AND IS NOTIt is written after a test ends, reporting what ran, the results with their uncertainty, whether thoseresults can be trusted, and the decision they drove, with its owner. It is NOT the experiment design docrestated: that document fixed the hypothesis, the variants, the metrics and the decision rule before thetest ran, and this document reports against that exact plan rather than repeating it. It is NOT atest-summary-report: that qa-docs document verifies a product increment against an agreed specificationand grades it pass or fail, with no hypothesis, no allocation between variants, and no statisticaldecision rule, while this document estimates an effect against a baseline. It is NOT a spike report:a spike reduces uncertainty about a technical question and carries no control group or statisticaldecision rule. It is NOT a live experiment scorecard or a one-time metrics export: those are tool-nativesurfaces for monitoring and raw data, not a decision-bearing document, and neither replaces this one orsits beside it as a substitute. See experiment-readout_companion.md section 8.
NO SOURCE MEASURED WHETHER WRITING ONE IMPROVES DECISIONSThe case for this document rests on the failure modes the sources name (peeking, cherry-picking, a lossreported as a win) and on keeping a record a later reader can audit, not on a controlled comparison ofdocumented against undocumented tests. See experiment-readout_companion.md section 1.
HOW TO FILL THIS IN1. Read the comment under each heading: WHAT it wants, WHY it matters (with a pointer into experiment-readout_companion.md), guiding questions to ASK, a GOOD and a WEAK example, and the TRAP to avoid. For table sections, PRIORITY explains the ordering rule and ROW HINT says what a good row contains.2. The experiment's name, a link to its design doc, the author, the decision-maker, the test's start and end dates, the decision, and the readout date live in the frontmatter above, not in a body section. Fill the design doc link first: every section below reports against that exact plan.3. Report the same hypothesis, metrics and decision rule the design doc fixed, applied to what actually happened. If anything ran differently from the plan, say so in Context and Setup rather than quietly reporting as if the plan and the run were identical.4. If a section, or a row in a table, does not apply, write "N/A" and one line of why, rather than deleting it silently. A reader cannot tell a deleted section from one nobody considered.5. Before you share it: self-grade against experiment-readout_guide.md, then DELETE every HTML comment. They are guidance, not content.-->
# {{experiment_name}} Experiment Readout
## Executive Summary
<!-- WHAT The decision itself, stated up front, not buried under the evidence that supports it. WHY Every section-order source names this section first, under three different titles: Eppo's "Executive Summary", Tanmayi Sai's "TL;DR", and Optimizely's "Purpose". A separate account of one team's own practice found that leading with what was learned, rather than a win or loss label, was what readers actually remembered. Deep dive: experiment-readout_companion.md section 3 (Anatomy > Executive Summary). ASK What is the decision, in one line? What is the single sentence a reader who stops here needs to walk away with? Does the summary lead with the finding, or does it make a reader hunt through the results sections below to find out what was decided? GOOD "Decision: stop. The one-click reorder button did not raise the 7-day reorder rate past what control already does; the result is a null, not a win or a loss, and the finding is that returning customers already use the existing reorder flow when they want one." WEAK "The test ran for three weeks and here is what we found." (no decision stated, forces the reader to hunt for it below) TRAP Treating the summary as a teaser for the sections below rather than the decision itself. A reader who reads only this paragraph should still know what was decided. -->
{{executive_summary}}
## Context and Setup
<!-- WHAT What ran: the variants, the dates, and the population, with a link to the design doc rather than a restatement of it, plus one line on anything that ran differently from the plan. WHY Sourced to Tanmayi Sai's "Experiment Setup" and Optimizely's "Details" field (number of variations, the dates the test ran, and the visitor counts by variation). The deviations line is new in this bundle's own research and is borrowed from the clinical trial-reporting lineage, not from product practice: it traces to CONSORT's own 2010 addition of an item for identifying "any changes to the primary and secondary outcome (endpoint) measures after the trial started," and is corroborated by Booking.com's pre-registration discipline, which fixes the metrics a hypothesis is tested against before the test begins. Deep dive: experiment-readout_companion.md section 3 (Anatomy > Context and Setup). ASK What variants ran, over what dates, against what population, as the design doc specified? Does this match the design doc exactly, or did anything change, such as the traffic split, the dates, or who was eligible, after the test launched? What is the one-line answer on what ran differently from the plan, if anything? GOOD "Ran 2026-09-01 through 2026-09-21, control and treatment at the planned 50/50 split, on the order-history page for all logged-in customers with at least one prior order, as the design doc specified. Deviations from the design doc: none; the test ran the full 21 days with no early stop." WEAK "We ran the test for about three weeks." (no dates, no population, no statement on deviations) TRAP Restating the design doc's hypothesis and metric baselines in full here. Link to it instead; this section's job is what actually ran, not what was planned. -->
{{context_and_setup}}
**What ran differently from the design doc.** {{deviations_from_design_doc}}
## Key Results: Primary Metrics
<!-- WHAT The primary metric's movement against its baseline, stated with its uncertainty. WHY The column shape is read verbatim from Eppo's own export: "Metric Control Test Change (abs) Change (rel) CI Sig?". Optimizely's narrative version asks for the same content in prose. The two change columns, absolute and relative, are corroborated from the clinical trial-reporting lineage, labeled here rather than presented as product practice: CONSORT's 2010 revision recommends that "for binary outcomes, presentation of both relative and absolute effect sizes is recommended." The interval's confidence level is stated in the row, never assumed as a field default; no source read states one level as a convention. Deep dive: experiment-readout_companion.md section 3 (Anatomy > Key Results: Primary Metrics) and section 6, item 8. ASK What is the primary metric's control value and test value? What is the absolute change, and the relative change? What is the confidence interval, and at what confidence level (state the level in the row)? Is the change significant at that level? PRIORITY List the primary metric row or rows first; the design doc capped these at one or two, and this table should not grow past what the design doc actually named as primary. ROW HINT A good row names the metric, both values, both change columns, an interval with its stated confidence level, and a significance call. A weak row reports a change with no interval, so a reader cannot tell it from noise. GOOD | 7-day reorder rate | 11.0% | 11.3% | +0.3 pts | +2.7% | -0.6 to +1.2 pts (95% CI) | No | WEAK | Reorder rate | up slightly | | | | | | TRAP Reporting the relative change alone. A small relative lift on a tiny base can read as dramatic; report both, per the clinical-lineage recommendation this bundle borrows for binary outcomes. A ratio metric's overall value can move opposite to its own numerator and denominator. Check both separately before trusting the ratio's own direction. -->
| Metric | Control | Test | Change (abs) | Change (rel) | CI | Sig? ||---|---|---|---|---|---|---|| {{metric_name}} | {{metric_control}} | {{metric_test}} | {{metric_change_abs}} | {{metric_change_rel}} | {{metric_ci}} | {{metric_significant}} |
## Key Results: Supporting Metrics
<!-- WHAT Every guardrail's movement, reported against the threshold the design doc fixed. WHY This is Eppo's own second results section. A guardrail metric is one that measures what a team does not want to degrade but does not expect to improve; a separate vendor's in-tool surface draws the same line between decision-making metrics and guardrail metrics. Each row here is read against the number the design doc pre-committed, not re-derived at readout time: that pre-commitment is what makes a breach legible rather than a judgment call made after the fact. Deep dive: experiment-readout_companion.md section 3 (Anatomy > Key Results: Supporting Metrics). ASK What is each guardrail's movement, control against test? What threshold did the design doc set for a breach? Did any guardrail breach its threshold, and if so, when? PRIORITY List every guardrail the design doc named, even one that held with no interesting movement; a guardrail silently dropped from the readout cannot be checked by a reader. ROW HINT A good row names the guardrail, the design doc's own threshold, both values, and a held or breached call. A weak row reports a number with no threshold to compare it to. GOOD | Checkout error rate | Design doc: no increase past 0.4% | 0.31% | 0.33% | Held | WEAK | Checkout errors | about the same | | | TRAP Re-deriving a new threshold here instead of using the one the design doc pre-committed. A threshold decided after seeing the data is not a guardrail, it is a rationalization. -->
| Guardrail | Design-doc threshold | Control | Test | Status ||---|---|---|---|---|| {{guardrail_name}} | {{guardrail_threshold}} | {{guardrail_control}} | {{guardrail_test}} | {{guardrail_status}} |
## Validity Checks
<!-- WHAT Whether the result can be trusted at all, checked before it is read as a finding: a sample ratio mismatch, a mismatch in a metric's own observation units, a novelty effect read by date, an early look before the test finished, a telemetry change mid-test, and interaction with another live experiment on the same surface. WHY This section's name and scope are this library's own POSITION: no single source names one section with exactly this job, and it is built by combining several individually sourced checks, the same way the design doc's own Validity Pre-Commitments section was built. Each check reports against the design doc's pre-commitment where one exists: an uneven traffic split invalidates the comparison itself; a decline in the lift across date segments is a named indicator of a novelty effect; reading results before the pre-committed duration ends inflates the reported significance level; and two sources note that interaction with another concurrently running experiment is worth designing around even though vendors report it is rare in practice. Deep dive: experiment-readout_companion.md section 3 (Anatomy > Validity Checks). ASK Did the sample ratio match the configured split on every scheduled check? Did any metric's own observation unit shift mid-test? Did the lift decay across date segments, which would flag a novelty effect? Was the result read or acted on before the pre-committed duration ended? Did any telemetry change mid-test? Did this test overlap with another live experiment on the same surface, and if so, how was that handled? PRIORITY Report every check the design doc pre-committed to running, ordered so a check whose failure would most undermine trust in the result comes first. A check the design doc did not pre-commit to is reported as "not pre-committed," not invented after the fact. ROW HINT A good row names the check, states the result plainly (passed, flagged, not applicable), and if flagged, what it changes about how the result should be read. A weak row says "looks fine" with no specific check named. GOOD | Sample ratio mismatch | Passed on every daily read | Configured 50/50; the largest daily gap stayed within the design doc's own chi-squared threshold | WEAK | Validity | Looks OK | No major issues noticed | TRAP Treating a passed sample-ratio check as proof the whole result is trustworthy. Each row here catches a different failure mode; a result needs all of them checked, not just the one that is easiest to run. -->
| Check | Result | Notes ||---|---|---|| {{validity_check_name}} | {{validity_check_result}} | {{validity_check_notes}} |
## Deep-Dive
<!-- WHAT The business questions the decision actually depends on, answered one at a time, with exploratory work kept out; an estimated business impact, stated with its own interval. WHY Eppo's own instruction is to "Structure this as business questions to be answered," and Eppo's own blog states the reason: a report discusses the why, not the what. Only the segments the design doc fixed in advance may bear on the decision; the trap this guards against is named directly by a separate source as a multiple-comparison problem, "slicing the data until it shows what you want to see." The clinical trial-reporting lineage states the same discipline as a rule rather than a warning, labeled here rather than presented as product practice: whenever a subgroup analysis is shown, the EMA's own subgroup guideline requires that "the analysis of the complement subset should also be displayed." Any estimated business impact goes here, stated with its own interval rather than as a bare number. Deep dive: experiment-readout_companion.md section 3 (Anatomy > Deep-Dive). ASK What business question does the decision actually turn on? For any segment the design doc fixed in advance, what did it show, and what did its complement, everyone else, show alongside it? What is the estimated business impact, stated with its own interval rather than a bare number? GOOD "Question: did the button change behavior for customers who had not reordered in 90 or more days, the segment the design doc fixed in advance? Lift in that segment: +1.1 points. Complement (customers who reordered more recently): +0.1 points. Neither is significant. Estimated impact if rolled out: between -$4,200 and +$9,800 in incremental monthly reorder revenue at 90% confidence, a range that includes zero." WEAK "We also looked at a few different customer segments and one looked promising." (no segment named in advance, no complement shown, no interval) TRAP Slicing the data by segments that were not fixed before the test ran, looking for one that moved. The clinical lineage's own caution, borrowed here rather than presented as product practice, is that examining enough subgroups will inevitably surface one that looks beneficial even with no real effect; show the complement every time a subgroup is shown. -->
{{deep_dive}}
**Estimated business impact.** {{business_impact_with_interval}}
## Decision: Ship, Iterate or Stop
<!-- WHAT The outcome the pre-committed decision rule actually produced, including what a null result does and does not rule out, and whether the decision is reversible. WHY "Ship, iterate or stop" is this family's own wording for the three-way choice, stated as the contract's own POSITION, not a phrase any source read here uses verbatim. The nearest conventions in the sources are a pre-committed "ship/no-ship decision," "Win Lose Flat" outcomes, "roll out to more users, end, iterate, etc.," and "inconclusive" or "hypothesis proved." New in this bundle's own research: a null result states what it rules out and what it does not. One source's own framework counts an adequately powered null as a result in its own right rather than a non-result, with its own default action of iterating, abandoning, or shipping if the change is infrastructure-only; a separate account states the same principle from the opposite direction, that a decision not to ship is still a finding. An ASK on the decision's reversibility is traced to a source outside the experimentation canon entirely: most decisions are changeable and reversible, and only a smaller set deserve the slower process a one-way door warrants. Deep dive: experiment-readout_companion.md section 3 (Anatomy > Decision) and section 6, items 2 and 8. ASK Which of ship, iterate or stop does the design doc's pre-committed decision rule actually produce for this result? If the result is a null, what effect size does it rule out, and what smaller effect does it not rule out (the interval's width, not the point estimate, answers this)? Is this decision reversible, and does that change how fast it needs to be made? GOOD "Null. The design doc's decision rule for a null result is to iterate on a different mechanism rather than re-run this exact test unchanged. This result rules out a lift of 1.2 points or more at 95% confidence; it does not rule out a smaller lift the test was not powered to detect. The decision is reversible: the button can be removed with no migration cost, so this is a two-way door." WEAK "It didn't really work so we're not shipping it." (no named outcome category, no statement of what the null rules out, no reversibility note) TRAP Treating a null result as proof the change has no effect. A null on a test powered to detect a 1.2-point lift says nothing about a 0.3-point one; state the interval's width, not just its center. -->
{{decision_narrative}}
**What this result rules out, and what it does not.** {{null_result_scope}}
**Reversibility.** {{decision_reversibility}}
## Decision Owner
<!-- WHAT Who is named as having made the call, written as a field rather than a signature buried in prose. WHY This section is this library's own POSITION, and it is unsourced: no source read names a field for the decision owner. The nearest is Optimizely's "Decision maker," which names a role among the people involved in a test, not a field this document itself carries. This section exists so that whoever that person turns out to be, their name is written down rather than implied. Deep dive: experiment-readout_companion.md section 3 (Anatomy > Decision Owner) and section 6, item 3. ASK Who made this call? Is it the same person the design doc named as decision-maker in its own frontmatter? If not, who, and in what forum was the call made? GOOD "Jordan Alvarez (PM, Checkout), the decision-maker named in the design doc's frontmatter, made this call at the 2026-09-22 experiment review." WEAK "The team decided to stop." (no named person, no venue, cannot be traced back to the pre-committed decision-maker) TRAP Leaving this section's named person different from the frontmatter's decision-maker field with no note of why. If the decision-maker changed, say so; if not, confirm the match rather than letting the two fields silently disagree. -->
{{decision_owner_statement}}
## Next Steps and Owners
<!-- WHAT The concrete actions the decision requires, each with a named owner. WHY Eppo's own filled example assigns an owner to its very first next step. A separate practitioner frames the same section more broadly, as the plan for the project going forward, which is why this section carries not only the immediate action the decision requires but anything else it implies, each with its own owner. Deep dive: experiment-readout_companion.md section 3 (Anatomy > Next Steps and Owners). ASK What does the decision itself require (a rollout, a rollback, a retirement of a tracking field)? What does it imply beyond that immediate action? Who, by name, owns each one? PRIORITY List the action the decision itself requires first, then anything downstream of it. ROW HINT A good row names a concrete action, one owner, and a status. A weak row names a hope with no owner. GOOD | Remove the reorder-button flag and the control path from the codebase | Sam Ruiz (Eng) | Not started | WEAK | Think about next steps | | | TRAP Naming "the team" as the owner. One name per row, the same discipline the Decision Owner section above asks for. -->
| Action | Owner | Status ||---|---|---|| {{next_step_action}} | {{next_step_owner}} | {{next_step_status}} |
## Implementation Details and Analysis Notes
<!-- WHAT The technical detail a future engineer or data scientist would want, kept out of the sections decision-makers actually read. WHY Eppo's own label for this section states its audience directly: this is where the detail goes that other data scientists and engineers would be interested in. The clinical trial-reporting lineage corroborates the same separation from its own, much stricter side: a reporting standard in that lineage states plainly that it addresses the reporting of what was done and what was found, leaving a report's deeper analysis notes outside its own scope by design. Deep dive: experiment-readout_companion.md section 3 (Anatomy > Implementation Details and Analysis Notes). ASK What query, cohort definition, or analysis code would someone need to reproduce this result? What edge case or exclusion is worth recording for whoever runs the next test on this surface? GOOD "Cohort for the inactive-90-day segment in Deep-Dive: `first_order_date < test_start - 90d`. Analysis notebook: {{analysis_notebook_link}}. One account had a refunded order mid-test and was dropped from both arms before the final read." WEAK "See the data team for details." (nothing recorded, nothing reproducible) TRAP Letting exploratory code or queries live only in a notebook nobody else can find. If it matters enough to mention, link it here. -->
{{implementation_details_and_analysis_notes}}
## Further Analysis
<!-- WHAT Interesting findings that do not bear on the decision, kept clearly separate from the sections that do. WHY A separate practitioner's own section of this name is explicitly optional, which is this bundle's own evidence for treating it as an addition rather than a default. It is labeled exploratory for a reason stated directly in the clinical trial-reporting lineage, labeled here rather than presented as product practice: examine enough subgroups, and a treatment with no real benefit at all will inevitably appear disproportionately beneficial in one or more of them. Nothing the decision rests on may be moved here instead of into Deep-Dive. Deep dive: experiment-readout_companion.md section 3 (Anatomy > Further Analysis). ASK Is there anything interesting in the data that the decision above does not depend on? Does it belong here, or did it actually drive the decision and belong in Deep-Dive instead? GOOD "Customers on the mobile app clicked the button at roughly double the rate of desktop customers, though neither arm's primary metric moved enough to change the decision. Worth a mobile-specific follow-up test, not a reason to revisit this one." WEAK "Lots of other interesting stuff in the data we didn't have time to dig into." (nothing named, cannot tell if it was exploratory or decision-relevant) TRAP Parking a result here because it did not turn out favorably rather than because the decision never depended on it. If it bore on the decision, it belongs in Deep-Dive, not here. -->
{{further_analysis}}---title: "Saved Views Adoption Nudge Experiment Readout"experiment_name: "Saved Views Adoption Nudge"design_doc: "../experiment-design-doc/experiment-design-doc_example.md"author: "Priya Nair (PM, Reporting)"decision_maker: "Priya Nair (PM, Reporting)"start_date: "2026-08-03"end_date: "2026-08-31"decision: "Null (stop): shelve the fifth-view prompt rather than rerun it unchanged."readout_date: "2026-09-04"doc_type: experiment-readoutsize: fullsource_template: experiment-readoutsource_template_version: 0.1.0---
> **Worked example.** A filled `experiment-readout`, full variant, reporting the same Saved Views> adoption-nudge test that> [the experiment-design-doc example](../experiment-design-doc/experiment-design-doc_example.md) planned> for Acme Analytics, chained onto that same [PRD](../prd/prd_example.md),> [KPI dashboard](../kpi-dashboard/kpi-dashboard_example.md), [OKRs](../okrs/okrs_example.md) and> [risk register](../risk-register/risk-register_example.md) thread. Dated 2026-09-04, four days after the> test's own end date of 2026-08-31. Its recommended outcome is a null, with the point estimate finishing> above control, chosen so that every commitment the design doc made (the daily sample-ratio check, the> week-by-week novelty read, the guardrail thresholds, and the decision rule's own null branch) is actually> exercised by this readout rather than skipped past on an easy win.>> All figures, account counts, and event names are illustrative.
# Saved Views Adoption Nudge Experiment Readout
## Executive Summary
Decision: stop. The fifth-view prompt did not raise weekly Saved Views adoption against the no-promptcontrol by a margin the platform calls statistically significant over the four weeks. Treatment finishedtwo points above control, and that gap sits inside an interval wide enough to still reach zero, which readsas inconclusive rather than as either side winning outright. The finding worth keeping is narrow: theprompt tested the design doc's explanation that analysts had not noticed the Views menu, and a nudge largeenough to matter for that explanation did not show up. The risk register's own explanation for R-04, deephabits in the legacy report flow, was not what this test measured. The fifth-view prompt is shelved ratherthan rerun unchanged, and a different mitigation for R-04 goes to the next Reporting Platform Modernizationsteering review.
## Context and Setup
Ran 2026-08-03 through 2026-08-31, the full four weeks the design doc specified, with no early stop.Control (no prompt) and treatment (the fifth-view prompt) ran 50/50, randomized by customer account ratherthan by individual analyst, exactly as the design doc fixed. 231 accounts were assigned over the window,117 to control and 114 to treatment, against the roughly 230 accounts the design doc estimated as theeligible population; the six design-partner pilot analysts and their accounts stayed excluded throughout,as planned. Full plan:[Saved Views Adoption Nudge Experiment Design Doc](../experiment-design-doc/experiment-design-doc_example.md).
**What ran differently from the design doc.** Nothing. Eligibility, allocation, the prompt's owntrigger condition, and the four-week duration all matched the design doc exactly; no mid-test change wasmade to any of them.
## Key Results: Primary Metrics
| Metric | Control | Test | Change (abs) | Change (rel) | CI | Sig? ||---|---|---|---|---|---|---|| Saved Views adoption (share of Recurring Analysts using a saved view weekly) | 43% | 45% | +2 pts | +5% | -11 to +15 pts (95% CI) | No |
The interval is at 95%, the counterpart of the 5% significance level the design doc fixed as the platform'sstandard setting, with 80% power, rather than a level this test chose for itself. At that level the lift is not significant, and theinterval's own width, 26 points end to end, is the real story: it is wide enough to contain both ameaningful loss and a meaningful gain, which is the direct consequence of sizing this test against roughly230 accounts rather than a larger population.
## Key Results: Supporting Metrics
| Guardrail | Design-doc threshold | Control | Test | Status ||---|---|---|---|---|| Weekly active analysts | Not below 480 at any point during the test | 495 (pre-test) | lowest daily reading, either arm combined: 487 | Held || Dashboard load error rate | Not above its pre-test level | pre-test baseline | stayed at the pre-test level throughout | Held || Shared-view permission incidents | Zero, either arm | 0 | 0 | Held |
No guardrail came close to its threshold at any point in the four weeks, so none of the three triggered thedesign doc's early-stop rule.
## Validity Checks
| Check | Result | Notes ||---|---|---|| Sample ratio mismatch | Checked daily, no flag raised | The realized 117/114 split stayed under the platform's chi-squared threshold on all 29 scheduled daily reads || Novelty effect (week-1 lift against weeks 2 through 4) | Flagged | Week-1 lift was +9 points, falling to +4, +3, and +2 over weeks 2 through 4; a decay toward zero, per the design doc's own pre-commitment, so the early lift is not counted as the finding and the week-4 read above governs instead || Early look | Not applicable | No decision-bearing look was taken before 2026-08-31, as pre-committed; the only reads before then confirmed the `nudge_variant_assigned` event was firing || Metric observation-unit mismatch | Passed | The adoption metric's denominator, Recurring Analysts in each arm's accounts, stayed balanced between the arms, and neither it nor the weekly-active-analysts count changed shape during the run || Telemetry change mid-test | Passed | `view_saved`, `view_switched`, `view_set_default`, `view_shared`, `view_load_error`, and `nudge_variant_assigned` were unchanged for the full window || Interaction with other live experiments | Not applicable | No other experiment ran on the dashboard surface during this window; the design-partner pilot's own accounts were already excluded from the eligible population above |
## Deep-Dive
The design doc pre-committed to no segment beyond the account-level control and treatment split itself; theeligible population was judged too small to slice further without turning each slice into its ownunderpowered test. No segment analysis beyond that two-arm split was run here, consistent with thatpre-commitment, and the business question the decision actually turns on is answered directly by theprimary-metric table above: does the prompt move adoption enough, at this population's size, to justifyretiring the no-prompt control? It does not, at the level the platform uses.
**Estimated business impact.** N/A. The design doc fixed no revenue or cost baseline for Saved Viewsadoption; the metric itself, not a dollar figure, is what the decision rule above reads against.
## Decision: Ship, Iterate or Stop
Null, per the design doc's own decision rule: no statistically significant difference between the arms atthe end of the full four weeks. Worth stating plainly, because the design doc's two commitments disagree onthis exact outcome: its hypothesis said it would fail only if treatment matched or trailed the no-promptcontrol's own figure, and treatment finished two points above control, so the hypothesis, read on its ownterms, did not fail. The difference was not significant, though, and the decision rule, not the hypothesis'sown failure condition, is what governs this document's decision: a null result is shelve-and-reconsider, notre-run-unchanged. That gap between the two commitments is recorded here as a lesson for the next design docto close, rather than resolved quietly in either commitment's favor.
**What this result rules out, and what it does not.** The interval's upper end, +15 points, is the largestlift this result leaves plausible, so anything larger is ruled out, including the 18-point lift the test wassized to detect. What it leaves open is a smaller real lift, anywhere up to that +15, which this populationwas too small to resolve.
**Reversibility.** Reversible. The fifth-view prompt is a client-side flag with no stored state beyond the`nudge_variant_assigned` event and no migration behind it; removing it costs nothing and nothing has to beunwound. That is part of why the decision rule calls for shelving the idea rather than treating this as aone-way door that needs a slower, more deliberative process.
## Decision Owner
Priya Nair (PM, Reporting) called the stop at the 2026-09-03 experiment review. That is the same name androle the design doc's heading fields already carry for this test, so planning and readout stayed with oneperson rather than handing off between them.
## Next Steps and Owners
| Action | Owner | Status ||---|---|---|| Shelve the fifth-view prompt; no re-run of this exact test | Priya Nair (PM, Reporting) | Done || Bring the null finding to the next Reporting Platform Modernization steering review and weigh a different mitigation for R-04 | Priya Nair (PM, Reporting) | Not started || Mark `nudge_variant_assigned` for retirement in the tracking plan | Lee Zhang (Data Eng) | Not started |
## Implementation Details and Analysis Notes
Cohort: every account carrying a `nudge_variant_assigned` event, joined to the entitlements database on`account_id`. Adoption is read from `view_saved`, `view_switched`, and `view_set_default`, attributed to theviewing account rather than the authoring one, matching the KPI dashboard's own adoption formula and itsown known limitation that a shared view counts for the viewer, not the creator. Each test week'sadoption is read on the dashboard's own trailing-7-day window, taken on that week's last day, so the readoutand the dashboard count a weekly user the same way. No account was droppedfrom either arm after assignment, and no refund, downgrade, or account-merge event touched either arm duringthe run.
## Further Analysis
Accounts with ten or more Recurring Analysts showed a larger raw gap between arms (treatment ahead ofcontrol by roughly 4 points) than accounts with fewer analysts (roughly 1 point). Neither slice waspre-registered in the design doc, and neither one changes the decision above; raised here only because it isa plausible lead for whoever designs the next mitigation for R-04, not a finding this readout rests on. Anaccount-size segment pre-registered in advance would be the honest way to chase it, rather than reopeningthis test's own data for a closer look.Provenance
Section titled “Provenance”The reasoning, the history and every source, in the repository:
- Companion - the long-form argument: why these sections, where the sources disagree, and what the bundle refuses to claim
- History - what changed in this bundle, and when
- Research log - every source consulted, with what each one actually supports
- Catalog metadata - the machine-readable record this page is generated from
Catalog record: 11 sections across 1 format(s).