Before the A/B test began, Priya M. asked Chloe B. to build the Resurface experiment dashboard in Amplitude so the team could monitor the experiment in real time and have the results analysis ready on test completion day. The dashboard needed to serve two audiences: Priya and the product team (daily monitoring during the test) and Marco S. (weekly executive check-ins requiring a single-screen summary). Chloe documented the requirements to ensure the dashboard was built once, correctly, rather than assembled ad hoc during the test window.
Source Notes:
Stephen Few, “Information Dashboard Design” (perceptualedge.com) . the dashboard design principles applied in the layout specification; Few’s guidance on minimizing chart junk, using small multiples for comparison, and placing the most important metric in the top-left position informed the visualization specifications.
Amplitude, “Dashboard Best Practices” (amplitude.com/blog/analytics-dashboards) . the Amplitude-specific guidance on chart types, cohort definitions, and filter configuration used to translate the requirements into implementable specs.
Edward Tufte, “The Visual Display of Quantitative Information” (edwardtufte.com) . the data-ink ratio principle applied to the chart specifications; Tufte’s emphasis on maximizing the data-to-ink ratio influenced the decision to use line charts over bar charts for time-series metrics and to avoid decorative chart elements.
Prompt
/dashboard-requirements
resurface experiment dashboard for amplitude. need it ready before
the a/b test starts (mar 9).
two audiences:
1. product team (priya, chloe, alex, jordan) . daily monitoring
during the 4-week test
2. marco (ceo) . weekly exec check-in, needs a single-screen summary
questions the dashboard should answer:
- is the treatment group returning more than control?
- are users clicking items in the digest?
- is the unsubscribe rate within the guardrail?
- what's the opt-in funnel conversion rate?
- are there segment differences (library size, cadence)?
charts i want:
1. 7-day return rate trend (treatment vs control, weekly)
2. email CTR trend (daily)
3. opt-in funnel (card viewed → opted in)
4. unsubscribe rate trend (weekly, with guardrail line)
Dashboard Name: Resurface Experiment (RSF-EXP-001)
Requestor: Priya M., Product Manager
Date: February 28, 2026
Priority: High
Target Delivery: March 7, 2026 (2 days before A/B test start)
Purpose and Questions
Primary Questions This Dashboard Answers
Is the treatment group’s 7-day return rate meaningfully higher than control, and is the difference growing or shrinking over time?
Are opted-in users engaging with the digest content (clicking items), and is engagement stable or decaying over the test window?
Is the email unsubscribe rate within the 2%-per-week guardrail, or is it trending toward the threshold?
What is the opt-in funnel conversion rate (card viewed → opted in), and is it meeting the 10% minimum target?
Do segment differences (heavy vs. light savers, daily vs. 3x/week cadence) suggest that the treatment effect varies by user type?
Decisions This Will Inform
Ship/iterate/kill decision for the Resurface feature at experiment close (April 5, 2026)
Whether to extend the experiment if results are directionally positive but not yet statistically significant
Whether to pause the experiment if the unsubscribe guardrail is breached
Whether to adjust the opt-in prompt if the opt-in rate is below the 10% target after week 1
What This Dashboard Is NOT For
Operational monitoring of the digest send pipeline (send failures, retry rates, Resend API health) . these are tracked in the Datadog operational dashboard, not Amplitude
Individual user debugging (“why didn’t user X receive their digest?”) . use Amplitude’s user timeline for individual-level investigation
Post-experiment deep analysis (regression models, causal inference) . the dashboard provides descriptive metrics; post-experiment analysis will be done in a separate notebook
Audience
Audience
Usage Frequency
Primary Questions
Priya M. (PM)
Daily during test
Return rate trend, CTR, unsubscribe rate
Chloe B. (Data)
Daily during test
All metrics; data quality checks
Alex R. (Engineering)
2 - 3x/week
Send success rate, item click distribution
Jordan L. (Growth)
Weekly
Opt-in rate, segment differences
Marco S. (CEO)
Weekly (Monday exec check-in)
Single-screen summary: return rate, CTR, guardrail status
Usage Context
When will this be viewed?
Daily at 9:00 AM by Priya and Chloe (after the morning digest cycle completes). Weekly on Mondays by Marco (5-minute screen share in the exec check-in).
What device/format?
Desktop browser (Amplitude web app). Marco views via screen share . the dashboard must be readable at a glance on a projected screen without scrolling.
All, Heavy (100+), Medium (50 - 99), Light (10 - 49)
Chart-Specific Filters
Chart
Filter
Type
Chart 2 (Email CTR)
Cadence
Dropdown: All, Daily, 3x/week
Chart 5 (Segment Table)
Metric toggle
Dropdown: Return Rate, CTR, Opt-In Rate
Segment Definitions
Segment Name
Definition
Use Case
Heavy savers
Users with 100+ saved items at experiment start
Analyze whether larger libraries produce stronger treatment effects
Medium savers
Users with 50 - 99 saved items at experiment start
Middle segment for comparison
Light savers
Users with 10 - 49 saved items at experiment start
Analyze whether small libraries limit topic-matching effectiveness
Daily cadence
Opted-in treatment users on daily delivery
Compare engagement by cadence
3x/week cadence
Opted-in treatment users on 3x/week delivery
Compare engagement by cadence
Data Sources
Primary Sources
Source
Type
Owner
Latency
Quality Notes
Amplitude events (client-side)
Event stream
Chloe B.
Near real-time (~30 seconds)
Standard Amplitude SDK; no known quality issues
Amplitude events (server-side)
Event stream
Chloe B.
Near real-time (~30 seconds)
Fired by cron job and redirect endpoint; depends on server availability
Amplitude Experiment (variant assignment)
Feature flag
Chloe B.
Instant (cached client-side)
Consistent bucketing by user_id
Data Pipeline Requirements
Refresh Frequency: Near real-time (Amplitude default)
Refresh Time: No batch processing required; all events stream to Amplitude in real time
Historical Data Needed: Experiment window only (March 9 - April 5, 2026); 2-week pre-experiment baseline for return rate comparison
Data Retention: 12 months (standard Amplitude retention)
Data Quality Considerations
Apple Mail Privacy Protection inflates resurface_digest_opened events; the dashboard does NOT include an open rate metric for this reason
The item_position property on resurface_item_clicked events should be validated: values must be 1 - 5; any values outside this range indicate an instrumentation bug
Server-side events (digest_sent, digest_skipped) are dependent on the cron job’s availability; if the cron job fails, these events will not fire, creating a data gap that must be reconciled with the Resend delivery logs
Access and Permissions
Access Levels
Role/Group
Access Level
Restrictions
Product team (Priya, Chloe, Alex, Jordan, Dan)
Full access
None
Marco S. (CEO)
View only
Cannot modify charts or filters
Engineering (Jess, Sam)
Full access
None
Sensitive Data
Data Element
Sensitivity
Handling
user_id
Internal identifier
Visible in Amplitude user timelines; not exposed in dashboard aggregations
destination_url
Potentially PII
Query parameters stripped at instrumentation time; URLs visible only in individual event detail views
Alerts and Thresholds
Condition
Threshold
Action
Recipients
Weekly unsubscribe rate exceeds guardrail
>2% in any single week [fictional]
Slack alert to #resurface
Priya M., Chloe B.
Daily email CTR drops below 5% [fictional] for 3 consecutive days
<5% CTR for 3 days
Slack alert to #resurface
Priya M., Chloe B.
Opt-in rate below 5% at day 7
<5% cumulative opt-in after 7 days [fictional]
Slack alert; trigger contingency plan review
Priya M., Jordan L.
Acceptance Criteria
All 5 charts render correctly with test data before the experiment starts (Mar 7 deadline)
KPI cards at the top show current return rate (by variant), current CTR, and current unsubscribe rate
Global filters (date range, variant, library size) apply correctly to all charts
Dashboard loads in under 5 seconds on desktop
Marco can access the dashboard in view-only mode
Alerts fire correctly when test thresholds are breached (validated with test events)
2-week pre-experiment baseline for return rate is visible in Chart 1 (as a reference period before the treatment line begins)
Open Questions
Should the dashboard include a “days remaining” countdown or “experiment progress” indicator for the 4-week test window?
Should Chart 1 (return rate trend) show daily or weekly granularity? Weekly is more stable but daily provides earlier signal. Current spec: weekly.
Appendix
Related Dashboards
Brainshelf Product Health Dashboard (existing) . overall MAU, retention, save rate