Measure Dashboard Requirements: Brainshelf Resurface

Scenario

Before the A/B test began, Priya M. asked Chloe B. to build the Resurface experiment dashboard in Amplitude so the team could monitor the experiment in real time and have the results analysis ready on test completion day. The dashboard needed to serve two audiences: Priya and the product team (daily monitoring during the test) and Marco S. (weekly executive check-ins requiring a single-screen summary). Chloe documented the requirements to ensure the dashboard was built once, correctly, rather than assembled ad hoc during the test window.

Source Notes:

Stephen Few, “Information Dashboard Design” (perceptualedge.com) . the dashboard design principles applied in the layout specification; Few’s guidance on minimizing chart junk, using small multiples for comparison, and placing the most important metric in the top-left position informed the visualization specifications.
Amplitude, “Dashboard Best Practices” (amplitude.com/blog/analytics-dashboards) . the Amplitude-specific guidance on chart types, cohort definitions, and filter configuration used to translate the requirements into implementable specs.
Edward Tufte, “The Visual Display of Quantitative Information” (edwardtufte.com) . the data-ink ratio principle applied to the chart specifications; Tufte’s emphasis on maximizing the data-to-ink ratio influenced the decision to use line charts over bar charts for time-series metrics and to avoid decorative chart elements.

Prompt

measure-dashboard-requirements

resurface experiment dashboard for amplitude. need it ready before
the a/b test starts (mar 9).

two audiences:
1. product team (priya, chloe, alex, jordan) . daily monitoring
   during the 4-week test
2. marco (ceo) . weekly exec check-in, needs a single-screen summary

questions the dashboard should answer:
- is the treatment group returning more than control?
- are users clicking items in the digest?
- is the unsubscribe rate within the guardrail?
- what's the opt-in funnel conversion rate?
- are there segment differences (library size, cadence)?

charts i want:
1. 7-day return rate trend (treatment vs control, weekly)
2. email CTR trend (daily)
3. opt-in funnel (card viewed → opted in)
4. unsubscribe rate trend (weekly, with guardrail line)
5. segment breakdown table (library size, cadence)

filters: date range, experiment variant, library size segment.

Output

Dashboard Requirements: Resurface Experiment Dashboard

Overview

Dashboard Name: Resurface Experiment (RSF-EXP-001) Requestor: Priya M., Product Manager Date: February 28, 2026 Priority: High Target Delivery: March 7, 2026 (2 days before A/B test start)

Purpose and Questions

Primary Questions This Dashboard Answers

Is the treatment group’s 7-day return rate meaningfully higher than control, and is the difference growing or shrinking over time?
Are opted-in users engaging with the digest content (clicking items), and is engagement stable or decaying over the test window?
Is the email unsubscribe rate within the 2%-per-week guardrail, or is it trending toward the threshold?
What is the opt-in funnel conversion rate (card viewed → opted in), and is it meeting the 10% minimum target?
Do segment differences (heavy vs. light savers, daily vs. 3x/week cadence) suggest that the treatment effect varies by user type?

Decisions This Will Inform

Ship/iterate/kill decision for the Resurface feature at experiment close (April 5, 2026)
Whether to extend the experiment if results are directionally positive but not yet statistically significant
Whether to pause the experiment if the unsubscribe guardrail is breached
Whether to adjust the opt-in prompt if the opt-in rate is below the 10% target after week 1

What This Dashboard Is NOT For

Operational monitoring of the digest send pipeline (send failures, retry rates, Resend API health) . these are tracked in the Datadog operational dashboard, not Amplitude
Individual user debugging (“why didn’t user X receive their digest?”) . use Amplitude’s user timeline for individual-level investigation
Post-experiment deep analysis (regression models, causal inference) . the dashboard provides descriptive metrics; post-experiment analysis will be done in a separate notebook

Audience

Audience	Usage Frequency	Primary Questions
Priya M. (PM)	Daily during test	Return rate trend, CTR, unsubscribe rate
Chloe B. (Data)	Daily during test	All metrics; data quality checks
Alex R. (Engineering)	2 - 3x/week	Send success rate, item click distribution
Jordan L. (Growth)	Weekly	Opt-in rate, segment differences
Marco S. (CEO)	Weekly (Monday exec check-in)	Single-screen summary: return rate, CTR, guardrail status

Usage Context

When will this be viewed? Daily at 9:00 AM by Priya and Chloe (after the morning digest cycle completes). Weekly on Mondays by Marco (5-minute screen share in the exec check-in).

What device/format? Desktop browser (Amplitude web app). Marco views via screen share . the dashboard must be readable at a glance on a projected screen without scrolling.

Key Metrics

Metric 1: 7-Day Return Rate (Primary Experiment Metric)

Attribute	Value
Business Definition	Percentage of users who had at least one session in week N and returned for at least one session in week N+1
Calculation	(Users with ≥1 session in week N+1) / (Users with ≥1 session in week N) × 100, computed per experiment variant
Data Source	Amplitude session events (existing instrumentation) + experiment variant assignment
Granularity	Weekly (computed each Monday for the prior 7-day window)
Current Baseline	18% [fictional]
Target	≥23% treatment (≥5pp lift over control) [fictional]
Notes	Intent-to-treat: all treatment users are included regardless of opt-in status

Metric 2: Email Click-Through Rate (Secondary Metric)

Attribute	Value
Business Definition	Percentage of delivered digest emails where the user clicked at least one item
Calculation	(Digests with ≥1 `resurface_item_clicked` event) / (Total `resurface_digest_sent` events) × 100
Data Source	`resurface_digest_sent` and `resurface_item_clicked` events
Granularity	Daily
Current Baseline	n/a (new metric)
Target	≥15% [fictional]
Notes	Treatment only (control receives no email). Do NOT use open rate . Apple MPP inflates it.

Metric 3: Email Unsubscribe Rate (Guardrail Metric)

Attribute	Value
Business Definition	Percentage of opted-in treatment users who unsubscribe in a given week
Calculation	(`resurface_unsubscribe` events in week N) / (Opted-in users at start of week N) × 100
Data Source	`resurface_unsubscribe` events + `digest_enabled` user property
Granularity	Weekly
Current Baseline	n/a (new metric)
Target	≤2% per week [fictional]
Notes	Guardrail . breaching this threshold triggers experiment pause and review

Metric 4: Opt-In Rate (Diagnostic Metric)

Attribute	Value
Business Definition	Percentage of treatment users who enable the Resurface digest during the test window
Calculation	(Treatment users with `resurface_opt_in` event) / (Total treatment users) × 100
Data Source	`resurface_opt_in` events + experiment variant assignment
Granularity	Cumulative (running total, updated daily)
Current Baseline	n/a
Target	≥10% [fictional]
Notes	Also track the funnel: card_viewed → opted_in for conversion rate

Metrics Summary Table

Metric	Definition	Source	Target
7-day return rate	Weekly return rate by variant	Amplitude sessions	≥23% treatment [fictional]
Email CTR	Daily click-through rate	`digest_sent` + `item_clicked`	≥15% [fictional]
Unsubscribe rate	Weekly unsubscribe % of opted-in users	`unsubscribe` events	≤2%/week [fictional]
Opt-in rate	Cumulative opt-in % of treatment users	`opt_in` events	≥10% [fictional]
Saved item revisit rate	30-day revisit rate by variant	Amplitude item events	Directional improvement

Visualization Specifications

Chart 1: 7-Day Return Rate Trend (Treatment vs. Control)

Attribute	Value
Purpose	Is the treatment group returning more than control, and is the gap growing?
Chart Type	Line chart (two series)
X-Axis	Week (Week 1, Week 2, Week 3, Week 4)
Y-Axis	7-day return rate (%)
Series/Breakdown	Two lines: Control (gray) and Treatment (blue). Include 95% confidence interval shading.
Interactivity	Tooltip on hover shows exact value, sample size, and CI for each point
Position	Top-left (most prominent position . this is the primary metric)

Chart 2: Email CTR Trend (Daily)

Attribute	Value
Purpose	Are users engaging with the digest content, and is engagement stable?
Chart Type	Line chart (single series)
X-Axis	Date (daily)
Y-Axis	Email CTR (%)
Series/Breakdown	Single line: treatment opted-in users. Include a horizontal reference line at 15% (target).
Interactivity	Tooltip shows CTR, total digests sent, total clicks, for each day
Position	Top-right

Chart 3: Opt-In Funnel

Attribute	Value
Purpose	What is the opt-in conversion rate from card viewed to opted in?
Chart Type	Funnel chart (horizontal)
X-Axis	Funnel steps: Treatment Users → Card Viewed → Opted In
Y-Axis	Count and conversion rate (%)
Series/Breakdown	Single funnel; show absolute count and % conversion at each step
Interactivity	Click on a step to see the user list for debugging
Position	Middle-left

Chart 4: Unsubscribe Rate Trend (Weekly)

Attribute	Value
Purpose	Is the unsubscribe rate within the guardrail, or trending toward the threshold?
Chart Type	Bar chart (weekly) with horizontal guardrail reference line
X-Axis	Week (Week 1, Week 2, Week 3, Week 4)
Y-Axis	Unsubscribe rate (%)
Series/Breakdown	Single series: weekly unsubscribe rate. Red horizontal line at 2% (guardrail threshold).
Interactivity	Tooltip shows unsubscribe count, opted-in base, and rate
Position	Middle-right

Chart 5: Segment Breakdown Table

Attribute	Value
Purpose	Does the treatment effect vary by library size or cadence?
Chart Type	Table
X-Axis	n/a (table)
Y-Axis	n/a (table)
Series/Breakdown	Rows: Heavy savers (100+), Medium savers (50 - 99), Light savers (10 - 49), Daily cadence, 3x/week cadence. Columns: Control return rate, Treatment return rate, Delta, p-value, CTR (treatment only).
Interactivity	Sortable columns
Position	Bottom (full width)

Dashboard Layout Sketch

┌─────────────────────────────────────────────────────────┐
│  [KPI: Return Rate]  [KPI: CTR]  [KPI: Unsub Rate]     │
├────────────────────────────┬────────────────────────────┤
│                            │                            │
│  Chart 1: Return Rate      │  Chart 2: Email CTR        │
│  Trend (weekly, by variant)│  Trend (daily)             │
│                            │                            │
├────────────────────────────┬────────────────────────────┤
│                            │                            │
│  Chart 3: Opt-In Funnel    │  Chart 4: Unsubscribe      │
│  (treatment only)          │  Rate (weekly + guardrail)  │
│                            │                            │
├────────────────────────────┴────────────────────────────┤
│                                                         │
│        Chart 5: Segment Breakdown Table                 │
│                                                         │
└─────────────────────────────────────────────────────────┘

Filters and Segments

Global Filters

Filter	Type	Default Value	Options
Date Range	Date picker	Full experiment window (Mar 9 - Apr 5, 2026)	Custom range, last 7 days, last 14 days
Experiment Variant	Dropdown	All (both variants)	All, Control, Treatment
Library Size Segment	Dropdown	All	All, Heavy (100+), Medium (50 - 99), Light (10 - 49)

Chart-Specific Filters

Chart	Filter	Type
Chart 2 (Email CTR)	Cadence	Dropdown: All, Daily, 3x/week
Chart 5 (Segment Table)	Metric toggle	Dropdown: Return Rate, CTR, Opt-In Rate

Segment Definitions

Segment Name	Definition	Use Case
Heavy savers	Users with 100+ saved items at experiment start	Analyze whether larger libraries produce stronger treatment effects
Medium savers	Users with 50 - 99 saved items at experiment start	Middle segment for comparison
Light savers	Users with 10 - 49 saved items at experiment start	Analyze whether small libraries limit topic-matching effectiveness
Daily cadence	Opted-in treatment users on daily delivery	Compare engagement by cadence
3x/week cadence	Opted-in treatment users on 3x/week delivery	Compare engagement by cadence

Data Sources

Primary Sources

Source	Type	Owner	Latency	Quality Notes
Amplitude events (client-side)	Event stream	Chloe B.	Near real-time (~30 seconds)	Standard Amplitude SDK; no known quality issues
Amplitude events (server-side)	Event stream	Chloe B.	Near real-time (~30 seconds)	Fired by cron job and redirect endpoint; depends on server availability
Amplitude Experiment (variant assignment)	Feature flag	Chloe B.	Instant (cached client-side)	Consistent bucketing by user_id

Data Pipeline Requirements

Refresh Frequency: Near real-time (Amplitude default) Refresh Time: No batch processing required; all events stream to Amplitude in real time Historical Data Needed: Experiment window only (March 9 - April 5, 2026); 2-week pre-experiment baseline for return rate comparison Data Retention: 12 months (standard Amplitude retention)

Data Quality Considerations

Apple Mail Privacy Protection inflates resurface_digest_opened events; the dashboard does NOT include an open rate metric for this reason
The item_position property on resurface_item_clicked events should be validated: values must be 1 - 5; any values outside this range indicate an instrumentation bug
Server-side events (digest_sent, digest_skipped) are dependent on the cron job’s availability; if the cron job fails, these events will not fire, creating a data gap that must be reconciled with the Resend delivery logs

Access and Permissions

Access Levels

Role/Group	Access Level	Restrictions
Product team (Priya, Chloe, Alex, Jordan, Dan)	Full access	None
Marco S. (CEO)	View only	Cannot modify charts or filters
Engineering (Jess, Sam)	Full access	None

Sensitive Data

Data Element	Sensitivity	Handling
user_id	Internal identifier	Visible in Amplitude user timelines; not exposed in dashboard aggregations
destination_url	Potentially PII	Query parameters stripped at instrumentation time; URLs visible only in individual event detail views

Alerts and Thresholds

Condition	Threshold	Action	Recipients
Weekly unsubscribe rate exceeds guardrail	>2% in any single week [fictional]	Slack alert to #resurface	Priya M., Chloe B.
Daily email CTR drops below 5% [fictional] for 3 consecutive days	<5% CTR for 3 days	Slack alert to #resurface	Priya M., Chloe B.
Opt-in rate below 5% at day 7	<5% cumulative opt-in after 7 days [fictional]	Slack alert; trigger contingency plan review	Priya M., Jordan L.

Acceptance Criteria

All 5 charts render correctly with test data before the experiment starts (Mar 7 deadline)
KPI cards at the top show current return rate (by variant), current CTR, and current unsubscribe rate
Global filters (date range, variant, library size) apply correctly to all charts
Dashboard loads in under 5 seconds on desktop
Marco can access the dashboard in view-only mode
Alerts fire correctly when test thresholds are breached (validated with test events)
2-week pre-experiment baseline for return rate is visible in Chart 1 (as a reference period before the treatment line begins)

Open Questions

Should the dashboard include a “days remaining” countdown or “experiment progress” indicator for the 4-week test window?
Should Chart 1 (return rate trend) show daily or weekly granularity? Weekly is more stable but daily provides earlier signal. Current spec: weekly.

Appendix

Brainshelf Product Health Dashboard (existing) . overall MAU, retention, save rate
Resend Delivery Dashboard (Datadog) . email send success rate, bounce rate, API latency

Reference Documents

Resurface experiment design (Measure phase)
Resurface instrumentation spec (Measure phase)
Resurface hypothesis document (Define phase)

Requirements version 1.0. Update as needs evolve.