Feature → Metric → Experiment: A 60‑Minute Workflow to Turn Ideas into A/B Tests and Actionable Data

Start by writing one sentence that explains the user problem the feature solves and the measurable user action you expect to change. Example: “Allow saving searches so weekly active users (WAU) who use saved searches increase by X%.” Narrowing the outcome avoids fuzzy success criteria.

Pick a single primary metric (proportion or continuous). If the expected effect is a change in behavior (signup, click, upgrade), use a proportion metric (conversion rate). If it’s engagement (time spent, sessions), use a continuous metric. This selection determines what sample size approach and statistical test you’ll use. Choose secondary metrics only to detect negative side‑effects (safety checks).

bullets':['Write a one‑sentence outcome statement: problem → feature → expected user action.','Pick one primary metric (conversion/proportion or engagement/continuous).','Limit secondaries to safety checks (e.g., error rate, retention).'],

sourceIds':'([statsig.com)']},{

Write a testable hypothesis in the pattern: “If we [feature change], then [primary metric] will change from baseline B to target T (MDE).” Choose a realistic Minimum Detectable Effect (MDE) tied to business value: larger MDEs require far less traffic and are pragmatic for early teams.

Design the cheapest valid experiment that isolates the feature’s effect. Options by engineering cost: feature flag rollout (most robust if you can bucket users), mocked UI behind a toggle, gated beta sent to a segment, or a landing‑page + signup flow to measure interest before building. Use a feature flag if you want production realism; use a landing page for demand validation before any engineering work.

bullets':['Hypothesis template: If we [change], then primary metric moves from baseline to target (pick MDE).','Experiment options: feature flag, mock UI, gated beta, landing page + signup.','Prefer the lowest engineering cost that still isolates causality.'],

sourceIds':'([statsig.com)']},{

You don’t need a complex power analysis to make pragmatic decisions. Use these heuristics: if baseline conversion is low (<1%) and you’re looking for small lifts (<10% relative), you likely need lots of traffic — consider raising the MDE (target a business‑meaningful lift) or switch to a high‑signal metric. For mid‑range baselines (1–10%), aim for MDEs of 10–25% for tests that finish in weeks, not months.

Practical shortcut: use any online sample size calculator to estimate visitors per variant (Statsig, AB Tasty, Evan Miller). As a rule‑of‑thumb for early startups: target 80% power and 5% significance, and pick an MDE that ties to revenue or retention (e.g., a 10% bump in trial to paid conversion). If traffic can’t meet sample size, use a gated beta or landing page funnel to increase signal or validate demand qualitatively first.

bullets':['If baseline <1% — prefer larger MDE or different metric (engagement).','Aim for 80% power and 5% alpha; use online calculators (Statsig, AB Tasty).','If traffic is insufficient, use gated beta or landing page validation.'],

sourceIds:'([statsig.com)']},{

Set up clear stop rules before launching: planned duration or required sample size, and safety checks to abort on negative side‑effects (error rates, latency, critical funnels). Do not peek and stop the test based on interim p‑values — follow the pre‑registered rule or use sequential testing methods if you plan to peek.

Log randomization keys, ensure consistent exposure (sticky bucketing), and delete experiment toggles after rollout. If you used a feature flag, keep it short‑lived: when a feature is rolled out permanently, remove the flag to avoid tech debt. Track both the primary metric and the pre‑selected safety secondaries throughout the run.

bullets':['Pre-register stop rules: sample size or planned duration and safety abort conditions.','Avoid peeking; use sequential testing if you must look early.','Keep feature flags short‑lived; remove after decision.'],

sourceIds':'([marmenlind.com)']},{

Apply a simple two‑axis decision rule: 1) statistical outcome on the primary metric (win, null, loss) using your pre‑registered test rule; 2) business impact and risk (expected revenue, implementation cost, technical debt). If the test is a statistically significant win and business impact is positive, keep and roll out. If null but sample was underpowered for a business‑meaningful MDE, iterate with a higher‑signal experiment or scrap if cost is high.

For losses or meaningful negative side‑effects, cut the feature and document learnings. For borderline results, use an escalation path: run an extended or larger follow‑up only if the projected ROI from a true uplift justifies the extra engineering and time. Record the hypothesis, sample size, results, and decision in a short experiment log so your team can learn fast and avoid repeating the same guesswork.

bullets':['Decision axes: statistical result (win/null/loss) × business impact (ROI, cost, risk).','Keep: stat sig win + positive ROI.','Iterate: underpowered null but plausible ROI.','Cut: loss or negative side‑effects.','Log all experiments for repeatability and learning.'],

sourceIds':'([arxiv.org)']}],