ASO Experiment Catalogue: 12 Hypotheses, Signal Thresholds & a 60‑Day Rotation Plan

Store listing experiments (Apple’s Product Page Optimization and Google Play Store experiments) behave differently from on-site web A/B tests: traffic is noise-heavy, effects are often small, and both platforms impose limits on concurrent tests and rollout. Before you start, pick one primary metric (browse-to-install conversion rate or impressions-to-installs depending on platform) and one guardrail metric (1‑day retention or crash rate) to catch regressions.

Enforce test discipline: change one primary visual or metadata element per experiment (icon OR first screenshot headline OR title) to keep interpretation simple; run each variant long enough to accumulate a reliable sample; and avoid peeking (stopping early when a result looks good) — that inflates false positives. Where possible, use platform-native experiments; supplement with analytics and UTM-tagged acquisition campaigns if you need more control or faster signal.

Prioritization rule: order tests by expected impact × ease (traffic required + creative cost). Start with bold visual assets that move browse-to-install (icon and first screenshot headline), then move to metadata with broader reach (title, subtitle/short description), then preview video and localization variants. Each hypothesis below includes the single variable to change, the rationale, and a simple success threshold.

The list below is ordered for a 60‑day handoff where early tests deliver directional wins and later tests refine localization and text-based discoverability.

Localization is two-fold: translated metadata and culture-aware visuals. Test localized screenshots and locale-specific value propositions against a single-language control. Prioritize locales by top-10 markets for your app by installs or revenue rather than vanity lists — localized visuals often move conversion more than translated text alone.

For text-heavy SEO hypotheses (keywords in title, subtitle, or short description), treat them as medium-impact, longer-duration experiments. Keyword movement sometimes shows up in store ranking reports after weeks; pair these tests with keyword tracking and don’t expect immediate large conversion uplifts solely from keyword swaps.

Decision-ready thresholds: treat tests as having three outcome zones — Win, Inconclusive, Revert. Win if effect size exceeds your minimum detectable effect (MDE) with a pre-specified alpha (0.05) and power (0.8) and guardrail metrics show no regression. Revert if a variant reduces conversion by a pre-specified negative threshold (for example >3% absolute drop) or harms a guardrail. Otherwise mark Inconclusive and schedule retest or escalation.

Sample-size heuristics founders can use quickly: if baseline browse-to-install is 5% and you want to detect a relative lift of 10% (i.e., to 5.5%), you’ll need large samples — often tens of thousands of visitors per variant. For quicker directional tests aim for MDEs of 8–12% to keep required sample sizes plausible. Use a sample-size calculator (Statsig, SampleSizer) for exact numbers and always account for platform split behavior (not all impressions are eligible to see variants).

High-level cadence: run 4 two-week experiments sequentially (weeks 1–8) with a brief analysis window after each, then use the remaining 4 weeks for parallel localization and metadata follow-ups where platform limits allow. The earliest two-week experiments should be the icon and first screenshot headline — these are low-cost creatives with high potential impact and quick learnings.

Practical handoff checklist for each 14-day experiment: 1) one-page brief (hypothesis, primary metric, MDE, sample-size estimate, guardrail), 2) assets (Figma files + exported variants), 3) store setup steps and tracking instructions, 4) analysis template with A/A baseline, 5) rollback plan. If a test concludes Inconclusive, extend by one full traffic cycle only if pre-specified in the brief; avoid open-ended extensions.