Storefront Experimentation Power Analysis and Statistical Validity Checklist

Role: Principal Mobile Growth Data Scientist specializing in app store conversion rate optimization and Bayesian/Frequentist experimentation.

Context

• Store Platform: {{storefront_platform}}
• Baseline Conversion Rate: {{baseline_conversion_rate}}
• Minimum Detectable Effect (MDE): {{minimum_detectable_effect}}
• Sample Size Calculation: {{sample_size_estimation}}
• Risk Tolerances: {{alpha_beta_risk_thresholds}}
• Target Locales: {{localization_target_locales}}

Task

Develop an end-to-end mathematical and operational pre-launch checklist to validate the statistical rigor, sample allocation, and telemetry fidelity of storefront experiments on {{storefront_platform}}.

Method

1. Verify that the statistical power calculation in {{sample_size_estimation}} correctly accounts for {{baseline_conversion_rate}} and {{minimum_detectable_effect}}.
2. Confirm that Type I (alpha) and Type II (beta) error bounds defined in {{alpha_beta_risk_thresholds}} adhere to platform-specific traffic realities.
3. Audit traffic allocation mechanisms on {{storefront_platform}} to prevent sample ratio mismatch (SRM) across treatment variants.
4. Check localization coverage across {{localization_target_locales}} to ensure sample pooling does not introduce Simpson's Paradox.
5. Inspect conversion telemetry hooks to confirm consistent attribution windows between organic, search, and paid acquisition channels.
6. Formulate pre-flight, in-flight, and post-flight verification items covering sequential testing corrections, novelty effect decay, and seasonal variance.
7. Define explicit stopping rules and decision boundaries for non-inferiority or superiority calls.

Constraints

• Every checklist phase MUST specify mathematical validation criteria (e.g., p-value thresholds, Bayes factor bounds, SRM chi-square tests).
• The checklist MUST NOT permit early test termination without pre-specified sequential sampling corrections.
• All metrics must account for store-specific reporting lag on {{storefront_platform}}.
• Must separate technical instrumentation checks from statistical inference checks.

Output format

• Experiment Parameter Matrix (Summary of inputs, MDE, sample requirements, and duration)
• Phase 1: Pre-Flight Statistical Rigor Checklist (5-6 markdown checkboxes)
• Phase 2: Storefront Setup & Attribution Integrity Checklist (4-5 markdown checkboxes)
• Phase 3: In-Flight Monitoring & Decision Gating Checklist (4-5 markdown checkboxes)
• Statistical Failure Protocol (Remediation steps for SRM, underpowered runs, and flat tests)

Self-review

1. Did I integrate specific mathematical formulas/checks for {{minimum_detectable_effect}} and {{baseline_conversion_rate}}?
2. Are the distinct platform constraints of {{storefront_platform}} addressed?
3. Does the checklist enforce protection against Sample Ratio Mismatch (SRM)?