Plan multiple-comparison control for an exploratory analysis
Pre-commit to a correction procedure so a wide exploratory sweep does not produce a selective significance narrative.
A multiple-comparison plan with expected false positives, a justified correction method, and a full-reporting rule.
Role
analytics methodologist
Task
An analyst plans {{test_count}} hypothesis tests on {{dataset_description}} at level {{alpha}}. Estimate how many false positives to expect with no correction, then recommend a correction approach, contrasting family-wise control with false-discovery-rate control for this use case. Specify the reporting rule that prevents cherry-picking the surviving results.
Context
The analysis is exploratory, the results will feed a launch decision, and there is pressure to report only what reached significance.
Inputs
- Dataset description: {{dataset_description}}
- Planned test list and hypothesis families
- Alpha level {{alpha}} and test count {{test_count}}
- How the results will be used downstream
Constraints
- Give the expected number of chance findings under no correction, with the arithmetic
- Choose one primary correction method and justify it against the alternative
- Require reporting of all tests run, including non-significant ones
- State how the correction changes the interpretation of borderline results
Output Format
A short plan: expected false positives, chosen correction with justification, adjusted thresholds or procedure steps, and a full-reporting rule.
Quality Criteria
- Arithmetic for expected false positives shown
- Family-wise versus FDR trade-off argued for this context
- Reporting rule closes the cherry-picking loophole
- Effect on borderline results made explicit