Copywriting AB Test Variant Performance Report

Role: Staff Growth Data Analyst & Experimentation Strategist

Context

• Experiment Data Table: {{experiment_table_schema}}
• Copy Variant Descriptions: {{copy_variant_definitions}}
• Primary Conversion Metric: {{primary_conversion_metric}}
• Target Significance Level: {{statistical_significance_threshold}}
• Monitored Guardrail Metrics: {{guardrail_metrics}}
• Demographic & Behavioral Segments: {{segmentation_dimensions}}

Task

Author an end-to-end copywriting experimentation report containing parameterized SQL queries for sample size, conversion lift, and p-value validation, alongside qualitative copy teardowns and rollout decisions.

Method

1. Query the {{experiment_table_schema}} to compute exposure counts, primary conversion events for {{primary_conversion_metric}}, and sample distribution across all variants in {{copy_variant_definitions}}.
2. Write SQL logic to test for Sample Ratio Mismatch (SRM) using a Chi-Square goodness-of-fit calculation.
3. Compute baseline conversion rates, absolute lift, relative lift, and standard errors per variant using SQL analytical aggregation.
4. Calculate Z-scores and approximate two-tailed p-values directly within the database query to evaluate against {{statistical_significance_threshold}}.
5. Segment conversion performance across {{segmentation_dimensions}} to uncover localized message resonance or copy fatigue.
6. Evaluate negative movement against {{guardrail_metrics}} to ensure conversion gains did not cause downstream churn or qualified lead degradation.
7. Extract thematic linguistic patterns from winning copy variants to inform overarching content strategy.
8. Determine concrete ship, iterate, or kill recommendations for each copy test branch.

Constraints

• The SQL script MUST calculate SRM checks before calculating conversion lift to ensure sample validity.
• You MUST NOT recommend rolling out a copy variant that breaches {{guardrail_metrics}}, regardless of lift on {{primary_conversion_metric}}.
• All statistical calculations in SQL must handle zero-division edge cases gracefully with NULLIF or COALESCE.
• Every copy variant recommendation must reference specific messaging elements (e.g., value proposition, tone, CTA phrasing).

Output format

Provide the report structured under the following mandatory sections:

1. Experiment Metadata & SRM Validation (summary table and test status)
2. Experimentation SQL Engine (complete, copy-paste executable SQL script including statistical calculations)
3. Variant Lift & Statistical Significance Summary (markdown table detailing exposures, conversions, lift, and p-values)
4. Guardrail & Sub-Segment Analysis (evaluation of {{guardrail_metrics}} across {{segmentation_dimensions}})
5. Copywriting Analysis & Strategic Deployment Directive (detailed recommendations per variant)

Self-review

• Ensure the SQL script computes both standard error and confidence bounds accurately.
• Check that the SRM check is clearly explained with pass/fail criteria.
• Confirm that copy variant critiques tie back directly to {{copy_variant_definitions}}.