Multimodal Diffusion Prompt Experiment Statistical Evaluation Email

Role: Principal Multimodal Experimentation Statistician with 12 years of experience in generative evaluation and Bayesian hypothesis testing.

Context

• Experiment identifier and scope: {{experiment_name}}
• Baseline control prompt formulation: {{baseline_prompt_strategy}}
• Treatment prompt formulation: {{candidate_prompt_strategy}}
• Total generated sample volume per arm: {{sample_size_per_variant}}
• CLIP score distribution metrics and variance: {{clip_score_data}}
• Human rater pairwise preference win rates: {{human_eval_win_rate}}

Task

Draft a rigorous executive email to engineering leadership evaluating whether {{candidate_prompt_strategy}} demonstrates statistically significant superiority over {{baseline_prompt_strategy}} without introducing distribution skew or variance degradation.

Method

1. Formulate the primary null ($H_0$) and alternative ($H_a$) hypotheses for both CLIP cosine similarity distributions and human rater preference scores.
2. Evaluate parametric assumptions across {{clip_score_data}} to determine whether Student's t-test or non-parametric Mann-Whitney U test applies.
3. Compute effect size (Cohen's d or Cliff's delta) alongside 95% confidence intervals across {{sample_size_per_variant}} generations per arm.
4. Analyze binomial distribution metrics and confidence bounds for {{human_eval_win_rate}} to check for inter-annotator agreement reliability.
5. Conduct a false discovery rate (FDR) correction (Benjamini-Hochberg) across secondary automated visual metrics.
6. Synthesize type I and type II error trade-offs and evaluate statistical power achieved in {{experiment_name}}.
7. Provide an unambiguous rollout verdict categorized as Deploy, Iterate, or Halt based strictly on statistical thresholds.

Constraints

• MUST report exact p-values, statistical power, and effect sizes rather than subjective impressions.
• MUST NOT recommend deployment if human win-rate lower confidence bound is below 50%.
• Tone must remain objective, mathematically precise, and executive-ready.
• Total word count must not exceed 450 words.

Output format

Email format with the following structure:

• Subject Line: [Stat Evaluation] {{experiment_name}} Rollout Recommendation
• Section 1: Executive Verdict & Core Metrics (3 bullets maximum)
• Section 2: Inferential Statistical Analysis (hypothesis results, effect size, confidence intervals)
• Section 3: Distribution & Power Diagnostics (normality, sample power, risk bounds)
• Section 4: Recommended Action Plan

Self-review

• Did I include explicit confidence intervals for both automated scores and human ratings?
• Are all 6 contextual variables explicitly referenced and contextualized?
• Is the mathematical reasoning sound without any generic placeholder language?