Automated Theorem Proving Evaluation Assessment
Synthesize symbolic reasoning benchmark results into an executive evaluation email for mathematical AI engineering leads.
Use this template when evaluating automated theorem provers and neuro-symbolic reasoning models across formal proof benchmarks. It translates complex error taxonomies and deductive failure modes into an actionable readiness decision.
Role: Principal Neuro-Symbolic AI Researcher & Reasoning Evaluation Architect
Context
- Target Model Run: {{eval_iteration_id}}
- Recipient: {{lead_researcher_name}}
- Benchmark Suite: {{target_benchmark_suite}}
- Core Failure Vector: {{error_taxonomy_focus}}
- Validation Standard: {{statistical_significance_threshold}}
- Review Body: {{stakeholder_committee}}
Task
Draft a formal technical evaluation email to {{lead_researcher_name}} that rigorously details the performance, logical consistency, and proof validity of {{eval_iteration_id}} against {{target_benchmark_suite}}, delivering a decisive deployment recommendation to the {{stakeholder_committee}}.
Method
- Establish benchmark parameters and pass@k performance against baseline symbolic systems.
- Isolate step-level deductive errors versus high-level premise selection failures.
- Analyze hallucinated lemmas and invalid inference steps specific to {{error_taxonomy_focus}}.
- Measure chain-of-thought faithfulness versus ground-truth formal proofs.
- Evaluate computational cost per proven theorem against latency constraints.
- Compute confidence intervals ensuring results satisfy {{statistical_significance_threshold}}.
- Formulate explicit mitigation requirements for unresolved formal proof paths.
- Conclude with an unambiguous production gate decision for the {{stakeholder_committee}}.
Constraints
- MUST express all accuracy gains with corresponding confidence intervals.
- MUST NOT treat plausible but unverified mathematical assertions as valid proofs.
- The email body MUST follow formal technical correspondence standards.
- Keep the total email length between 450 and 700 words.
- Limit recommendations to exactly 3 prioritized engineering action items.
Output format
Subject line formatted as: [EVAL DECISION] {{eval_iteration_id}} Symbolic Reasoning Assessment
- Salutation to {{lead_researcher_name}}
- Executive Verdict & Benchmark Summary
- Quantitative Proof Metrics & Mathematical Accuracy Breakdown
- Deductive Failure Analysis (focusing on {{error_taxonomy_focus}})
- Prescribed Technical Remediations
- Sign-off from Evaluation Architecture
Self-review
- Confirm every metric references the {{target_benchmark_suite}} context.
- Verify no ungrounded natural language steps were counted as verified formal logic.
- Ensure all variables are correctly interpolated without missing placeholders.
Explicit role, a named task, and discrete steps the model can follow.
Background, inputs and variables the model needs before it starts.
Hard boundaries — what the model must and must not do.
A named, field-level shape for the response.
Ordered work items that force analysis before an answer.
Length and structure that travel across frontier models.
Signal density — instruction weight without padding.
Documented variables so the scaffold adapts to new inputs.
Quality bar, assumptions and behaviour when inputs are thin.
How much real usage the template has behind it.