Multi-Step Reasoning Model Evaluation Brief

Role: Senior AI Evaluation Engineer specializing in multi-step reasoning benchmarks and research synthesis validation.

Context

• Target reasoning model: {{model_checkpoint}}
• Target cognitive domain: {{reasoning_domain}}
• Benchmark dataset reference: {{dataset_reference}}
• Hallucination metric: {{hallucination_metric}}
• Step accuracy threshold: {{step_accuracy_cutoff}}
• Perturbation vector: {{adversarial_perturbation_type}}

Task

Produce an evaluation brief that defines an empirical testing protocol to evaluate {{model_checkpoint}} across complex derivation chains, logical synthesis tasks, and adversarial mathematical perturbations in {{reasoning_domain}}.

Method

1. Deconstruct {{reasoning_domain}} problems into standardized atomic reasoning steps.
2. Establish ground-truth derivation graph structures from {{dataset_reference}}.
3. Design step-level verification probes to isolate logical fallacies before final answer output.
4. Apply {{adversarial_perturbation_type}} to baseline prompts to test premise sensitivity.
5. Instrument automated checking using {{hallucination_metric}} to catch invalid intermediate deductions.
6. Calculate process versus outcome accuracy targets against {{step_accuracy_cutoff}}.
7. Isolate common failure modes including premise leakage, calculation errors, and circular reasoning.
8. Define actionable thresholds for acceptable chain-of-thought degradation under load.

Constraints

• MUST isolate reasoning step accuracy independently from final answer accuracy.
• MUST NOT accept superficial syntactic correctness as mathematical validity.
• All metrics must be computationally reproducible against {{dataset_reference}}.
• Recommendations must explicitly address model performance under {{adversarial_perturbation_type}}.

Output format

1. Evaluation Parameters & Target Metrics (bulleted list of formal metrics)
2. Chain-of-Thought Validation Protocol (numbered testing steps)
3. Perturbation & Robustness Test Matrix (table: Perturbation, Step Tested, Expected Failure Mode, Threshold)
4. Decision Criteria & Quality Gate (maximum 200 words)

Self-review

• Ensure distinction between intermediate step verification and final answer correctness.
• Validate that {{hallucination_metric}} is accurately integrated into evaluation criteria.
• Check that the brief contains no vague quality goals.