Multi-Step Symbolic Math Reasoning Harness Generator

Role: Principal Formal Verification and AI Alignment Researcher with 15+ years in computational logic and automated theorem proving.

Context

• Evaluation target dataset: {{agent_reasoning_dataset}}
• Target symbolic computation engine: {{symbolic_engine_target}}
• Numerical and symbolic equivalence tolerance: {{tolerance_threshold}}
• Expected JSON schema for proof step nodes: {{proof_step_schema}}
• Logic failure categorization schema: {{failure_classification_taxonomies}}

Task

Write an production-grade Python evaluation script that parses multi-step reasoning traces from {{agent_reasoning_dataset}}, checks equation balance and deductive step validity against {{symbolic_engine_target}}, and outputs a structured error diagnostic log.

Method

1. Import verification libraries (such as SymPy, Z3, or NumPy) configured for {{symbolic_engine_target}}.
2. Define structured Pydantic models to validate the agent trace structure according to {{proof_step_schema}}.
3. Ingest and parse individual reasoning steps, isolating variable definitions, assumptions, intermediate assertions, and conclusions.
4. Build a stateful symbolic context accumulator that tracks active mathematical constraints across successive derivation steps.
5. Evaluate each derivation step against preceding context using symbolic subtraction and zero-equivalence checks within {{tolerance_threshold}}.
6. Detect and categorize logical fallacies, circular definitions, and unjustified intermediate steps using {{failure_classification_taxonomies}}.
7. Compute composite accuracy metrics: Step Accuracy Rate (SAR), Premise Consistency Score (PCS), and Global Soundness Index (GSI).
8. Output executable CLI evaluation commands and export consolidated verification summaries to a JSON/CSV artifact.

Constraints

• The script MUST be self-contained, fully typed with Python type hints, and ready for immediate headless CI/CD execution.
• The script MUST NOT accept purely stochastic token-matching as proof of step correctness; verification must rely on symbolic execution.
• Handle symbolic timeouts and division-by-zero edge cases with graceful exception isolation.
• Include explicit assertion logging for any deduction step classified as invalid.

Output format

Provide the response organized in four distinct sections:

1. Architecture Overview (100-150 words summarizing the evaluation pipeline and solver integration).
2. Complete Python Evaluation Script (120-250 lines of robust, runnable code including docstrings and error handling).
3. Test Fixture Example (A sample synthetic reasoning trace JSON demonstrating valid and invalid steps).
4. CLI Execution & Analysis Guide (Step-by-step terminal instructions for running the test harness).

Self-review

• Does the script parse and validate trace data strictly against {{proof_step_schema}}?
• Are step evaluations strictly isolated so one broken proof step does not crash the evaluation of subsequent independent test cases?
• Does the code properly handle symbolic edge cases using {{tolerance_threshold}}?