Causal Inference and Logic Trace Benchmark Scorer

Role: Staff Cognitive Systems Architect and Autonomous Agent Evaluator with deep expertise in Pearlian causal inference and formal logic.

Context

• Causal benchmark graph definitions: {{causal_graph_benchmark_set}}
• Agent execution trajectory logs: {{agent_execution_logs}}
• Counterfactual search depth parameter: {{counterfactual_depth_level}}
• Penalty coefficient for redundant reasoning loops: {{backtracking_penalty_factor}}
• Diagnostic export destination: {{scorer_output_directory}}

Task

Write a Python scoring script that analyzes agent execution logs from {{agent_execution_logs}}, evaluates the agent's causal graph traversal against {{causal_graph_benchmark_set}}, and calculates causal discovery and counterfactual reasoning efficiency scores.

Method

1. Ingest causal directed acyclic graph (DAG) ground truth definitions from {{causal_graph_benchmark_set}} using NetworkX.
2. Ingest agent action traces, tool calls, and deduction steps from {{agent_execution_logs}}.
3. Map the agent's inferred causal links against true graph edges to identify true positives, false discovery, and inverted causal arrows.
4. Trace counterfactual reasoning paths up to {{counterfactual_depth_level}} steps to evaluate intervention calculations (do-calculus consistency).
5. Identify redundant reasoning loops, circular causal attributions, and unnecessary backtracking, applying {{backtracking_penalty_factor}}.
6. Compute benchmark metrics: Structural Hamming Distance (SHD), Interventional Accuracy (IA), and Reasoning Efficiency Ratio (RER).
7. Write consolidated benchmark reports and node-level error attribution data into {{scorer_output_directory}}.

Constraints

• The script MUST utilize deterministic graph matching algorithms rather than heuristic LLM-as-a-judge approximations.
• The script MUST NOT fail or crash when an agent log contains malformed JSON or interrupted execution traces.
• Output pure, executable Python script with no pseudo-code or missing method bodies.
• Include unit test fixtures directly within the script using the unittest or pytest framework.

Output format

Provide the solution organized into four strict sections:

1. Mathematical Formulation (Equations and criteria for SHD, Interventional Accuracy, and Efficiency scoring).
2. Complete Python Scorer Script (Full script including graph parsers, log analyzers, and file writers).
3. Sample DAG & Trace Fixture (A minimal working DAG JSON and sample agent reasoning log for testing).
4. Verification & Validation Instructions (Commands to execute tests and interpret output files in {{scorer_output_directory}}).

Self-review

• Does the scoring script properly handle interventions at depth {{counterfactual_depth_level}}?
• Are backtracking and loop penalties scaled correctly using {{backtracking_penalty_factor}}?
• Are all outputs and logs properly directed to {{scorer_output_directory}}?