Autonomous Literature Synthesis and Citation Audit Pipeline

Role: Lead Quantitative Research Auditor and AI Benchmarking Engineer specializing in scientific factuality and natural language inference.

Context

• Synthesis corpus input path: {{synthesis_corpus_path}}
• Ground truth citation database: {{citation_ground_truth_db}}
• Natural Language Inference endpoint: {{nli_model_endpoint}}
• Minimum claim extraction granularity: {{claim_extraction_granularity}}
• Hallucination penalty weight: {{hallucination_penalty_weight}}
• Evaluation artifact output format: {{report_export_format}}

Task

Construct an end-to-end Python evaluation script that parses research synthesis outputs from {{synthesis_corpus_path}}, matches extracted atomic claims against {{citation_ground_truth_db}} via {{nli_model_endpoint}}, and generates an empirical citation fidelity audit report.

Method

1. Load and parse synthesized research documents into segmented structural units (abstract, thematic sections, claim blocks).
2. Deconstruct complex paragraphs into atomic verifiable propositions based on {{claim_extraction_granularity}}.
3. Extract inline citation anchors and link each atomic claim to its corresponding bibliography entry in {{citation_ground_truth_db}}.
4. Execute cross-encoder entailment checks via {{nli_model_endpoint}} to classify claim-citation pairs into Entailment, Neutral, or Contradiction.
5. Detect phantom citations, bibliographic mismatches, and over-extrapolated claims unsupported by the cited source text.
6. Compute aggregate metrics: Citation Recall, Entailment Precision, and Faithfulness Score weighted by {{hallucination_penalty_weight}}.
7. Format and export the diagnostic output into {{report_export_format}} containing granular claim-level audit traces.

Constraints

• The script MUST implement deterministic batch processing with exponential backoff for {{nli_model_endpoint}} requests.
• The script MUST NOT mark a claim as verified if the cited source is missing from {{citation_ground_truth_db}}.
• Maintain an execution speed benchmark that processes at least 20 claims per second under local caching.
• Include structured logging with explicit severity levels (DEBUG, INFO, WARNING, CRITICAL).

Output format

Provide the evaluation suite organized into four structured components:

1. Pipeline Design & Metrics Formulation (A concise overview of the NLI evaluation logic and scoring math).
2. Standalone Python Verification Script (Full runnable code containing async batching, data loading, and metric calculations).
3. Sample Evaluation Config File (YAML configuration defining model endpoints, weights, and thresholds).
4. Metric Interpretation Guide (A concise table mapping score distributions to release recommendations).

Self-review

• Does the script accurately calculate penalties using {{hallucination_penalty_weight}}?
• Are atomic claims cleanly mapped to primary sources in {{citation_ground_truth_db}} without index mismatches?
• Is the final report formatted according to the exact schema specified in {{report_export_format}}?