Statutory Public Records Redaction Agent Audit and Benchmark Pipeline

Role: Senior Government Information Governance & Compliance Technologist specializing in automated FOIA exemption verification and auditing.

Context

• Government Agency: {{government_entity}}
• Exemption Taxonomy: {{exemption_category_matrix}}
• Ground Truth Test Corpus: {{ground_truth_corpus_path}}
• Permissible Over-Redaction Margin: {{acceptable_over_redaction_rate}}
• Critical PII Leakage Threshold: {{pii_leakage_tolerance}}
• Output Report Bucket: {{report_output_bucket}}

Task

Author a Python-based evaluation pipeline script that bench-tests {{government_entity}} automated redaction agents against {{ground_truth_corpus_path}}, calculating precision-recall metrics for every category in {{exemption_category_matrix}} and flagging statutory non-compliance.

Method

1. Load labeled ground-truth government documents and corresponding redaction masks from {{ground_truth_corpus_path}}.
2. Submit unredacted source documents concurrently to the redaction agent via secure client calls.
3. Perform character-level and token-level intersection-over-union (IoU) comparisons between agent redaction bounding boxes and ground truth.
4. Categorize errors into PII leakage (false negatives) versus excessive information suppression (false positives).
5. Validate that every applied blackline redaction is paired with an exact statutory citation from {{exemption_category_matrix}}.
6. Compute precision, recall, F1-score, and over-redaction index, comparing against {{acceptable_over_redaction_rate}} and {{pii_leakage_tolerance}}.
7. Generate an executive compliance matrix and export structured findings to {{report_output_bucket}}.

Constraints

• MUST trigger an immediate non-zero exit code if PII leakage exceeds {{pii_leakage_tolerance}}.
• MUST NOT store unredacted PII in plaintext logs, debug strings, or standard output.
• Bounding box and text span matching must account for OCR alignment variances within a 3% character drift tolerance.
• All exemption classifications must be validated strictly against keys defined in {{exemption_category_matrix}}.

Output format

• Script Dependencies and Virtual Environment Setup Instructions
• Complete Python Evaluation Script (span parser, agent client, evaluation metrics engine, auditor reporter)
• Inline Configuration Object parsing {{acceptable_over_redaction_rate}} and {{pii_leakage_tolerance}}
• Automated Verification Runner Script (Bash wrapper for CI/CD integration)

Self-review

• Check that all 6 context variables ({{government_entity}}, {{exemption_category_matrix}}, {{ground_truth_corpus_path}}, {{acceptable_over_redaction_rate}}, {{pii_leakage_tolerance}}, {{report_output_bucket}}) appear and function in the script.
• Verify token-level and character-level IoU mathematical logic is robustly implemented.
• Ensure strict exit status codes are enforced for critical privacy regressions.