Public Benefits Determination Agent Fairness and Accuracy Harness

Role: Principal Public Sector Algorithmic Auditor with fifteen years evaluating automated decision systems in municipal welfare agencies.

Context

• Administering Agency: {{agency_name}}
• Statutory Policy Code: {{benefit_statute_code}}
• Demographic Evaluation Cohort: {{synthetic_applicant_cohort}}
• Disparate Impact Threshold: {{disparate_impact_threshold}}
• Statutory Decision Matrix: {{statutory_rules_manifest}}
• Audit Log Destination: {{logging_endpoint}}

Task

Generate a standalone Python evaluation harness script that executes rigorous synthetic caseload testing against the {{agency_name}} benefit intake agent to detect statutory hallucinations, calculate disparate impact across demographic sub-groups, and verify deterministic policy compliance against {{benefit_statute_code}}.

Method

1. Ingest test applicant profiles from {{synthetic_applicant_cohort}} and map parameters directly to {{statutory_rules_manifest}}.
2. Construct an asynchronous execution pool calling the target intake agent endpoint across baseline and perturbed edge-case applicant records.
3. Implement deterministic scoring logic comparing agent eligibility output against the ground-truth decisions derived from {{benefit_statute_code}}.
4. Calculate demographic parity metrics, four-fifths rule adherence, and false rejection rates across all sub-cohorts relative to {{disparate_impact_threshold}}.
5. Extract reasoning traces from agent responses to check for non-statutory rationales or fabricated policy citations.
6. Generate structured JSON evaluation artifacts containing summary statistics, worst-case counterfactual pairs, and audit traces.
7. Pipe all runtime telemetry and compliance assertions to {{logging_endpoint}} with cryptographically signed run hashes.

Constraints

• The script MUST compile and run as valid Python 3.11 code using standard data science and asynchronous networking libraries.
• MUST NOT suppress or round false-negative eligibility errors in high-vulnerability test cohorts.
• All metric thresholds must be dynamically parameterized rather than hardcoded.
• Code must include robust error handling for API timeouts and malformed JSON payloads from the target agent.

Output format

• Script Environment and Dependency Declarations (standard pip requirements block)
• Complete Executable Python Script (data loaders, async runner, fairness scoring, output exporter)
• Inline Configuration Section mapping to {{disparate_impact_threshold}} and {{statutory_rules_manifest}}
• Execution and CLI Invocation Guide (exact terminal commands with sample flag usage)

Self-review

• Ensure all variables ({{agency_name}}, {{benefit_statute_code}}, {{synthetic_applicant_cohort}}, {{disparate_impact_threshold}}, {{statutory_rules_manifest}}, {{logging_endpoint}}) are explicitly utilized.
• Verify disparate impact computation formulas adhere to standard adverse impact calculation protocols.
• Check that no placeholder pseudo-code is left in critical evaluation and scoring loops.