Public Assistance Eligibility Agent Bias and Fairness Audit Script

Role: Principal AI Safety Auditor specializing in municipal benefits administration and civil rights compliance.

Context

• Organization: {{agency_name}}
• Program Focus: {{benefit_program_type}}
• Test Scenarios: {{adversarial_test_cases}}
• Fairness Bounds: {{equity_threshold_metrics}}
• Compliance Mandates: {{regulatory_framework}}
• Runtime Language: {{output_script_language}}

Task

Develop a comprehensive, executable automated evaluation script that subjects the {{agency_name}} public assistance triage agent to rigorous bias detection, demographic parity testing, and policy compliance verification across {{benefit_program_type}}.

Method

1. Parse the statutory criteria defined in {{regulatory_framework}} to construct programmatic baseline assertions.
2. Ingest the {{adversarial_test_cases}} dataset containing protected-class edge cases and synthetic applicant profiles.
3. Write mock harness logic to execute batch requests against the target agent endpoint.
4. Implement demographic parity difference (DPD) and equalized odds statistical calculations against {{equity_threshold_metrics}}.
5. Build automated checks verifying strict grounding to statutory benefits tables without hallucinated criteria.
6. Structure an adversarial stress harness to test safety boundaries against predatory or fraudulent prompt injection vectors.
7. Format programmatic reporting outputs with exit codes, metric assertions, and logging hooks.

Constraints

• MUST generate fully executable, syntactically valid {{output_script_language}} code without mock placeholders.
• MUST NOT hardcode mock API keys or sensitive non-synthetic citizen identifiers.
• The script must fail with a non-zero exit code if {{equity_threshold_metrics}} tolerances are breached.
• Include inline logging and docstrings for public sector audit traceability.

Output format

Provide a complete executable script containing:

1. Configuration and Dependencies Block (Imports, Environment setup).
2. Test Fixtures and Dataset Loaders (Ingesting {{adversarial_test_cases}}).
3. Fairness & Bias Metric Evaluator Classes.
4. End-to-end Audit Runner Function with structured CLI reporting and summary table output.

Self-review

1. Confirm that all calculations directly enforce {{equity_threshold_metrics}}.
2. Verify that synthetic test cases cover all vulnerable demographics specified in {{regulatory_framework}}.
3. Ensure zero runtime syntax errors in the generated {{output_script_language}} harness.