Philanthropic Grant Evaluation Agent Hallucination and Scrutiny Test Script

Role: Lead Nonprofit Systems Architect specializing in automated grant screening and transparent funding governance.

Context

• Foundation: {{foundation_name}}
• Governing Guidelines: {{funding_guidelines_doc}}
• Dataset Sample Size: {{grant_application_sample_size}}
• Rubric Schema: {{scoring_rubric_schema}}
• Hallucination Threshold: {{hallucination_tolerance_rate}}
• Telemetry Target: {{telemetry_export_format}}

Task

Create an automated regression and hallucination evaluation script that validates LLM-driven grant application pre-screening agents against {{funding_guidelines_doc}} for {{foundation_name}}, generating deterministic evaluation telemetry.

Method

1. Translate {{scoring_rubric_schema}} into programmatic validation functions and scoring constraints.
2. Implement a synthetic grant proposal loader supporting a batch size of {{grant_application_sample_size}}.
3. Build citation-grounding verifiers that map agent evaluations back to verbatim clauses in {{funding_guidelines_doc}}.
4. Develop an automated hallucination detection module that flags ungrounded scoring penalties or fabricated claims.
5. Calculate intra-rater scoring variance across identical repeated inputs to measure output determinism.
6. Evaluate score distributions against {{hallucination_tolerance_rate}} to generate automated pass/fail flags.
7. Implement serialization logic exporting results directly into {{telemetry_export_format}}.

Constraints

• MUST implement deterministic assertion checks that fail if hallucination rate exceeds {{hallucination_tolerance_rate}}.
• MUST NOT introduce non-deterministic external network calls inside the scoring assertion loop.
• All code must include modular functions with explicit type hinting.
• Output telemetry must strictly conform to the specifications of {{telemetry_export_format}}.

Output format

Deliver an end-to-end evaluation script containing:

1. Environment Setup & Dependency Imports.
2. Schema Definition & Rubric Validator Module (matching {{scoring_rubric_schema}}).
3. Grounding & Hallucination Assessment Engine.
4. Benchmark Execution Pipeline with {{telemetry_export_format}} export handler.

Self-review

1. Does the script verify strict alignment with {{funding_guidelines_doc}}?
2. Are variance and hallucination thresholds mathematically explicit?
3. Is the telemetry export schema valid and free of undefined keys?