Agentic Benchmark and Guardrail Specification Quality Checklist

Role: Principal AI Safety & Evaluation Documentation Strategist specializing in autonomous agent policy standards and deterministic validation matrices.

Context

• Benchmark Dataset Scope: {{eval_dataset_type}}
• Autonomy Level Classification: {{agent_autonomy_level}}
• Safety & Governance Policy: {{safety_taxonomy}}
• Tool Invocation Boundaries: {{tool_invocation_boundaries}}
• Scoring Rubric & Metrics: {{metric_scoring_rubric}}

Task

Author a comprehensive evaluation checklist to review long-form benchmark specifications, safety boundary definitions, and testing rubrics designed to measure autonomous tool-calling agents against hallucination, prompt injection, and unauthorized side-effects.

Method

1. Examine {{eval_dataset_type}} to ensure the benchmark documentation covers adversarial edge cases, out-of-distribution inputs, and multi-turn scenarios.
2. Cross-reference the documented capabilities with {{agent_autonomy_level}} to ensure test rigor matches autonomous operational risk.
3. Audit safety policy definitions against {{safety_taxonomy}}, verifying explicit boundary tests for harmful actions, privilege escalation, and data exfiltration.
4. Assess evaluation scenarios governing {{tool_invocation_boundaries}}, checking for parameter validation and unauthorized resource access.
5. Review scoring rubrics in {{metric_scoring_rubric}} for objective calculation of precision, recall, safety breach frequency, and tool call accuracy.
6. Formulate checks for test environment isolation, ensuring zero side effects occur on production services during test harness execution.
7. Organize findings into structured evaluation tiers with explicit remediation instructions for ambiguous benchmark criteria.

Constraints

• Every checklist item MUST include a quantitative evaluation metric or verifiable documentation standard.
• The checklist MUST NOT allow self-reported or ungrounded accuracy metrics.
• Must explicitly validate compliance against {{safety_taxonomy}} and {{tool_invocation_boundaries}}.
• Must require reproducible test harness specifications in {{metric_scoring_rubric}}.
• Maintain formal, safety-critical language throughout all checklist criteria.

Output format

Structure the checklist into four distinct verification categories:

1. Dataset Representation & Adversarial Coverage (5 items)
2. Tool Invocation & Boundary Enforcement (5 items)
3. Safety Taxonomy & Vulnerability Mitigation (4 items)
4. Metric Repeatability & Statistical Rigor (4 items) Format each line as: [ ] **[Check Identifier]**: [Validation Requirement] | [Required Evidence / Metric Threshold].

Self-review

• Verify all 5 context variables are explicitly referenced within the checklist logic.
• Ensure exactly 18 items are generated across the four specified categories.
• Confirm that boundary enforcement and adversarial safety mechanisms are rigorously tested.