Autonomous Tool-Calling Evaluation Harness and Assertion Script

Role: AI Systems Quality & Governance Documentation Architect specializing in autonomous agent testing harnesses.

Context

• Target Model Provider: {{agent_model_target}}
• Tool Registry Specifications: {{tool_registry_definition}}
• Adversarial Scenario Suite: {{adversarial_test_cases}}
• Assertion & Evaluation Framework: {{assertion_engine}}
• Concurrency & Rate Limit Ceiling: {{rate_limit_concurrency}}

Task

Develop an automated tool-calling benchmark script and behavioral documentation suite that evaluates LLM agent tool selection accuracy, detects hallucinated parameters under adversarial pressure, and logs deterministic audit reports.

Method

1. Parse the available function schemas and guardrail limits defined in {{tool_registry_definition}}.
2. Synthesize test matrices combining nominal user prompts with boundary conditions from {{adversarial_test_cases}}.
3. Configure parallel request batching with strict concurrency controls matching {{rate_limit_concurrency}}.
4. Bind agent invocation clients configured for {{agent_model_target}} with structured response interceptors.
5. Construct automated evaluators using {{assertion_engine}} to score parameter extraction accuracy, hallucination rates, and schema adherence.
6. Generate automated markdown governance documentation summarizing model reliability, failure modes, and safety violations.
7. Produce an exportable telemetry log with timestamped traces, input tokens, and validation scores for CI/CD integration.

Constraints

• The benchmark script MUST run idempotently with zero side effects on external production tools.
• Output MUST include non-zero assertion tests for parameter hallucination and schema over-generation.
• API keys and environment secrets MUST NOT be hardcoded in script variables.
• Concurrency limit MUST strictly cap active requests at {{rate_limit_concurrency}}.

Output format

Deliver an executable script structured in the following order:

1. Environment setup and harness configuration
2. Tool Schema Registry and Mock Execution Environment
3. Adversarial Test Runner & Concurrency Worker Pool
4. Assertion Suite & Statistical Evaluation Matrix
5. Markdown Audit Report Auto-Exporter

Self-review

• Verify that all test cases in {{adversarial_test_cases}} are represented in the evaluation suite.
• Check that assertion logic correctly identifies invalid tool names or missing required arguments.
• Confirm that rate-limiting throttling matches {{rate_limit_concurrency}} to prevent API rejections.