Multi-Turn Autonomous Tool-Calling Evaluation Scenario Script

Role: Senior Agent Safety and Evaluation Engineer specializing in autonomous reasoning benchmarks and tool-calling validation suites.

Context

• Operational Domain: {{agent_domain}}
• Target Tool Catalog: {{allowed_toolset}}
• Ambiguous Intent Scenarios: {{ambiguous_intents}}
• Adversarial Test Vectors: {{adversarial_edge_cases}}
• Benchmark Strictness Criteria: {{strictness_threshold}}
• Execution Harness: {{test_runner_framework}}

Task

Generate an automated test suite script using {{test_runner_framework}} that evaluates an autonomous agent's tool-calling accuracy, hallucination resistance, and recovery behavior across {{agent_domain}} scenarios.

Method

1. Define mock implementations of all tools in {{allowed_toolset}} that simulate successes, rate limits, and malformed responses.
2. Construct synthetic multi-turn conversation fixtures incorporating {{ambiguous_intents}} to assess clarification triggers.
3. Formulate adversarial injection test cases from {{adversarial_edge_cases}} to evaluate agent tool authorization gates.
4. Implement deterministic assertion functions that evaluate tool choice correctness, argument validity, and invocation order.
5. Write negative assertion tests ensuring zero invocations of non-existent or hallucinated functions.
6. Compute aggregate reliability scores based on {{strictness_threshold}} across parameter precision and sequence efficiency.
7. Structure parameterized test fixtures to enable parallel execution against varied model backend endpoints.

Constraints

• Test script MUST be written as a valid, executable {{test_runner_framework}} test file with zero placeholder mock data.
• Tests MUST assert that the agent asks clarifying questions when faced with {{ambiguous_intents}} instead of guessing tool parameters.
• Assertions must evaluate both raw JSON schema compliance and semantic validity of tool payloads.
• Hallucinated function calls MUST result in an immediate hard failure of the test scenario.

Output format

Provide the response structured as follows:

1. Test Suite Configuration & Fixtures (Mock tools and environment setups)
2. Parameterized Test Cases Script (Full test methods covering nominal, ambiguous, and adversarial runs)
3. Benchmark Evaluation Metric Hook (Custom metric calculators outputting summary pass/fail metrics)

Self-review

• Check that every edge case in {{adversarial_edge_cases}} has a corresponding explicit test method.
• Confirm that mock tool fixtures accurately simulate boundary conditions without external network dependencies.
• Ensure assertions strictly enforce the criteria stated in {{strictness_threshold}}.