E-Commerce Order Resolution Agent Benchmark Generator

Role: Principal AI Evaluation Engineer specializing in retail customer care automation and LLM-as-a-judge benchmarking systems.

Context

• Target Retailer: {{brand_name}}
• Live Agent Endpoint: {{agent_endpoint}}
• Authoritative Policy Source: {{return_policy_rules}}
• Test Corpus Location: {{edge_case_dataset}}
• Maximum Concession Margin: {{tolerance_threshold}}
• Primary Evaluation Metrics: {{eval_metric_suite}}

Task

Generate a production-grade Python evaluation script that sends synthetic customer scenarios to {{agent_endpoint}}, evaluates the conversational agent responses against {{return_policy_rules}} using LLM-as-a-judge techniques, and asserts that financial concessions do not exceed {{tolerance_threshold}}.

Method

1. Parse {{edge_case_dataset}} into structured synthetic customer sessions containing order histories, return reasons, and emotional escalation markers.
2. Construct an asynchronous execution loop to replay sessions against {{agent_endpoint}} while recording request-response pairs and token latency.
3. Build an evaluation engine incorporating {{eval_metric_suite}} to score policy adherence, factual grounding, and tone consistency.
4. Embed deterministic boundary assertions that detect illegal refund approvals exceeding {{tolerance_threshold}} without human-in-the-loop triggers.
5. Implement an LLM judge prompt that scores the agent transcripts against exact clauses inside {{return_policy_rules}}.
6. Generate summary aggregate statistics including pass rates, false refund rates, hallucinated policy rates, and 95th percentile latency.
7. Format execution logs into structured JSON artifacts and terminal console outputs with actionable failure traces.

Constraints

• The generated script MUST be self-contained, executable Python (>=3.10) using standard async libraries (aiohttp, pydantic, pytest-compatible).
• The output MUST NOT hardcode mock secrets or omit error handling for network timeouts.
• Evaluation logic MUST isolate financial over-concession as a blocking catastrophic failure.
• Include type hints and inline docstrings explaining scoring thresholds across {{eval_metric_suite}}.

Output format

• Section 1: Environment prerequisites and installation instructions (Bash).
• Section 2: Complete executable Python evaluation script containing test data parsers, async runner, judge logic, and assertion suite.
• Section 3: Sample JSON report schema outputted by the runner (20-30 lines of JSON).
• Total script length should not exceed 250 lines of code.

Self-review

1. Verify all {{return_policy_rules}} and {{tolerance_threshold}} constraints are verified in assertions.
2. Ensure async execution handles rate limits gracefully when querying {{agent_endpoint}}.
3. Check that evaluation metric definitions align directly with {{eval_metric_suite}}.