Agent Tool-Use Cost-Utility and Selection Drift Benchmark Spec

Role: Staff Agentic Performance & Unit Economics Analyst.

Context

• Foundation Model Family: {{model_family}}
• Active Tool Catalog: {{registered_tools_catalog}}
• Input/Output Token Rates: {{cost_per_million_tokens}}
• Baseline Evaluation Window: {{drift_baseline_window}}
• Benchmark Dataset Reference: {{benchmark_dataset_id}}
• Task Success Definition: {{success_criteria_rubric}}

Task

Construct an end-to-end Cost-Utility and Tool Selection Drift Benchmark Specification to quantify unit economics, redundant tool-call overhead, and policy selection accuracy for {{model_family}} utilizing {{registered_tools_catalog}}.

Method

1. Establish baseline tool-calling profiles using {{benchmark_dataset_id}} across {{drift_baseline_window}} historic performance data.
2. Define token expenditure equations linking tool description overhead, payload input/output tokens, and {{cost_per_million_tokens}}.
3. Formulate the Cost-Utility Ratio (CUR) measuring financial cost incurred per point on {{success_criteria_rubric}}.
4. Design a Tool Selection Confusion Matrix tracking True Invocations, Redundant Invocations, Hallucinated Tools, and Missed Invocations.
5. Calculate Over-Tooling Penalty (OTP) to penalize agents that call multi-step tools when a single direct generation or simpler tool suffices.
6. Specify statistical drift detection algorithms (e.g., Population Stability Index or Jensen-Shannon Divergence) for tool usage distributions.
7. Detail automated regression testing criteria to gate prompt or model version deployments based on CUR and drift thresholds.

Constraints

• MUST calculate unit costs down to four decimal places per completed workflow.
• MUST NOT rely on subjective evaluations; all scoring must map to {{success_criteria_rubric}} benchmarks.
• MUST isolate system prompt token overhead from dynamic tool payload token cost.
• Drift detection algorithms must specify clear statistical confidence intervals (e.g., 95% CI).

Output format

1. Financial & Token Cost Formulas (CUR, Cost-per-Action, OTP equations)
2. Tool Selection Precision & Recall Metric Definitions
3. Drift Measurement Methodology (statistical test, baseline comparison window, trigger levels)
4. Automated Regression Gating Policy (Pass/Fail criteria for CI/CD)

Self-review

• Are all 6 variables ({{model_family}}, {{registered_tools_catalog}}, {{cost_per_million_tokens}}, {{drift_baseline_window}}, {{benchmark_dataset_id}}, {{success_criteria_rubric}}) integrated accurately?
• Are unit economics and model drift calculations mathematically coherent?
• Does the gating policy establish clear binary thresholds for agent deployment?