Agent Tool Telemetry and Failure Cascade Evaluation Spec

Role: Principal Agent Analytics Engineer specializing in autonomous execution observability.

Context

• Target Agent System: {{agent_workflow_name}}
• Integrated Tool Registry: {{target_tool_registry}}
• Observability Platform: {{telemetry_sink}}
• End-to-End Latency Target: {{latency_sla_ms}}
• Allowable Error Threshold: {{error_budget_percentage}}
• Analytics Reporting Interval: {{evaluation_cadence}}

Task

Author a comprehensive Tool-Calling Telemetry and Evaluation Metrics Specification to monitor schema conformity, failure cascades, and invocation latencies across {{agent_workflow_name}} running against {{target_tool_registry}}.

Method

1. Map every tool invocation event to standard OpenTelemetry semantic conventions for tool-calling agents.
2. Define span attributes for payload serialization, argument hallucination rates, and parameter schema validation passes.
3. Establish latency decomposition metrics isolating model generation time, network serialization, and external API execution in {{telemetry_sink}}.
4. Formalize cascade error tracking rules to pinpoint when a single malformed tool output causes multi-step recovery loops.
5. Specify retry and backoff analytical counters to detect infinite retry traps under {{error_budget_percentage}} limits.
6. Formulate aggregate quality KPIs including Tool Selection Precision (TSP), Tool Argument Accuracy (TAA), and Execution Success Ratio (ESR).
7. Define alerting thresholds pegged to {{latency_sla_ms}} breaches and anomalous schema mismatch spikes for {{evaluation_cadence}} audits.

Constraints

• MUST define exact JSON-schema schemas for log payload events.
• MUST NOT use opaque black-box scoring; every metric must map to verifiable runtime telemetry attributes.
• MUST include explicit fallback handling telemetry states (e.g., recovered, fatal, unhandled).
• Metric calculations must maintain computational feasibility across high-throughput agent runs.

Output format

1. Architectural Telemetry Schema (JSON-Schema format, max 40 lines)
2. Core Metric Registry Table (Metric Name, Definition, Formula, Granularity)
3. Failure Mode Diagnostic Matrix (5 specific failure classes with diagnostic signals)
4. Alerting & Governance Rules (exact trigger conditions and thresholds)

Self-review

• Are all variables ({{agent_workflow_name}}, {{target_tool_registry}}, {{telemetry_sink}}, {{latency_sla_ms}}, {{error_budget_percentage}}, {{evaluation_cadence}}) referenced appropriately?
• Does the telemetry specification support real-time parsing without ambiguous fields?
• Are failure cascade metrics cleanly separated from pure network failure metrics?