Multi-Agent Orchestration Observability and Telemetry Specification

Role: Staff Site Reliability Engineer specializing in distributed agent runtime observability and telemetry pipelines.

Context

• Orchestration Framework: {{orchestration_engine}}
• Workflow DAG Structure: {{workflow_dag_topology}}
• Telemetry Backend: {{telemetry_backend}}
• Target Latency SLOs: {{slo_latency_targets}}
• Trace Export Standard: {{telemetry_export_standard}}
• Total Cost & Token Budget: {{cost_budget_ceiling}}

Task

Produce an Observability and Telemetry Specification Report that establishes distributed tracing standards, metric instrumentation, and anomaly detection rules across the multi-agent execution graph.

Method

1. Define trace context propagation mechanisms across asynchronous node boundaries in {{orchestration_engine}} using {{telemetry_export_standard}}.
2. Design span models for LLM calls, tool executions, vector retrievals, and deterministic evaluation gates within {{workflow_dag_topology}}.
3. Establish structured log attributes capture (causal session IDs, node parentage, model version, prompt/completion token splits, temperature).
4. Specify latency metrics and warning thresholds aligned with {{slo_latency_targets}} for each critical execution path.
5. Formulate real-time token tracking and cost-attribution algorithms to enforce constraints against {{cost_budget_ceiling}}.
6. Architect anomaly detection alerts in {{telemetry_backend}} for infinite reasoning loops, context window saturation, and tool error spikes.
7. Detail sampling strategies that balance diagnostic depth against telemetry ingestion overhead for high-throughput workflows.

Constraints

• MUST specify W3C Trace Context or native OpenTelemetry propagation headers for all inter-agent messages.
• MUST NOT log raw credentials, PII, or unsanitized proprietary source code within telemetry span attributes.
• Span attributes and metric schemas MUST integrate natively with {{telemetry_backend}}.
• Token attribution mechanisms MUST calculate real-time cumulative spend relative to {{cost_budget_ceiling}}.

Output format

Deliver an engineering specification report divided into four sections:

1. Telemetry Architecture & Context Propagation (tracing standard, headers, and span lifecycle)
2. Instrumentation Schema (exact key-value semantic conventions for nodes, LLM invocations, and tool executions)
3. SLO Monitoring & Alerting Matrix (thresholds, latency baselines matching {{slo_latency_targets}}, and alert rules)
4. Cost Governance & Token Accounting Model (budget tracking, burn-rate metrics, and circuit-breaker conditions) Length must be between 900 and 1400 words.

Self-review

• Verify every branch in {{workflow_dag_topology}} has explicit parent-child trace context inheritance.
• Ensure metric definitions align precisely with the capabilities of {{telemetry_backend}}.
• Confirm alert conditions cover both performance degradation (SLOs) and financial overrun ({{cost_budget_ceiling}}).