Multi Agent Workflow Canary Rollout and Telemetry Plan

Role: Lead Site Reliability Engineer for distributed agentic architectures.

Context

• Multi-agent orchestrator: {{agent_orchestrator}}
• Workflow DAG definition: {{workflow_dag_definition}}
• Traffic routing proxy: {{traffic_routing_tool}}
• Observability and trace backend: {{observability_backend}}
• Permissible error budget threshold: {{error_budget_threshold}}
• Fallback backup model provider: {{fallback_model_provider}}

Task

Deliver an operational canary rollout plan that manages progressive traffic shifting, real-time OpenTelemetry trace analysis, and automated degradation rollbacks for {{workflow_dag_definition}}.

Method

1. Analyze {{workflow_dag_definition}} to isolate critical path agent transitions and state handoffs.
2. Configure {{traffic_routing_tool}} to execute weighted canary routing starting at a 5% baseline.
3. Instrument distributed tracing hooks across {{agent_orchestrator}} sending spans directly to {{observability_backend}}.
4. Establish continuous monitoring for context-window exhaustion, infinite recursion loops, and invocation timeouts.
5. Define canary health metrics tracking against the {{error_budget_threshold}}.
6. Architect dynamic failover policies routing failed tool calls directly to {{fallback_model_provider}}.
7. Structure a multi-stage promotion cadence expanding traffic through 10%, 25%, 50%, and 100% phases.
8. Formulate an automated rollback trigger protocol that arrests execution and drains canary traffic within 30 seconds of anomaly detection.

Constraints

• MUST define OpenTelemetry span naming conventions for all agent-to-tool handshakes.
• MUST NOT permit manual sign-offs for stage rollbacks once {{error_budget_threshold}} is breached.
• Traffic shifting steps must include explicit soak durations between increments.
• Rollout architecture must remain fully agnostic of the underlying cloud compute provider.

Output format

• Progressive Delivery Matrix (Tabular overview of traffic percentages, soak windows, metric gates, and validation commands)
• Telemetry and Trace Schema (Detailed listing of 5-8 telemetry attributes required per agent span)
• Anomaly & Auto-Rollback Protocol (Step-by-step incident containment workflow under 400 words)
• Secondary Failover Blueprint (Specific routing rules utilizing {{fallback_model_provider}})

Self-review

• Confirm all 6 variables are referenced cleanly in context or method steps.
• Check that the Method section contains exactly eight sequentially numbered steps.
• Verify the constraints contain clear MUST and MUST NOT rules regarding automated operations.