Distributed Trace Anomaly Triage and Incident Remediation Chain

Role: Principal Distributed Systems Architect specializing in observability and autonomous incident response.

Context

• Microservice Topology: {{microservice_topology}}
• Telemetry & Tracing Stack: {{telemetry_stack}}
• Service Level Objective Breach Profile: {{slo_breach_profile}}
• Agent Orchestration Engine: {{agent_orchestrator_engine}}
• Parallel Execution Concurrency Limit: {{concurrency_limit}}

Task

Architect an automated workflow chain utilizing {{agent_orchestrator_engine}} that analyzes distributed trace anomalies during a {{slo_breach_profile}} incident across {{microservice_topology}}, leverages {{telemetry_stack}} for root-cause isolation, and outputs a validated remediation plan respecting {{concurrency_limit}}.

Method

1. Define the telemetry ingestion node to query spans, latency histograms, and error metrics from {{telemetry_stack}}.
2. Design the topological correlation agent that traverses upstream and downstream dependency graphs of {{microservice_topology}}.
3. Formulate the anomaly isolation node to isolate anomalous span attributes, database locks, or resource saturation points.
4. Construct the hypothesis evaluation agent chain running parallel diagnostic queries up to {{concurrency_limit}}.
5. Specify the causal graph reconstruction mechanism to rank potential root causes with confidence scoring.
6. Detail the remediation synthesis agent to map root causes to pre-approved runbook actions or configuration rollbacks.
7. Define safety validation gates to verify that proposed mitigations do not induce cascading failures across dependent services.

Constraints

• MUST enforce read-only access for all exploratory diagnostic queries executed by agents.
• MUST NOT trigger active remediation actions without satisfying deterministic confidence criteria (>90%).
• Graph traversal algorithms MUST terminate within predefined timeout limits to prevent circular dependency deadlocks.
• All agent trace analyses must produce human-auditable causal dependency paths.

Output format

1. Telemetry Ingestion & Correlation Architecture (data flow diagram & query contracts)
2. Agent Chain State Machine & Tool Matrix (node definitions, inputs, context retention, tool schemas)
3. Safety and Concurrency Policy (rules enforcing {{concurrency_limit}} and query throttling)
4. Incident Diagnostic & Remediation Output Template (structure for root cause analysis and action plan)

Self-review

• Ensure the workflow handles missing or corrupted distributed trace spans gracefully.
• Confirm that parallel hypothesis validation strictly respects {{concurrency_limit}}.
• Check that every remediation step maps to safe, deterministic operational parameters.