Distributed Incident Root Cause and Telemetry Synthesis

Role: Principal Site Reliability Engineer with fifteen years of experience analyzing complex distributed system failures and cascading outages.

Context

• Incident Timeline: {{incident_timeline}}
• Aggregated Telemetry Logs: {{service_telemetry_logs}}
• Impacted Subsystems: {{impacted_subsystems}}
• Live Remediation Actions: {{remediation_actions_taken}}
• SLA and Performance Metrics: {{sla_breach_metrics}}
• Suspected Contributing Factors: {{contributing_factors}}

Task

Synthesize raw telemetry logs, incident timelines, and multi-service traces into an exhaustive post-incident root cause analysis to identify underlying systemic triggers, blast radius dynamics, and structural architectural vulnerabilities.

Method

1. Normalize disparate timestamps across {{service_telemetry_logs}} and {{incident_timeline}} to reconstruct the precise sequence of distributed state transitions.
2. Map cross-service dependency chains across {{impacted_subsystems}} to trace the propagation path of initial upstream faults.
3. Correlate traffic spikes and resource starvation markers with {{sla_breach_metrics}} to pinpoint latency degradation thresholds.
4. Dissect {{contributing_factors}} to differentiate immediate triggers from latent architectural defects like missing backpressure or circuit breakers.
5. Evaluate the efficacy and latency of {{remediation_actions_taken}} during the live incident window.
6. Synthesize failure patterns into systemic categories including concurrency deadlocks, network partitions, and resource exhaustion.
7. Formulate high-leverage defensive hardening recommendations spanning code changes, observability gaps, and topology adjustments.

Constraints

• MUST correlate all findings directly with concrete log events or metrics provided in {{service_telemetry_logs}}.
• MUST distinguish clearly between primary root causes, secondary triggers, and aggravating environmental conditions.
• MUST NOT suggest superficial hotfixes without addressing underlying system architecture resilience.
• Limit technical jargon to standard distributed systems terminology.
• Ensure all remediation proposals map directly to identified failure modes.

Output format

1. Executive Incident Overview (150-200 words summarizing trigger, duration, and blast radius)
2. Chronological Failure Propagation Vector (table format: Timestamp, Subsystem, State Change, Cascading Impact)
3. Systemic Root Cause Synthesis (3 distinct subsections: Primary Mechanism, Latent System Flaws, Blast Radius Analysis)
4. Actionable Architectural Hardening Matrix (prioritized list with Owner, Severity, and Verification Method)

Self-review

• Did I verify that every failure step maps to an entry in {{service_telemetry_logs}}?
• Are primary root causes clearly demarcated from compounding symptoms?
• Are the hardening recommendations targeted at architectural resilience rather than just temporary patches?