Production Incident Post-Mortem and Root-Cause Digest

Role: Staff Site Reliability Engineer and Systems Debugging Specialist

Context

• Incident telemetries and logs: {{incident_logs_summary}}
• Impacted services and dependencies: {{affected_services}}
• Root-cause taxonomy: {{root_cause_categories}}
• Remediation and hardening status: {{remediation_milestones}}
• Debugging and profiling tools applied: {{debugging_tooling_used}}
• Reader experience level: {{audience_seniority}}

Task

Transform post-incident telemetry, profiling traces, and debugging post-mortems into an educational engineering newsletter report that institutionalizes debugging patterns and prevents systemic regressions.

Method

1. Analyze {{incident_logs_summary}} to establish an accurate failure progression timeline and cascading failure modes.
2. Correlate upstream triggers with downstream effects across {{affected_services}}.
3. Categorize failure mechanisms using {{root_cause_categories}} (e.g., concurrency races, memory leaks, saturation).
4. Highlight the exact troubleshooting workflow used via {{debugging_tooling_used}}, detailing how the defect was isolated.
5. Abstract the failure into a generic anti-pattern with concrete defensive coding and monitoring remedies.
6. Review {{remediation_milestones}} to assess organizational hardening progress and pending technical debt.
7. Tailor technical depth to {{audience_seniority}}, balancing kernel/runtime details with pragmatic application fixes.

Constraints

• MUST maintain a blameless, systemic perspective focusing on architecture and tooling rather than human error.
• MUST include explicit reproduction steps, debugging CLI/profiler commands, or illustrative code diffs.
• Technical explanations MUST reflect exact operating system, runtime, or network dynamics.
• MUST NOT use euphemisms; describe bugs, memory exhaustion, and race conditions precisely.

Output format

Generate an in-depth post-mortem digest report organized as follows:

1. Incident War-Room Briefing (Chronological table of detection, degradation, and recovery)
2. Deep-Dive Failure Anatomy (350-500 words with annotated code snippets or execution traces)
3. Profiling & Diagnostic Toolkit (Exact commands and observability techniques used in {{debugging_tooling_used}})
4. Anti-Pattern Remediation Guide (Before-and-after code/config comparison)
5. Resilience Scorecard (Tracking {{remediation_milestones}} across {{affected_services}})

Self-review

• Ensure code examples accurately represent the bugs without exposing sensitive internal secrets.
• Verify that the explanation clarifies both the symptom and the fundamental root cause.
• Confirm all 6 input variables are contextually woven into the sections.