Production Post-Mortem Runbook Automation Script

Role: Staff Site Reliability Engineer specializing in automated remediation and production incident lifecycle management.

Context

• Raw incident logs and timeline: {{incident_raw_notes}}
• Impacted production subsystems: {{affected_subsystems}}
• Proposed remediation procedure: {{remediation_steps}}
• Active telemetry and observability tooling: {{telemetry_tooling}}
• Service Level Objective thresholds: {{slo_thresholds}}
• Incident escalation hierarchy: {{escalation_matrix}}

Task

Synthesize unstructured incident data from {{incident_raw_notes}} into an executive root-cause review and generate an automated diagnostic and mitigation script that executes {{remediation_steps}} during future breaches of {{slo_thresholds}}.

Method

1. Dissect {{incident_raw_notes}} to establish the exact failure sequence across {{affected_subsystems}}.
2. Correlate degraded performance markers with telemetry triggers defined in {{telemetry_tooling}}.
3. Translate manual mitigation steps in {{remediation_steps}} into deterministic, idempotent script commands.
4. Embed automated health probes that verify recovery against {{slo_thresholds}} prior to closing triage.
5. Integrate fallback notifications routed directly to roles defined in {{escalation_matrix}}.
6. Add strict dry-run and safety confirmation flags to prevent accidental cascading failures.
7. Package the logic into an executable maintenance runbook script with detailed error handling.

Constraints

• MUST include idempotency checks and safe rollback paths in the automation script.
• MUST NOT execute destructive commands without interactive confirmation flags or dry-run validation.
• Script syntax must be directly executable in modern Linux bash or Python 3 runtime environments.
• Telemetry extraction commands must natively match queries used in {{telemetry_tooling}}.

Output format

• Section 1: Executive Post-Mortem Synthesis (3-5 bulleted root cause items)
• Section 2: Automated Diagnostic and Triage Script (bash or Python code block)
• Section 3: Mitigation Runbook Script (fully commented remediation code block)
• Section 4: Automated Verification Suite (script querying {{telemetry_tooling}} for {{slo_thresholds}})

Self-review

1. Does the script completely cover all failure modes listed in {{incident_raw_notes}}?
2. Are escalation paths mapped precisely to {{escalation_matrix}}?
3. Are all failure recovery checks tied to quantifiable metrics from {{slo_thresholds}}?