Production Incident Severity and Failover Decision Matrix

Role: Principal Site Reliability Engineer and Infrastructure Operations Director

Context

• Service Classification and SLAs: {{service_tier_definitions}}
• Error Budget and SLO Thresholds: {{current_slo_budget}}
• Impaired Dependencies and Nodes: {{degraded_subsystems}}
• Real-Time Impact Telemetry: {{blast_radius_telemetry}}
• Available Disaster Recovery Mechanisms: {{failover_mechanisms}}
• Compliance and Data Residency Limits: {{regulatory_uptime_mandates}}

Task

Generate a comprehensive Production Incident Severity and Failover Decision Matrix that cross-evaluates active subsystem failures, data consistency risks, and operational routing strategies to guide immediate mitigation and automated recovery.

Method

1. Analyze {{blast_radius_telemetry}} against {{service_tier_definitions}} to classify overall incident severity across affected user journeys.
2. Correlate {{degraded_subsystems}} with {{current_slo_budget}} to determine remaining acceptable downtime before breach.
3. Evaluate {{failover_mechanisms}} against {{regulatory_uptime_mandates}} to flag any compliance or data sovereignty constraints during cross-region traffic shifting.
4. Calculate state synchronization lag and potential data loss risk (RPO) versus recovery execution time (RTO) for each failover route.
5. Establish deterministic threshold gates that dictate when manual intervention must yield to automated circuit-breaking and DNS repointing.
6. Formulate fallback rollbacks for split-brain scenarios or mid-failover network partitioning.
7. Synthesize findings into a multi-dimensional triage matrix mapping operational actions directly to live telemetry trigger points.

Constraints

• All operational decisions MUST balance Recovery Point Objective (RPO) against Recovery Time Objective (RTO).
• MUST NOT recommend manual data reconciliation steps inside critical path P0/P1 incident branches.
• Trigger thresholds must cite specific telemetry metrics rather than vague qualitative assessments.
• Blast-radius mitigation steps MUST explicitly identify stateful versus stateless recovery hazards.

Output format

• Section 1: Executive Incident Classification & Impact Summary (max 150 words)
• Section 2: Core Failover Decision Matrix (Markdown table with columns: Subsystem, Severity Level, Trigger Metric, Failover Mechanism, Data Loss Risk/RPO, Recovery Time/RTO, Owner Role)
• Section 3: Split-Brain & Rollback Circuit Breakers (4-6 actionable bullet points)

Self-review

1. Are all failover mechanisms technically valid for the degraded subsystems listed?
2. Does the matrix avoid data loss under active replication delays?
3. Are role responsibilities unambiguously mapped to operational roles?