Grid Outage Management Database Failover Verification Checklist

Role: Senior Reliability Systems Database Engineer responsible for mission-critical electrical distribution and OMS infrastructure.

Context

• OMS Database Platform: {{oms_database_engine}}
• Target RPO: {{rpo_target_seconds}}
• Target RTO: {{rto_target_minutes}}
• Secondary Standby Region: {{secondary_cluster_region}}
• Drill Maintenance Window: {{dr_test_window_hours}}
• Dependent Grid Services: {{critical_grid_services}}

Task

Author a high-availability and disaster recovery verification checklist that grid operations engineers must execute to validate failover readiness, quorum consistency, and health telemetry for the {{oms_database_engine}} cluster supporting {{critical_grid_services}}.

Method

1. Establish baseline replication latency measurements between primary nodes and {{secondary_cluster_region}}.
2. Verify health-check thresholds and automatic quorum failover triggers to prevent split-brain scenarios.
3. Validate write-ahead log shipping and transaction replay rates to guarantee RPO within {{rpo_target_seconds}}.
4. Design connection pool rerouting and DNS switchover tests against {{critical_grid_services}} within {{rto_target_minutes}}.
5. Audit read-only standby accessibility for dispatch consoles during failover transition states.
6. Formulate fallback procedures to safely re-establish primary-replica sync without data loss if failover is aborted.
7. Structure pre-flight, failover execution, and post-restoration audit checks within {{dr_test_window_hours}}.

Constraints

• Failover steps MUST include an explicit RPO validation check against {{rpo_target_seconds}}.
• Failover execution steps MUST NOT disrupt offline dispatcher caches if database reconnection exceeds {{rto_target_minutes}}.
• The checklist MUST specify exact verification commands or log indicators for {{oms_database_engine}}.
• Output MUST format every step using markdown checkboxes - [ ] with explicit step sequencing.

Output format

Provide the review formatted as three sequential checklist sections:

1. Pre-Failover Health & Replication Checks (4-5 checklist items)
2. Execution & Cluster Promotion Steps (4-5 checklist items)
3. Post-Promotion Verification & Grid Service Reconnection (4-5 checklist items) Each item must use this format: - [ ] [Step #] Action | Expected Result: <measurable state>.

Self-review

• Ensure all variables ({{oms_database_engine}}, {{rpo_target_seconds}}, {{rto_target_minutes}}, {{secondary_cluster_region}}, {{dr_test_window_hours}}, {{critical_grid_services}}) are present.
• Verify all checklist items contain actionable instructions and measurable success criteria.
• Confirm there are no markdown syntax errors or generic filler phrases.