Microservice Latency Degradation and Circuit Breaker Audit Checklist

Role: Principal Systems Architect and Observability Operations Lead

Context

• Network routing overlay: {{service_mesh_topology}}
• High-percentile latency limit: {{p99_latency_sla_ms}}
• Distributed tracing infrastructure: {{observability_backend_platform}}
• Upstream dependency graph: {{upstream_caller_services}}
• Ingress and egress routing layer: {{egress_gateway_clusters}}
• Active resiliency configuration: {{circuit_breaker_policy}}

Task

Produce an operational triage and circuit-breaker verification checklist for diagnosing microservice latency degradation, isolating bottlenecked spans, and validating protective load-shedding mechanisms across the service mesh.

Method

1. Define operational triage steps to isolate latency anomalies breaching {{p99_latency_sla_ms}} within {{observability_backend_platform}}.
2. Construct verification items for ingress/egress saturation across {{egress_gateway_clusters}}.
3. Detail inspection routines for thread pool exhaustion, socket starvation, and queue depth inside {{service_mesh_topology}}.
4. Design specific audit checkpoints to verify {{circuit_breaker_policy}} tripping behavior under sustained latency.
5. Outline blast-radius containment checks to shield {{upstream_caller_services}} from connection pool depletion.
6. Formulate synthetic degradation drills to confirm graceful degradation and fallback cache returns.
7. Detail operational recovery verification before resetting tripped circuit breakers.

Constraints

• MUST define specific distributed trace tags and metric query types for each diagnostic step.
• MUST structure checklist items in active operational phrasing targeting on-call reliability engineers.
• MUST NOT recommend manual service restarts without prior thread dump and trace capture.
• The complete checklist MUST contain between 16 and 22 actionable verification items.

Output format

1. Telemetry Isolation & Root Cause Identification Checklist

(5-6 checklist items [ ] covering trace span analysis, P99 breach identification, and log correlation)

2. Mesh & Gateway Contention Audit Checklist

(5-6 checklist items [ ] checking egress pools, connection limits, and proxy memory limits)

3. Circuit Breaker & Resiliency Validation Checklist

(4-5 checklist items [ ] verifying trip thresholds, fallback mechanisms, and shed ratios)

4. Safe Recovery & Traffic Re-Arming Checklist

(3-4 checklist items [ ] detailing gradual ramp-up, warm-up routines, and circuit breaker reset)

Self-review

• Verify all 6 context variables ({{service_mesh_topology}}, {{p99_latency_sla_ms}}, {{observability_backend_platform}}, {{upstream_caller_services}}, {{egress_gateway_clusters}}, {{circuit_breaker_policy}}) are incorporated.
• Confirm every checkpoint has a distinct observability or infrastructure validation target.
• Ensure no placeholder text or vague instructions appear anywhere in the output.