Distributed Tracing and Latency Triage Dashboard Specification

Role: Principal Site Reliability Engineer specializing in distributed systems observability

Context

• Service mesh infrastructure: {{service_mesh_name}}
• P99 latency threshold target: {{target_p99_latency_ms}} ms
• Observability and tracing tools: {{telemetry_stack}}
• Operational stability context: {{incident_frequency}}
• Monitored request workflows: {{critical_service_paths}}

Task

Author an engineering dashboard specification report that defines widget layouts, query primitives, and visualization rules for debugging distributed microservice latency spikes.

Method

1. Analyze the upstream-to-downstream dependencies documented in {{critical_service_paths}} to map out required span hierarchies.
2. Formulate telemetry query logic for {{telemetry_stack}} capturing RED metrics (Rate, Errors, Duration) across the {{service_mesh_name}} mesh.
3. Establish latency heatmaps isolating anomalous long-tail distributions relative to {{target_p99_latency_ms}} ms.
4. Design span waterfall drill-down panels that correlate p99 tail events with specific downstream database and remote RPC calls.
5. Incorporate visual alerting thresholds that reflect patterns identified in {{incident_frequency}}.
6. Specify dynamic dashboard filtering mechanisms based on HTTP status codes, deployment versions, and route tags.
7. Map out remediation action buttons directly linking widget views to source-code traces and runbooks.

Constraints

• MUST express queries using syntax compatible with {{telemetry_stack}}.
• MUST focus exclusively on low-overhead, high-signal operational debugging metrics.
• MUST NOT include vanity aggregate metrics like raw total request count without error and latency context.
• All panel specifications must include a defined refresh rate and aggregation window.

Output format

Provide a technical report structured in four sections:

1. Architectural Scope & Metric Primitives (150-200 words)
2. Panel-by-Panel Layout Specification (numbered list with panel name, metric query, visualization type, and drill-down action)
3. Threshold & Alerting Matrix (markdown table: Metric, Warning, Critical, Trigger Window)
4. On-Call Triaging Workflow (bulleted runbook steps, under 150 words)

Self-review

• Ensure every critical path listed in {{critical_service_paths}} has a dedicated span view.
• Verify that p99 latency boundaries match {{target_p99_latency_ms}} ms exactly across panels.
• Confirm metric queries adhere to {{telemetry_stack}} conventions.