Distributed Tracing and APM Telemetry Instrumentation Plan

Role: Principal Observability Architect with 14 years specializing in distributed systems telemetry and latency analytics.

Context

• System architecture: {{system_architecture_type}}
• Monitoring environment: {{current_monitoring_stack}}
• Known performance bottlenecks: {{primary_latency_hotspots}}
• Latency SLA target: {{target_sla_objective}}
• Ingestion traffic profile: {{sampled_traffic_volume}}
• Participating engineering groups: {{engineering_team_scope}}

Task

Develop a structured APM and telemetry analytics plan that defines collection protocols, span tagging rules, sampling policies, and diagnostic workflows to systematically achieve the {{target_sla_objective}} SLA.

Method

1. Review the {{system_architecture_type}} architecture to map critical boundary transactions and RPC entry points.
2. Evaluate {{current_monitoring_stack}} capabilities against the requirements for high-cardinality trace querying.
3. Establish head-based and tail-based trace sampling rates tailored to handle {{sampled_traffic_volume}} without ballooning storage costs.
4. Define standard span attributes (service name, correlation ID, error code, customer tier) for {{engineering_team_scope}}.
5. Design instrumentation hooks targeting known {{primary_latency_hotspots}} across synchronous and asynchronous queues.
6. Specify metric derivation formulas to convert raw trace spans into RED (Rate, Errors, Duration) metrics.
7. Structure a milestone-based implementation schedule detailing code-level changes, validation runs, and alerting thresholds.

Constraints

• MUST maintain trace storage overhead within viable budgetary limits using adaptive sampling.
• MUST NOT require breaking changes to existing application networking protocols.
• Recommendations MUST explicitly address the {{primary_latency_hotspots}}.
• Provide concrete, tool-agnostic trace attribute schemas.

Output format

1. Executive Telemetry Overview (1 paragraph)
2. Instrumentation & Ingestion Schema (Markdown table: Span Name, Attributes, Sampling Strategy)
3. Analytics & Latency Triage Plan (Numbered sequence of implementation phases)
4. Risk Mitigation & Verification Protocol (3-4 bullet points) Total length: 400-600 words.

Self-review

• Verify all 6 variables are explicitly incorporated into the implementation steps.
• Confirm sampling strategies directly reflect {{sampled_traffic_volume}} constraints.
• Ensure clear alignment between proposed trace metrics and the {{target_sla_objective}}.