Microservice Distributed Trace Anomaly Harvester Script

Role: Director of Engineering Operations & Observability Systems Lead

Context

• Target Kubernetes Namespace: {{cluster_namespace}}
• Observability Engine: {{observability_backend}}
• Metric Degradation Rate: {{target_error_budget_burn}}
• Trace Sampling Ratio: {{sampling_rate_percent}}
• Tracking Ticket: {{incident_ticket_id}}
• Critical Upstream/Downstream Services: {{critical_service_dependencies}}

Task

Author a high-performance Python operations script that connects to {{observability_backend}}, extracts anomalous distributed spans exceeding normal variance, isolates downstream bottlenecks across {{critical_service_dependencies}}, and exports an annotated debugging bundle tagged with {{incident_ticket_id}}.

Method

1. Configure authenticated client sessions with retry backoff for {{observability_backend}}.
2. Query trace indexes filtered by {{cluster_namespace}} matching the active burn window of {{target_error_budget_burn}}.
3. Apply dynamic filtering based on {{sampling_rate_percent}} to capture high-value degraded traces without memory exhaustion.
4. Walk the span tree to identify critical path latency contributors and HTTP/gRPC status codes greater than 499.
5. Correlate parent-child span anomalies specifically across {{critical_service_dependencies}} to localize network timeouts or thread pool saturation.
6. Strip and sanitize sensitive runtime data (PII, authorization headers, bearer tokens) from span attribute payloads.
7. Aggregate common error patterns into signature groups using normalized stack trace fingerprinting.
8. Serialize the resulting diagnostics into a structured JSON report and push an incident summary to engineering channels.

Constraints

• MUST implement client-side PII redacting prior to dumping trace payloads to disk or remote streams.
• MUST NOT exceed 256MB memory footprint during extraction; use streaming generators for span processing.
• Must use Python standard libraries plus official client SDKs (requests, opentelemetry, or specific vendor packages).
• Output bundle must be formatted strictly for automated ingestion by triage bots.

Output format

Provide the deliverable in two distinct sections:

1. SCRIPT_SETUP: Virtual environment requirements, SDK installation commands, and environment variable declarations.
2. SCRIPT_CODE: Complete, type-hinted Python 3.11+ diagnostic script (max 280 lines) including error handling and structured terminal output.

Self-review

• Confirm memory usage is bounded via streaming iterators over trace query responses.
• Verify that regex redaction rules scrub authorization headers and personal data markers.
• Validate that span graph traversal accounts for asynchronous and detached spans without recursion limit errors.