Telemetry Log Sanitization Specification
Define automated cleaning, deduplication, and PII masking rules for high-throughput distributed application logging pipelines.
Use this specification template when engineering data pipelines to clean raw application logs and distributed traces prior to ingestion into observability backends. It provides strict parsing, filtering, and redaction logic for software engineering teams.
Role: Principal Observability Engineer with 12 years of experience architecting distributed telemetry pipelines and structured logging systems.
Context
- Target System: {{system_name}}
- Raw Ingestion Format: {{log_source_format}}
- Masking Requirements: {{pii_redaction_rules}}
- Volume & Drop Policy: {{sampling_threshold}}
- Destination Storage: {{downstream_sink}}
- Lifecycle Constraints: {{retention_window}}
Task
Draft a comprehensive data cleaning specification to transform raw, noisy log streams into structured, sanitized, and query-optimized telemetry records for {{downstream_sink}}.
Method
- Identify all malformed structural tokens in the incoming {{log_source_format}} stream and define fallback parsing schemas.
- Map regex patterns against string fields to enforce {{pii_redaction_rules}} without breaking JSON payloads.
- Establish deduplication keys across identical trace spans generated during recursive error loops.
- Design discard criteria based on {{sampling_threshold}} to drop non-actionable debug logs.
- Standardize timestamp formatting to ISO-8601 UTC across all microservice origins in {{system_name}}.
- Normalize log severity levels into standard RFC-5424 numerical and string constants.
- Detail dead-letter queue routing for corrupted records that fail normalization.
- Specify field-level compression strategies to align with the {{retention_window}} storage budget.
Constraints
- MUST guarantee that raw credit card numbers, auth tokens, and personal data are irrecoverably masked.
- MUST NOT drop ERROR or FATAL log lines regardless of the {{sampling_threshold}}.
- Processing latency overhead must remain under 5 milliseconds per record.
- Output schemas MUST conform precisely to the ingest API of {{downstream_sink}}.
Output format
Provide the specification organized into four sections:
- Ingestion & Pre-Filtering Rules (bulleted criteria)
- Field-Level Transformation Matrix (table: Source Field, Cleaning Action, Regex/Rule, Target Type)
- Dead-Letter & Error Quarantine Protocol (step-by-step handling)
- Sink Alignment & Performance Constraints (numbered list) Limit response to under 750 words.
Self-review
- Confirm that every variable including {{system_name}} and {{pii_redaction_rules}} is addressed.
- Verify all transformation actions handle null or empty string corner cases.
- Ensure no PII patterns are inadvertently exposed in dead-letter routing.
Explicit role, a named task, and discrete steps the model can follow.
Background, inputs and variables the model needs before it starts.
Hard boundaries — what the model must and must not do.
A named, field-level shape for the response.
Ordered work items that force analysis before an answer.
Length and structure that travel across frontier models.
Signal density — instruction weight without padding.
Documented variables so the scaffold adapts to new inputs.
Quality bar, assumptions and behaviour when inputs are thin.
How much real usage the template has behind it.