Smart Meter Interval Data Reconciliation Pipeline

Role: Principal Power Distribution Data Engineer specializing in Advanced Metering Infrastructure (AMI) analytics.

Context

• Regional grid operator: {{grid_operator}}
• Raw meter payload format: {{meter_data_format}}
• Meter read cadence: {{interval_duration_minutes}}
• Billing tariff matrix: {{tariff_schedule_table}}
• Raw file storage bucket: {{staging_bucket_uri}}
• Pipeline failure sink: {{error_log_destination}}

Task

Develop an automated Python batch processing script that ingests raw interval consumption files from {{staging_bucket_uri}} in {{meter_data_format}}, validates interval continuity according to {{interval_duration_minutes}}, computes aggregate kWh usage matched to {{tariff_schedule_table}}, and routes corrupted records to {{error_log_destination}} for {{grid_operator}}.

Method

1. Initialize storage connection clients and ingest batch files from {{staging_bucket_uri}}.
2. Parse incoming {{meter_data_format}} meter records and map fields to internal data classes.
3. Validate meter sequence timestamps against the expected {{interval_duration_minutes}} cadence to identify gaps.
4. Perform validation checks on raw registers: flag zero-consumption anomalies, reverse energy flow, and negative values.
5. Map interval timestamps to active time-of-use pricing windows defined in {{tariff_schedule_table}}.
6. Aggregate interval kilowatt-hours into peak, shoulder, and off-peak billing buckets per meter identifier.
7. Route failed records and syntax anomalies into {{error_log_destination}} with machine-readable error reasons.
8. Emit a structured reconciliation summary containing total meters processed, error count, and aggregated kWh.

Constraints

• MUST process data using streaming chunk iterators to avoid out-of-memory errors on large batch files.
• MUST NOT drop invalid records silently; all discarded reads must append to {{error_log_destination}}.
• Execution runtime MUST complete without depending on proprietary enterprise billing SDKs.
• Script must provide deterministic outputs when re-run against identical raw source files.

Output format

1. Script Configuration

• Configuration block mapping cloud storage pointers and parser flags (under 20 lines).

2. Processing Script

• Complete Python ETL script containing parsing, validation, and tariff mapping functions (110-190 lines).

3. Verification Commands

• Shell commands demonstrating local execution and error destination verification (under 15 lines).

Self-review

• Ensure the gap detection logic correctly accounts for daylight saving time boundaries.
• Verify all variables from context are explicitly referenced and configured in the script.
• Confirm that mathematical conversions between kW instantaneous demand and kWh interval energy are accurate.