Bash and Python Pipeline for Sanitizing Building IoT Sensor Telemetry

Role: Smart Building Systems Integrator and Telemetry Data Quality Specialist.

Context

• Raw Sensor File Directory: {{telemetry_source_directory}}
• Target HVAC Zone: {{hvac_zone_identifier}}
• Expected Sampling Cadence: {{sensor_sampling_rate_minutes}}
• Outlier Threshold (Z-score): {{outlier_std_cutoff}}
• Max Consecutive Gap Imputation: {{gap_fill_window}}
• Clean Telemetry Directory: {{destination_archive_path}}

Task

Create an automated data sanitization script combining Bash automation and a Python core to ingest raw telemetry CSV logs from {{telemetry_source_directory}} for {{hvac_zone_identifier}}, eliminate sensor reading spikes beyond {{outlier_std_cutoff}} standard deviations, resample timestamps to uniform {{sensor_sampling_rate_minutes}}-minute intervals, impute small gaps up to {{gap_fill_window}} periods, and save cleaned outputs to {{destination_archive_path}}.

Method

1. Scan {{telemetry_source_directory}} using Bash to identify uncompressed CSV telemetry streams matching {{hvac_zone_identifier}}.
2. Load individual sensor records (temperature, humidity, CO2, power draw) into a unified time-indexed data structure.
3. Detect and remove hardware sensor malfunction values (e.g., -999.0, nulls, out-of-range physical sensor extremes).
4. Apply a rolling z-score filter with {{outlier_std_cutoff}} to identify and mask transient spikes without shifting baseline trends.
5. Resample the sensor time-series to a strict {{sensor_sampling_rate_minutes}}-minute regular grid.
6. Interpolate missing values for signal gaps smaller than or equal to {{gap_fill_window}} using forward-fill or linear interpolation.
7. Flag and isolate large gaps exceeding {{gap_fill_window}} without inventing synthetic continuous data.
8. Write the sanitized time-series to {{destination_archive_path}} in partitioned Parquet or clean CSV format with a processing metadata summary.

Constraints

• The solution MUST handle streaming files sequentially without exceeding available system memory limits.
• The script MUST NOT interpolate or synthesize values when sensor gaps exceed {{gap_fill_window}} consecutive intervals.
• Provide clear Bash error trapping (set -euo pipefail) alongside Python try-except blocks.
• All generated files in {{destination_archive_path}} must preserve original sensor hardware IDs.

Output format

Provide the complete implementation divided into two clear script blocks:

1. clean_telemetry.py: Python script accepting CLI arguments for {{hvac_zone_identifier}}, {{outlier_std_cutoff}}, and {{sensor_sampling_rate_minutes}}.
2. run_pipeline.sh: Wrapper Bash script managing directories, verifying file presence in {{telemetry_source_directory}}, and invoking the Python worker.

Self-review

• Ensure all 6 context variables are present and correctly referenced across the scripts.
• Check that time-series resampling does not cause timestamp drifting or duplicate index errors.
• Verify that Bash scripts contain appropriate directory existence checks before execution.