Environmental Monitoring for Durham, NC

A comprehensive, cloud-native system for collecting, processing, and analyzing high-resolution environmental data across Durham.

View on GitHub

Weather Analysis Notebook

Interactive Sensor Map 🔗

Data Source Note: The map and analysis currently display data exclusively from public Weather Underground (WU) stations. No private TSI data sensors are shown.

Key Features

Fully Automated: Daily data collection, processing, and verification.
High-Resolution Data: Research-grade 15-minute interval data.
Cloud-Native: Leverages GCS and BigQuery for scalability.
Continuous Verification: Daily workflows ensure data integrity.
Data Quality Monitoring: Automated checks for NULLs, coverage, and consistency.
Secure & Auditable: Keyless authentication with Workload Identity Federation.

Data Pipeline Overview

Collection: A Cloud Scheduler job triggers a Cloud Run job that executes the daily data collector script.
Storage (Raw): Raw data is uploaded as Parquet files to a GCS bucket, partitioned by source and date.
Materialization: The raw data is then materialized into partitioned BigQuery tables.
Transformation: A scheduled GitHub Actions workflow runs SQL scripts to transform the raw data into analytics-ready tables.
Quality Checks: Another GitHub Actions workflow runs quality checks against the BigQuery tables.
Visualization: Looker Studio dashboards are connected to the BigQuery tables for visualization and analysis.

Documentation Navigation

Weather Analysis Outputs

📈 Time Series & Correlation

Analyze historical trends and relationships between sensors over time.

Time Series Humidity (Daily) Time Series Pressure (Daily) Time Series Temperature (Daily) Sensor Correlation

🌍 Global Forecasts

View aggregated, forward-looking forecasts for the entire Durham area.

Global Humidity Forecast Global Pressure Forecast Global Temperature Forecast

🗺️ Spatial Analysis

Explore the geographic distribution and data from individual sensors.

Interactive Sensor Map