Author: Cristhian Carlos Mamani Cori
A useful data engineering project should do more than download an API response. It should preserve the source data, transform it into a clear analytical model, reject invalid results, and publish an output that anyone can inspect.
This tutorial walks through Peru Weather Observatory, a small batch pipeline for Tacna, Lima, and Arequipa. It uses Python, DuckDB, dbt, GitHub Actions, and GitHub Pages. No AWS services or AWS credentials are used.
1. Define the question and the data contract
The dashboard answers a simple question: how did temperature and precipitation differ across three Peruvian cities over a recent 14-day period?
The source is the Open-Meteo Historical Weather API. It provides gridded weather reanalysis, which must not be confused with direct observations from a local weather station. The pipeline requests daily maximum temperature, minimum temperature, and precipitation, using the America/Lima timezone.
The window ends seven days before the run date to allow time for reanalysis availability. Each run expects exactly 42 city-day records: three cities × 14 days.
One metric deserves a precise definition. The dashboard's average daily temperature midpoint is:
average((daily maximum temperature + daily minimum temperature) / 2)
This is a convenient comparison metric, but it is not the true daily mean temperature.
2. Choose an architecture that matches the workload
Open-Meteo API
↓
Python ingestion → raw JSON snapshots
↓
DuckDB: raw.weather
↓
dbt: stg_weather → city_summary
↓
Data quality tests
↓
Static HTML + CSV + JSON
↓
GitHub Pages
GitHub Actions runs the sequence after a push to main, on manual dispatch, and on a daily schedule. This workload does not require a continuously running orchestration server. dbt manages the SQL model dependencies, while GitHub Actions handles the surrounding execution and deployment steps.
DuckDB keeps the analytical database in a local file. This makes the project easy to reproduce on a laptop and on an ephemeral runner. The dbt-duckdb adapter connects dbt to that file.
3. Ingest data without hiding failures
The ingestion script, pipeline/ingest.py, requests one payload per city. Requests have a timeout and up to three attempts with backoff. Before loading the database, the script verifies the exact dates, array lengths, and presence of every requested measurement.
Each raw snapshot includes the source URL, fetch timestamp, and original response. These files make it possible to understand what entered the pipeline.
Only after all three responses pass ingestion validation does the script replace raw.weather within a transaction. A partially fetched dataset is therefore not promoted as a successful raw load.
The key loading pattern is:
con.execute('begin')
con.execute('create schema if not exists raw')
con.execute('create or replace table raw.weather(...)')
con.executemany('insert into raw.weather values (?, ?, ?, ?, ?)', rows)
con.execute('commit')
The abbreviated table definition above illustrates the transaction; the repository contains the complete executable script.
4. Transform the data with dbt
The staging model assigns explicit types and analytical names. The summary model then aggregates the daily records:
select
city,
count(*) as days,
min(weather_date) as start_date,
max(weather_date) as end_date,
round(avg((max_temp_c + min_temp_c) / 2), 1)
as avg_midpoint_temp_c,
round(max(max_temp_c), 1) as peak_temp_c,
round(sum(precipitation_mm), 1) as total_precipitation_mm
from {{ ref('stg_weather') }}
group by city
The ref function declares the dependency on the staging model. This is more maintainable than manually sequencing unrelated SQL files.
5. Make data quality a deployment gate
The workflow uses dbt build --profiles-dir . to materialize models and execute their tests. Column tests reject null city names, dates, and weather measurements. The summary city must be unique and non-null.
A custom SQL test also rejects:
- Duplicate city/date keys.
- A minimum temperature greater than the maximum.
- Negative precipitation or temperatures outside broad physical bounds.
- A city with anything other than 14 daily records.
- A summary containing anything other than three cities.
A dbt SQL test fails when its query returns rows. For example:
select city
from {{ ref('stg_weather') }}
where min_temp_c > max_temp_c
or precipitation_mm < 0
These checks are not a guarantee that the weather estimates are scientifically exact. They establish a practical contract for what the dashboard is willing to publish.
The export script also inspects target/run_results.json and refuses to publish without passing dbt tests. If ingestion, transformation, validation, or export fails, the deployment job does not run. The previous public dashboard remains available with its original update timestamp.
6. Deploy through automation
The repository contains the full workflow at .github/workflows/pipeline.yml. Its central execution steps are:
- name: Ingest public weather data
run: python pipeline/ingest.py
- name: Transform and validate
run: dbt build --profiles-dir .
- name: Export tested dashboard
run: python pipeline/export.py
- uses: actions/upload-pages-artifact@v3
with:
path: site
A separate job has needs: build and uses actions/deploy-pages@v4. Only that job receives pages: write and id-token: write; ordinary build steps receive read access to the repository.
In Settings → Pages → Build and deployment, the publishing source is GitHub Actions. This follows GitHub's custom Pages workflow model.
The daily trigger is 17 11 * * *: 11:17 UTC, or 06:17 in Peru. Scheduled GitHub Actions runs can be delayed, so the dashboard displays its actual generation timestamp rather than promising an exact refresh time.
7. Inspect the public result
The published dashboard contains one summary card per city, the date window, the generation timestamp, and the count of passing data tests. Readers can expand the complete daily table or download CSV and JSON files.
The workflow also retains raw responses, the DuckDB database, and dbt execution artifacts for seven days. The Actions history provides a public trail from a source commit to the build and deployment jobs.
8. Reproduce the project locally
With Python 3.11 installed:
git clone https://github.com/Cristhian465/peru-weather-data-pipeline.git
cd peru-weather-data-pipeline
python -m venv .venv
source .venv/bin/activate
pip install -r requirements.txt
python pipeline/ingest.py
dbt build --profiles-dir .
python pipeline/export.py
python -m http.server 8000 --directory site
Open http://localhost:8000 to inspect the dashboard. The same scripts run in GitHub Actions, so local development and public deployment share the same transformation logic.
Tradeoffs and next steps
This project rebuilds a rolling window on each run. The database on a hosted runner is temporary; the workflow artifacts provide short-term evidence, not a permanent historical warehouse. There is no fallback to fabricated sample data when the API is unavailable.
For a larger system, useful next steps would be persistent storage, incremental loads, alerts for stale releases, a fully locked dependency graph, and more detailed source monitoring. Kafka or Airflow would become appropriate if the workload required streaming, many dependent jobs, or more operational control.
For this educational batch workload, the result is small enough to understand end to end: a public source, a typed analytical model, executable data contracts, an automated deployment, and a public SaaS-hosted output.
Data attribution: Weather data by Open-Meteo, licensed under CC BY 4.0. The free API is intended for non-commercial use; consult current provider terms for other uses.
This project and article were prepared with AI assistance. The source code and workflow history are public so readers can inspect and reproduce the implementation.
Top comments (0)