DEV Community: Bechir Bejaoui

From YAML validator to event-driven projects: 14 releases of Hydra ETL in six weeks

Bechir Bejaoui — Wed, 04 Nov 2026 22:34:34 +0000

At the end of September, Hydra ETL did one thing well: you described a data job in four YAML files, hdrctl validate checked them before any data was read, and the same job ran from the terminal, a REST API or a visual editor.

Six weeks and 14 releases later, a Hydra ETL project can start itself when a file lands in a folder, roll back what it wrote when something fails, aggregate 20 million rows in about half a minute, and call another project as a step. The latest one, 0.19.0, shipped on November 4, which also happens to be my birthday. Not a bad way to celebrate.
This post walks through that evolution in five steps, and what stayed the same.

The starting point: a job you can read and check

A Hydra ETL job is four files: sources.yaml, transformations.yaml, destinations.yaml, pipeline.yaml. Nothing runs until the manifests are valid. That was the promise in September, and every release since has kept it: each new feature comes with its own checks, run before anything touches your data.

1. The project becomes the unit (0.11.4 → 0.12.3)

Teams don't run one job; they run a set of them in a certain order.

hdrctl init now scaffolds a project: jobs, workflows, parameters, environments.
workflows/thread.yaml declares the order: stages, sequential or parallel, delays, loops.
hdrctl run <project> checks the whole project — every workflow, every job, every pipeline.yaml — and lists every problem before starting anything.
Studio stopped overwriting files edited outside it: a change made in your editor or by git is detected, and the latest change wins.

2. Failures stop leaving half-written data (0.13 → 0.14)

This is the part I care most about. A pipeline that fails halfway is worse than one that never starts.

All-or-nothing load per job: a job writes through one load session. A failure or Ctrl+C leaves the destination as it was (transactions on PostgreSQL, MySQL/MariaDB and SQL Server; temporary file then swap on CSV and Parquet).
Workflow rollback: with rollback: true, a failed workflow undoes the steps it already did, last one first. Actions that touch the outside world (webhook, email, scripts) must declare how to undo them, or explicitly say they can't.
Fallback workflows: a backup workflow can take over — before the start, when a condition is true (not source_available('crm_db')), or after a failure, but only if the failed workflow published nothing.
Timeouts per workflow.

3. Volume (0.15 → 0.16)

pandas stays the default, but above two million estimated rows Hydra ETL can read CSV with pyarrow and run aggregations, sorts and joins in DuckDB. Results are identical to pandas, sums included.

Measured on 20 million rows, 2 CPUs:

aggregation: 112.8 s / 5.4 GB → 34.3 s / 1.4 GB
right join then aggregation: 71.8 s / 3.4 GB → 27.7 s / 2.6 GB

No silent fallback: when a step can't run in DuckDB, the log says so. And a new engine: key chooses the engine per job.

4. Secrets and environments (0.17 → 0.18)

Secrets are masked everywhere Hydra ETL writes text: logs, errors, tracebacks, the output of scripts, stored runs.
Secrets can be encrypted with SOPS (age, PGP, cloud KMS) and are decrypted in memory only.
One project, several environments: global variables per machine, parameter values and secrets per environment, managed with hdrctl env, hdrctl param and hdrctl secret. A secret value is never passed on the command line.

5. Projects that start themselves (0.19)

The latest release adds project triggers. A triggers.yaml at the project root:

version: "1.0"
triggers:
  - name: nightly
    type: cron
    cron: "0 2 * * *"
    timezone: Europe/Paris

  - name: new_orders
    type: file
    path: inbox/*.csv
    debounce: 2s

  - name: after_crm
    type: after_project
    project: ../crm
    "on": success

hdrctl trigger validate checks it, then hdrctl watch runs it. Dropping a CSV into inbox/, then finishing the crm project, gives:

  Waiting for events. Ctrl+C to stop.
  2026-11-04 21:29:36 ▶ sales/new_orders  files: inbox/orders_2026-11-04.csv  env: none
  2026-11-04 21:29:39 ✓ sales/new_orders  2.6s
  2026-11-04 21:30:07 ▶ sales/after_crm  project crm ended: success  env: none
  2026-11-04 21:30:12 ✓ sales/after_crm  4.9s

Seven trigger types in total: cron, interval, file, marker file, change in a source table, webhook with a token, and the end of another project.

The guarantees are what make this usable:

a file is only read once it has stopped changing;
a trigger already running is not started twice;
a change to triggers.yaml is picked up without restarting;
after_project loops are refused.

The same release adds sub-projects: a workflow step type: project runs another project, with cycle detection and results the next steps can read.

The six weeks at a glance

Version	Date	What changed
0.11.3–0.11.4	Sep 27 – Oct 17	Secrets resolve; PostgreSQL upsert fixed; projects
0.12.x	Oct 22–24	`thread.yaml`, `hdrctl run <project>`, loops, Studio ↔ disk sync
0.13–0.14	Oct 26–28	All-or-nothing loads, rollback, fallback, timeouts
0.15–0.16	Oct 28–29	DuckDB large-volume engine, `engine:` per job
0.17–0.18	Nov 1–2	Masked secrets, SOPS, environments
0.19	Nov 4	Project triggers, `hdrctl watch`, sub-projects

What didn't change

Still one pip install, no database, no broker, no Node. Still YAML you can review in a pull request. Still checked before it runs.

What's not there yet

Hydra ETL is in beta, with one maintainer. File triggers watch local folders; SFTP and S3 are planned. If you try it and something breaks, an issue helps more than a star.

A Hydra Studio Quick tour

Try Hydra ETL in playground: killercoda.com/hydra-etl
Every release, with its notes: hydraetl.com/changelog
Source: github.com/bejaouibechir/Hydra
Install: pip install -U hydra-etl

Clean a million rows with Hydra ETL — no database, no Docker

Bechir Bejaoui — Sun, 20 Sep 2026 20:07:00 +0000

Originally published on hydraetl.com.

Everything below runs on a laptop with one install and no service to start: no database, no Docker, no cluster. You will generate a million-row CSV, describe the job in four small YAML files, check it before it runs, execute it, then run the exact same job on the optional Rust engine and compare.

Total time: about five minutes.

1. Install

pip install hydra-etl        # 0.10.2 or later

Python 3.9+, Linux, macOS or Windows. Nothing else.

One rule for the whole tutorial: create an empty folder, and run every command from it. That folder is the Hydra ETL job — the manifests sit at its root and data/ sits inside it. By the end it looks like this:

orders-demo/            ← your terminal stays here, always
├── gen.py
├── sources.yaml
├── transformations.yaml
├── destinations.yaml
├── pipeline.yaml
├── workflow.yaml
└── data/
    ├── orders.csv       (generated in step 2)
    └── orders_clean.csv (written in step 5)

mkdir orders-demo && cd orders-demo

If you use a virtual environment, create it here too (python -m venv .venv) and activate it before installing.

2. Make the data (no download, no hunting on your disk)

Save this as gen.py and run python gen.py. It uses the standard library only and takes a few seconds.

import csv, random
random.seed(42)
with open("data/orders.csv", "w", newline="") as f:
    w = csv.writer(f)
    w.writerow(["id", "customer_id", "country", "price", "qty"])
    for i in range(1, 1_000_001):
        w.writerow([i, random.randint(1, 50_000), random.choice(["TN","FR","DE","ES","PT"]),
                    round(random.uniform(-5, 500), 2), random.randint(1, 10)])

mkdir data && python gen.py     # from orders-demo/

You get data/orders.csv, 1,000,000 rows, about 25 MB. Note the negative prices: that is the dirt we are going to filter out.

id,customer_id,country,price,qty
1,41906,TN,7.63,5
2,16050,FR,65.47,2

3. Describe the job — four files, 30 lines

sources.yaml

version: "1.0"
sources:
  src_orders:
    type: csv
    connection:
      base_path: data
    extract:
      table: orders.csv
      batch_size: 100000

transformations.yaml

version: "1.0"
steps:
  - cast:
      mapping:
        id: int
        customer_id: int
        price: float
        qty: int
  - filter:
      expr: "price > 0"
  - calculate:
      column: total
      expr: "price * qty"

destinations.yaml

version: "1.0"
destinations:
  dst_clean:
    type: csv
    connection:
      base_path: data
    load:
      table: orders_clean.csv
      mode: replace

pipeline.yaml

version: "1.0"
pipeline:
  from: src_orders
  to: dst_clean

That is the whole job: fix the types, drop invalid rows, compute a column. A reviewer can read it in a pull request without running anything.

4. Check it before it runs

The . means "the job in the current folder", so run this from orders-demo/, not from data/:

hdrctl validate --strict .

  ok  sources.yaml            — Pydantic valid
  ok  destinations.yaml       — Pydantic valid
  ok  transformations.yaml    — Pydantic valid
  ok  pipeline.from           — resolved: src_orders
  ok  pipeline.to             — resolved: dst_clean

  Strict checks
  ok  Recognized operations   — cast, filter, calculate
  ok  filter.expr             — 'price > 0'
  ok  calculate.total         — 'price * qty'

  ✅ DSL valid — no errors detected.

No data was read and nothing was written. Try breaking something on purpose — rename dst_clean in pipeline.yaml, or write price * (qty — and run it again: validation fails with exit code 1, which is what you want in CI.

5. Run it

Still from orders-demo/:

hdrctl run .

The result lands in data/orders_clean.csv: 990,055 rows (about 1% of the rows had a negative price and were dropped) plus the new total column.

id,customer_id,country,price,qty,total
1,41906,TN,7.63,5,38.15

6. Same job, Rust engine

Part of the engine has an optional Rust implementation. It is off by default; turning it on changes nothing else about the job.

pip install "hydra-etl[native]"
HYDRA_BACKEND=rust hdrctl run .          # from orders-demo/

On Windows PowerShell, set the variable first:

$env:HYDRA_BACKEND = "rust"
hdrctl run .

Measured on two machines, three runs each, on the 1M-row file above:

Machine	Python (default)	Rust (`HYDRA_BACKEND=rust`)	Faster by
Windows 11 (25H2), Python 3.13.7	6.4 · 6.5 · 6.7 s	3.5 · 3.7 · 3.8 s	~42%
Linux VM, Python 3.10.12	8.5 · 8.7 · 9.2 s	6.4 · 6.8 · 7.6 s	~22%

Between 20% and 45% faster end to end, depending on the machine. Be careful with bigger claims you may read elsewhere, including ours: the Rust part accelerates CSV reading, and reading is only one stage of this pipeline. Speeding up one stage by 4× does not make the whole job 4× faster — and the share it represents depends on your disk, your filesystem and what else runs on the machine.

One run out of the six on the Windows laptop came out at 7.7 s with the Rust engine, twice the other two. That is what a background process looks like in a benchmark; we report it rather than hide it. Run each engine at least three times before drawing a conclusion on your own hardware.

The output is the same, byte for byte:

md5sum data/orders_clean.csv   # identical after a Python run and after a Rust run

That is the point of the Rust backend: same results, less time on the read, and an automatic fallback to Python whenever that guarantee cannot be kept.

7. Let Hydra ETL time itself

Timing a job with a stopwatch is fine once. In practice you want the pipeline to report on itself — and that is what workflows are for. A workflow orders several steps in a DAG; a step is either a job or an action (Python, Bash, PowerShell, SSH, webhook, email…).

Here, three steps: start a timer, run the job, report. The actions are Python, so the same file works on Linux, macOS and Windows.

workflow.yaml

workflow:
  name: "timed_clean_orders"
  description: "Clean one million rows and report how long it took"
  trigger:
    type: manual
  steps:
    - name: "start_timer"
      type: action
      action: python
      params:
        script: "import time, pathlib; pathlib.Path('.hydra_t0').write_text(str(time.time())); print('timer started')"
      depends_on: []

    - name: "clean_orders"
      type: job
      job: "."
      depends_on: ["start_timer"]
      on_failure: fail

    - name: "report_time"
      type: action
      action: python
      params:
        script: |
          import time, pathlib
          t0 = float(pathlib.Path(".hydra_t0").read_text())
          rows = sum(1 for _ in open("data/orders_clean.csv")) - 1
          seconds = time.time() - t0
          print(f"{rows:,} rows written in {seconds:.1f} s ({rows/seconds:,.0f} rows/s)")
      depends_on: ["clean_orders"]

Workflows are validated before they run, exactly like jobs:

Again from orders-demo/, where workflow.yaml now sits next to the four manifests:

hdrctl workflow validate ./workflow.yaml
hdrctl workflow run ./workflow.yaml

  Workflow  timed_clean_orders
  ────────────────────────────────────────────────────────

  ✓  Step 'start_timer' OK (0.0s)
     │ timer started
  ✓  Step 'clean_orders' OK (7.8s)
  ✓  Step 'report_time' OK (0.2s)
     │ 990,055 rows written in 7.9 s (124,977 rows/s)

  ✅ Workflow 'timed_clean_orders' completed in 8.0s — 3/3 steps OK

Two things happened here. The runner reported the duration of each step, which is the cheapest profiling you will ever set up. And the report came out of the pipeline itself, so the same action could just as well push it to a webhook, send it by email, or write it next to the data.

Run the same workflow with HYDRA_BACKEND=rust and you get the Python-versus-Rust comparison above without touching a stopwatch.

What this shows

A pipeline is a few readable files, versioned with your code.
hdrctl validate fails fast on your laptop or in CI, before a single row is read.
One pip install and no infrastructure.

When not to use Hydra ETL

If you orchestrate hundreds of heterogeneous tasks across a cluster, use Airflow or Dagster. If your data already lives in a warehouse, dbt is the natural fit. Hydra ETL targets file-and-database pipelines that should stay readable and run without infrastructure. It is beta: pin your version.

Code and docs: github.com/bejaouibechir/Hydra
Try it in the browser: hydraetl.com