DEV Community: Alisson Rosa

Leoflow Lite: a local development environment for Apache Airflow

Alisson Rosa — Wed, 01 Jul 2026 11:54:06 +0000

TL;DR — Leoflow Lite is a local development environment for Apache Airflow.
Run leoflow lite and you get a Docker-free, Kubernetes-free local loop that
compiles and runs standard Airflow 3.2 DAGs — real provider operators, sensors,
connections, variables, XCom — with hot-reload on save and the Airflow UI,
and it picks up your local GCP / AWS / Azure credentials automatically. Install,
point it at a DAG, watch it run.

What it is

If you write Apache Airflow DAGs, you know the local-dev pain: a docker-compose with
a scheduler, a webserver, a metadata DB, a Redis, and a worker — minutes to boot,
heavy to keep running, awkward to iterate.

Leoflow Lite is the opposite. One command, no containers required, and you're
editing a DAG and watching it run against the real Airflow UI:

# install (pin the version on the sh side of the pipe)
curl -fsSL https://raw.githubusercontent.com/neochaotic/leoflow/main/install.sh | sh

leoflow lite --postgres managed     # embedded Postgres, no Docker needed
# → scaffolds a starter DAG, serves the Airflow 3.2 UI at http://localhost:8088,
#   and hot-reloads on every save.

These are standard airflow.sdk DAGs — no new DSL:

from airflow.sdk import DAG, task
from airflow.providers.standard.operators.bash import BashOperator

with DAG("hello", schedule="@daily"):
    greet = BashOperator(task_id="greet", bash_command="echo 'hi {{ ds }}'")

    @task
    def count() -> int:
        return 42

    greet >> count()

Save the file and the DAG reloads in the UI in a couple of seconds.

Lite ships a simple in-browser editor for quick edits — hit the IDE button in
the UI (bottom-right in the screenshot above) to tweak dag.py / leoflow.yaml
without leaving the page. For real work, use your favorite IDE (VS Code, PyCharm,
Vim — whatever) on the files locally; either way, every save hot-reloads.

What's supported

Lite runs the same Airflow-compatible engine as Leoflow's production mode:

Airflow 3.2 DAGs via airflow.sdk — @task, >>, schedules, trigger rules, fan-in/fan-out.
Operators & sensors — BashOperator/PythonOperator natively, and any provider operator (e.g. SQLExecuteQueryOperator, cloud transfer operators) runs for real; sensors too, including mode='reschedule'.
Connections — create them in the UI; they're delivered to the task as AIRFLOW_CONN_<ID>.
Variables — {{ var.value.x }} and the Admin → Variables UI.
XCom between tasks, Jinja templating, the run context ({{ ds }}, …).

Your local cloud credentials just work

Lite's subprocess executor runs each task as a local process under your user,
inheriting your environment and $HOME. So whatever you've already signed into with
your cloud's normal CLI is exactly what the task uses — authenticate once:

GCP — gcloud auth application-default login (Application Default Credentials), or set GOOGLE_APPLICATION_CREDENTIALS.
AWS — aws configure or aws sso login (AWS CLI configuration), or AWS_PROFILE / AWS_ACCESS_KEY_ID in the env.
Azure — az login (sign in with the Azure CLI), or AZURE_* in the env.

Then a task just uses the provider SDK — no Connection needed for local dev:

from airflow.sdk import DAG, task

with DAG("gcs_peek", schedule=None):
    @task
    def list_buckets() -> list[str]:
        from google.cloud import storage              # leoflow.yaml: dependencies: [google-cloud-storage]
        return [b.name for b in storage.Client().list_buckets()]   # uses your local ADC
    list_buckets()

storage.Client() resolves your gcloud Application Default Credentials through
Google's normal chain, exactly as it would in a script you run by hand — same idea
for boto3 (AWS) and the Azure SDKs. Author a DAG that hits BigQuery, S3, or Blob
Storage and run it against your real accounts, locally, in the dev loop.

In production/cluster mode the same DAG gets its credentials from managed
Connections instead — pods don't inherit your laptop's env.

Leoflow Lite vs the usual local-Airflow setups

	`docker-compose` Airflow	`astro dev`	Leoflow Lite
Docker required	yes	yes	no (`--postgres managed`)
Boot time	minutes	~a minute	seconds
Hot-reload	restart-ish	restart-ish	on save
Real provider operators	yes	yes	yes
Local cloud creds	manual mounts	manual	inherited automatically
Airflow UI	yes	yes	yes (3.2)

FAQ

Can I use Leoflow to develop Apache Airflow DAGs locally?
Yes. leoflow lite is a local development environment for Airflow: write a standard
airflow.sdk DAG, run it, and iterate with hot-reload — no Docker or Kubernetes.

Does it run real Airflow operators and sensors?
Yes. Native operators (bash, python) run directly; any other provider operator or
sensor runs the genuine Airflow class in the task. Reschedule-mode sensors are
supported.

Which Airflow version is it compatible with?
Apache Airflow 3.2 — including the Airflow 3.2 web UI, which Lite serves locally.

Do I need Docker?
No. leoflow lite --postgres managed uses an embedded Postgres and the subprocess
executor. (If Docker is present it can use it; if Docker is wedged, Lite falls back to
the managed Postgres automatically.)

Can it use my existing GCP/AWS/Azure credentials?
Yes — the local task inherits your environment and $HOME, so the provider SDKs find
your credentials through their normal default chains.

Is it production-ready / what about deploying?
Lite is for local development; the same DAGs deploy to Leoflow's Kubernetes mode (one
pod per task) for production. Same authoring, two runtimes.

Try it

curl -fsSL https://raw.githubusercontent.com/neochaotic/leoflow/main/install.sh | sh
leoflow lite --postgres managed

Open source, Apache 2.0: github.com/neochaotic/leoflow.
If you've been running Airflow under docker-compose just to develop a DAG, this is
the lighter loop.

How we parse Apache Airflow DAGs without importing Airflow

Alisson Rosa — Mon, 29 Jun 2026 22:16:11 +0000

TL;DR — Leoflow runs a Go control plane that never imports Apache Airflow,
yet compiles standard airflow.sdk DAGs. It does it with a structural shim: a
pure-stdlib stand-in for airflow that the parser puts on the import path, then
execs your dag.py to record the graph (without running task bodies or
installing a single provider). Arbitrary provider operators are captured by
class + kwargs at compile time and run for real in the task pod at runtime.
The parser drops from 262 MB / 136 packages to ~44 KB / zero dependencies.
This is the engineering behind Leoflow v0.1.0.

Where the shim sits

Before the trick, the shape of the whole system. The shim is one small box —
the parser, at compile time — and everything downstream of dag.json is Go,
with the real Airflow operator only ever appearing inside the task's pod:

Keep that picture in mind: the only place Python (and Airflow) lives is the
parser sidecar and the worker pod. The scheduling path in the middle is pure Go.

The constraint that forces the design

Leoflow's scheduler is Go — no GIL, no Python in the hot path (that's the whole
point: Airflow's Python control plane is what makes it slow). But a Leoflow DAG is a
standard Apache Airflow 3.2 DAG, written against airflow.sdk:

from airflow.sdk import DAG, task
from airflow.providers.standard.operators.bash import BashOperator

with DAG("etl", schedule="@daily"):
    pull = BashOperator(task_id="pull", bash_command="echo '[1,2,3]' > /tmp/raw.json")

    @task
    def transform() -> int:
        import json
        return len(json.load(open("/tmp/raw.json")))

    pull >> transform()

So: how does a control plane that never imports Airflow read a DAG written against
the Airflow SDK? Importing real Airflow into the parser would drag in the GIL, the
dependency tree, and parse-time side effects — exactly what we're escaping. The
answer (ADR 0024) is to not import Airflow at all.

The weight you don't carry

This isn't a micro-optimization. We measured the "just import the SDK" path —
pip install ./parser apache-airflow-task-sdk on Python 3.12:

Parser dependency	Size	Third-party packages
Real `apache-airflow-task-sdk` (→ pulls `apache-airflow-core`)	262 MB	136
Leoflow's structural shim	~44 KB	0

The 262 MB is grpc, babel, cryptography, sqlalchemy, libcst, pydantic,
opentelemetry, aiohttp… none of which a parser uses — it constructs DAG and
operator objects and reads a handful of attributes. And it can't be trimmed:
task-sdk → core, and providers-standard/http → apache-airflow (meta) → core.
Dropping Airflow makes the parser pure Python and small enough to embed in the Go
binary — no parser venv, no pip at install time, no Airflow-version coupling.

To be precise: this is the parser's weight, not the whole system's. The real
task SDK and a DAG's providers do get installed — in the task image, per DAG
(pip install at build time, or the Lite venv), because that's where the operator
actually runs them. Leoflow doesn't delete that weight; it moves it off the
scheduling hot path (the Go control plane never installs or imports Airflow) and
splits it per DAG (each image carries only its own providers — never one fat
shared worker for all 1,500). The parser is the part that gets to be ~44 KB.

The shim: a structural stand-in for `airflow`

The parser ships a pure-standard-library package that looks like airflow —
same import paths, same attribute surface the compiler reads — and nothing else.
It's put ahead of any real Airflow on the import path, and then the parser simply
exec's your dag.py:

import runpy
runpy.run_path("dag.py", run_name="__leoflow_dag__")  # `airflow` resolves to the shim

Running the file builds structure. Here's the core of the shim (paraphrased):

_CURRENT: list = []     # stack of DAGs being defined
COLLECTED: dict = {}    # dag_id -> DAG, filled as each DAG context is entered

class DAG:
    def __init__(self, dag_id, schedule=None, tags=None, **kw):
        self.dag_id, self.schedule, self.task_dict = dag_id, schedule, {}
        COLLECTED[dag_id] = self
    def __enter__(self):  _CURRENT.append(self); return self
    def __exit__(self, *e): _CURRENT.pop()

class BaseOperator:
    def __init__(self, task_id, **kwargs):
        self.upstream_task_ids, self.downstream_task_ids = set(), set()
        # attach to the active DAG and store every kwarg as an attribute
        dag = kwargs.get("dag") or (_CURRENT[-1] if _CURRENT else None)
        if dag: dag.task_dict[task_id] = self
    def __rshift__(self, other):    # a >> b records the edge
        self.downstream_task_ids.add(other.task_id)
        other.upstream_task_ids.add(self.task_id)
        return other

with DAG(...) registers; constructing an operator attaches it to the active DAG and
stores its kwargs; >> records edges; @task builds the node but never runs the
body. The compiler then reads COLLECTED — exactly the attributes it needs
(dag_id, tags, task_dict, and per task task_id, upstream_task_ids,
trigger_rule, python_callable, op_args/op_kwargs, bash_command,
endpoint/method) — and emits an immutable dag.json.

Two properties fall straight out of this:

Unsupported constructs can't be faked. A from airflow.<thing> the shim doesn't model raises ModuleNotFoundError, which the loader turns into a clear "not supported by Leoflow" error — at compile time, never a silent half-run.
Parsing has no side effects. @task bodies never execute during parsing, so a DAG file can't trigger its own work just by being read — the thing that makes Airflow's dag-parsing both slow and risky.

The control plane now has the graph without importing Airflow or installing one
provider.

The long tail: capture, don't reimplement

Modeling all 1,500+ provider operators in the shim would be a treadmill. So for
anything beyond the native handful (bash, python, http, empty), the shim has a
meta-path finder (ADR 0040) that synthesizes any
airflow.providers.<x>.{operators,sensors,transfers}.<Class> on demand. It doesn't
implement the operator — it captures it: records the operator's real dotted
class path and its constructor kwargs, then registers it like any node:

# in the dag.py — a provider operator the shim has never heard of
from airflow.providers.common.sql.operators.sql import SQLExecuteQueryOperator
SQLExecuteQueryOperator(task_id="rollup", conn_id="warehouse", sql="insert into ...")
# captured as: { class: "airflow.providers.common.sql.operators.sql.SQLExecuteQueryOperator",
#                kwargs: { conn_id: "warehouse", sql: "insert into ..." } }

No provider is installed in the parser. The dotted path and kwargs are just data in
dag.json.

What the shim won't do (on purpose)

The shim does all the structural parsing — but it draws three deliberate lines, and
each one is a feature, not a gap:

It never runs your task bodies. @task and operators only build structure at parse time; the code inside a task runs later, in the pod. Reading a DAG can't trigger its work.
It doesn't capture hooks. airflow.providers.*.hooks.* is intentionally not synthesized — a hook is a runtime client (it opens real connections), so it belongs inside a @task body that runs in the pod, never at parse time. Operators and sensors are captured; hooks are left to the runtime.
It rejects what it can't model, loudly. Anything outside the supported surface and the generic provider path — an unknown from airflow.<thing>, or a file with no dag_id / multiple DAGs — fails at compile time with a precise message, instead of being silently mis-parsed.

So "does the shim do everything?" — it does everything structural, and
deliberately hands execution (and hooks) to the runtime. That boundary is the
design.

The seam: the real operator runs in the pod

At runtime, inside the task's own pod — where the provider is installed, baked into
that DAG's image — the agent reconstructs and runs the genuine operator:

import_string(dotted_class)(**captured_kwargs).execute(context)

The real Airflow operator executes, with the real provider, against the real
connection — while the control plane that scheduled it never imported either. Here is
the whole life of a DAG, with the shim wired to every component it touches across the
three phases:

Read it left to right:

Compile time (no Airflow). The shim's _core handles the supported ops; _generic captures every provider operator as class + kwargs. In parallel, connectors: in leoflow.yaml pip installs the real providers into the DAG's image. Out comes dag.json (graph + task types + captured class/kwargs).
Schedule time (Go). The control plane consumes dag.json and dispatches a pod per task. It never parses a DAG — it reads an immutable artifact.
Run time (the pod). The agent reconstructs the operator from the captured class + kwargs (delivered in the task spec) and execute()s it — and the real provider is right there in the image.

Compile time: structure, dependency-free, in Go's world. Run time: the real Airflow
operator, in an isolated pod. That split is the entire design — it's how you get
Airflow's ecosystem fidelity without Airflow's control plane.

Fidelity: golden tests are the contract

A shim is only safe if it produces exactly what real Airflow would. So fidelity is
pinned by golden tests: for every shipped example, the shim's structural output is
asserted byte-equal to the real Airflow-based compiler's output. Drift is caught
in CI without installing Airflow (the golden corpus is regenerated from the real
compiler only when the supported surface changes). When we first built it, those
golden diffs caught two real fidelity gaps — duplicate task_id auto-suffixing and
list fan-in — which the shim now handles. There's also an escape hatch:
LEOFLOW_PARSER_BACKEND=airflow runs the real DagBag for side-by-side diffing.

Why it matters

No GIL, no Airflow imports in scheduling — the control plane stays fast and Go-native.
No dependency hell — each DAG owns its image; the parser needs zero providers, and 136 transitive packages leave your supply-chain surface.
No parse-time surprises — reading a DAG can't run it.
Full operator fidelity — the actual provider operator runs in the pod, guarded by golden tests.

It's all open source (Apache 2.0): github.com/neochaotic/leoflow.
ADR 0024 (the shim) and ADR 0040 (operator capture) have the gory details.

Leoflow v0.1.0 — run your Airflow DAGs on a Go control plane (no Airflow in the hot path)

Alisson Rosa — Sun, 28 Jun 2026 20:09:06 +0000

TL;DR — Leoflow v0.1.0 is the first stable release. You write standard
Apache Airflow 3.2 DAGs in Python; Leoflow's parser turns them into an immutable
dag.json using a structural shim that imports zero Airflow, the scheduler
(Go, no GIL) runs one pod per task, and the real Airflow provider operator
executes in the pod — so the control plane never imports Airflow, but your tasks
get full provider fidelity. Ships with 86 connection types, generic provider
operators + sensors (including reschedule-mode), and a Docker-free local Lite
edition that self-heals. GitHub: neochaotic/leoflow.

The bet

Airflow's execution model is right: a DAG of tasks, each task in its own pod
(KubernetesExecutor proved it). What's slow is the Python control plane — a
scheduler that imports your DAGs (and all their dependencies) into a GIL-bound
process, re-parses them constantly, and turns "add a provider" into a dependency-hell
negotiation across every DAG.

Leoflow keeps the model and rewrites the control plane in Go. No GIL. No Airflow
in the scheduling path. Each DAG is its own container image, so dependencies are
the DAG's problem, not the platform's. And the public API speaks Airflow 3.2, so
the real Airflow UI runs on top, unmodified.

The catch: if the control plane is Go and never imports Airflow, how does it read a
DAG written against the Airflow SDK? That's the shim — and it's the most interesting
piece of v0.1.0.

Here's the shape of it — your DAG becomes an immutable artifact, and a Go control
plane schedules it onto a pod per task:

You write real Airflow DAGs

No new DSL. This is a Leoflow DAG:

from airflow.sdk import DAG, task
from airflow.providers.standard.operators.bash import BashOperator

with DAG("sales", schedule="@daily"):
    pull = BashOperator(task_id="pull", bash_command="echo '[1, 2, 3]' > /tmp/raw.json")

    @task
    def transform() -> int:
        import json
        return len(json.load(open("/tmp/raw.json")))

    pull >> transform()

@task, >>, schedule, trigger rules, fan-in/fan-out, PythonOperator,
BashOperator — the constructs you already know. You compile it:

leoflow compile ./sales   # → dag.json + a container image

How the shim works (`dag.py → dag.json`, no Airflow imported)

Here's the trick (ADR 0024). The parser exec's your dag.py — but with a
structural stand-in for airflow on the import path. Pure standard library,
zero third-party deps. It reproduces exactly the attribute surface the compiler
reads, and nothing else:

Two consequences fall straight out of this design:

An unsupported construct can't be faked. A from airflow.providers.foo... that the shim doesn't model raises ModuleNotFoundError — which the loader turns into a clear "not supported by Leoflow" error at compile time, never a silent half-run. Loud beats subtle.
Task bodies never execute during parsing. @task calls only build the graph. Parsing a DAG can't trigger its side effects — the thing that makes Airflow's DAG-parsing both slow and dangerous.

So the control plane gets the graph without importing Airflow or installing a
single provider.

…but the real operator runs in the pod

Airflow's ecosystem is 1,500+ operators; modeling each in the shim would be a
treadmill. So Leoflow splits them (ADR 0040):

A native fast path for the hottest few — bash, python, http, empty — which Leoflow runs with its own Go/runtime code. No Airflow in the pod at all; this is the "no Python in the hot path" part. A deliberate, growing whitelist.
A generic path for the long tail. The shim's meta-path finder synthesizes any airflow.providers.<x>.{operators,sensors,transfers}.<Class> on demand and captures it — recording the operator's real dotted class path and its constructor kwargs, without the provider installed in the parser.

Then, at runtime, inside the task's own pod (where the provider is installed, via
the image), the agent does essentially:

import_string(dotted_class)(**captured_kwargs).execute(context)

The genuine Airflow operator runs, with the genuine provider, in an isolated pod —
while the control plane that scheduled it never imported either. Compile-time:
structure, dependency-free. Run-time: the real thing, in a pod. That seam is the
whole design.

Operators, sensors & 86 connectors

A provider operator is just an import. Anything outside the native fast path is
captured at compile time and runs for real in the pod — e.g. a SQL rollup against
your warehouse, its conn_id resolving to a managed connection:

from airflow.providers.common.sql.operators.sql import SQLExecuteQueryOperator
from airflow.sdk import DAG

with DAG("rollup", schedule="@daily", tags=["example"]):
    SQLExecuteQueryOperator(
        task_id="daily_rollup",
        conn_id="warehouse",   # a managed Connection (created in the UI)
        sql="insert into rollup select day, count(*) from events group by day",
    )

(BashOperator/PythonOperator/HttpOperator are the native path — Leoflow runs
those itself, no Airflow in the pod. Everything else takes the generic path above.)

Run it locally: in leoflow lite, add a warehouse Postgres connection
(Admin → Connections) plus events/rollup tables, then trigger the DAG — the
real SQLExecuteQueryOperator resolves the connection and writes the rollup. This
exact path is validated end-to-end.

Providers as a one-liner. A DAG declares what it needs; connectors: is sugar
(ADR 0038) that expands to the apache-airflow-providers-* packages and bakes them
into that DAG's image — no shared worker, no platform-wide dependency vote:

# leoflow.yaml
dag_id: sales
connectors: [postgres, http]      # → providers baked into THIS dag's image

86 connection types, generated from real Airflow (ADR 0039) so the connection
forms match field-for-field, are available in the UI. A managed connection is
delivered to the task pod as AIRFLOW_CONN_<ID> — the credential never lives in the
image:

@task
def load(rows: list[tuple]) -> None:
    import os, psycopg2
    dsn = os.environ["AIRFLOW_CONN_PG_TARGET"]   # a managed Connection, injected in-pod
    with psycopg2.connect(dsn) as conn:
        conn.cursor().executemany("INSERT INTO cats VALUES (%s, %s)", rows)

Sensors, including reschedule mode. A mode='reschedule' sensor releases its
pod between checks:

from airflow.providers.standard.sensors.date_time import DateTimeSensor

DateTimeSensor(task_id="wait_until_six", target_time="{{ ds }}T06:00:00+00:00",
               mode="reschedule")

A sensor waiting six hours isn't holding a pod for six hours: each not-ready poke
surfaces up_for_reschedule, frees the pod, and is re-dispatched when it's time to
check again.

Lite: zero to a local grid, Docker-free

leoflow lite is the local edition — no Kubernetes, no cloud:

leoflow lite --postgres managed     # embedded Postgres, no Docker required

It scaffolds a starter DAG, brings up an embedded Postgres, starts the control
plane, and serves the Airflow 3.2 UI at localhost:8088 — hot-reloading on every
save. v0.1.0 hardened it to be resilient:

Docker wedged? It keeps working. If the Docker daemon is present but unresponsive, Lite falls back to the managed (Docker-free) Postgres instead of failing on docker compose up.
It self-heals its state. Reusing a metadata DB used to leave "ghost" DAGs and stale import errors you couldn't remove from the UI. Now, on boot, Lite reconciles the registered DAGs against your workspace — deregistering what's gone and clearing orphan import errors — fail-safe (it never wipes on an unreachable control plane). It's gated by an end-to-end CI test so it can't silently regress.

Leoflow vs an Airflow control plane

	Airflow	Leoflow v0.1.0
Control plane	Python (GIL)	Go (no GIL)
DAG parsing	imports Airflow + your deps; bodies can run	structural shim, zero deps, bodies never run
Provider deps	shared, platform-wide	per-DAG image (`connectors:`)
Operator fidelity	real	real (runs in the pod via captured class+kwargs)
Task isolation	pod-per-task (K8s executor)	pod-per-task
DAG artifact	mutable in the dagbag	immutable `dag.json` + image
UI	Airflow UI	the same Airflow 3.2 UI
Local dev	needs the stack	`leoflow lite`, Docker-free

Why use it

You already write Airflow DAGs — keep them. The shim reads standard airflow.sdk; your operators run for real in the pod.
You're tired of dependency hell — each DAG owns its image; adding a provider to one DAG never touches another.
You want the control plane off the critical path — Go, no GIL, no DAG imports, no parse-time side effects.
You want the Airflow UI without the Airflow scheduler — v0.1.0 serves the real 3.2 UI on a Go core.
You want a real local loop — leoflow lite, no Kubernetes, that doesn't fall over when your Docker does.

Status

v0.1.0 is the first stable release (the v0.1.0-rc.N series soaked and promoted —
SemVer carries the maturity; no alpha/beta). It ships the shim, 86 connectors,
generic provider operators + sensors (reschedule included), the resilient Lite
edition, and the embedded Airflow 3.2 UI. dbt-native rendering is next (v0.1.1).

Try it in 30 seconds:

curl -fsSL https://raw.githubusercontent.com/neochaotic/leoflow/main/install.sh | sh
leoflow lite

→ github.com/neochaotic/leoflow — point
leoflow lite at a DAG and watch it light up the grid. Tell us where it bites.

Apache 2.0. Thanks for reading.

We rewrote Apache Airflow's control plane in Go (and kept the UI)

Alisson Rosa — Wed, 03 Jun 2026 18:23:45 +0000

TL;DR — Leoflow v0.0.1 just shipped. It speaks the Airflow API, runs the Airflow 3.2.x UI unmodified, but replaces the Python control plane with Go. Pod-per-task is the only execution mode. Each DAG is its own container image. Fan-in (map-reduce) is a Python list comprehension. Install: curl -fsSL https://raw.githubusercontent.com/neochaotic/leoflow/main/install.sh | sh. GitHub: neochaotic/leoflow — stars and issues warmly accepted.

The 3 AM pager

You know the one. The scheduler stalled again. Or the triggerer suffocated under 500 sensors. Or a worker leaked file descriptors until Kubernetes OOMKilled it mid-run. Or someone bumped pandas for the new DAG and broke six legacy ones because they all share the same image.

Apache Airflow is the most widely deployed workflow orchestrator on earth. It is also the one that bleeds the most in production. None of those wounds are bugs — they are structural consequences of running orchestration through a Python control plane. You cannot patch the GIL. You cannot make DagBag reparse cheap. You cannot make Celery workers ephemeral without rewriting them.

So we did the only thing left: we kept everything Airflow got right and replaced everything that bleeds.

What "kept" means

We did not invent a new model. Airflow's KubernetesExecutor proved years ago that pod-per-task is correct: each task gets its own container, its own resources, its own lifecycle. You can't leak a process that exits.

We also did not invent a new UI. The Airflow 3.2.x React SPA ships embedded inside the Leoflow server binary. Your team's muscle memory survives the migration.

What we kept:

The pod-per-task execution model
The Airflow 3.2.x UI (literally the same React build, served from /)
The HTTP API shape (/api/v2/dags/..., /api/v2/dagRuns/..., etc.)
The vocabulary: DAG, TaskInstance, DagRun, XCom, Trigger Rules
The DAG-authoring dialect — from airflow.sdk import DAG, task, TaskFlow, classic operators

What we threw out:

The Python scheduler. It's Go now.
The Python triggerer. Sensors are 2 KB goroutines.
The shared /dags folder. Each DAG is its own immutable container image.
The "long-lived Celery worker" model. Every task is an ephemeral pod.

What it looks like to write a DAG

Two files. That's the whole project.

# leoflow.yaml — your deploy concerns
dag_id: etl_sales
python_version: "3.11"
dependencies:
  - pandas==2.1.0
  - requests==2.31.0

# dag.py — your DAG, in real Airflow SDK 3.2.x
from airflow.sdk import DAG, task

@task
def fetch() -> list[dict]:
    import requests
    return requests.get("https://api.example.com/orders").json()

@task
def transform(orders: list[dict]) -> list[dict]:
    return [{"id": o["id"], "value": o["amount"] * 1.1} for o in orders]

with DAG("etl_sales", schedule="0 5 * * *", catchup=False):
    transform(fetch())

leoflow compile .              # generates Dockerfile, builds image, emits dag.json
leoflow push ./dag.json        # registers a new versioned DAG

No Dockerfile. No requirements.txt. No Helm values.yaml for this DAG. No pyproject.toml. The compiler reads leoflow.yaml, generates a Dockerfile against the official base image (leoflow/python-runtime:3.11), builds, pushes to your registry, and registers a versioned dag.json with the control plane. That's the whole inner loop.

For local development, leoflow lite provisions a managed Postgres, hot-reloads on every save, gives each DAG its own per-DAG virtualenv at ~/.leoflow/dev/venvs/<dag_id>/, and auto-detects uv on PATH for 5–10× faster cold installs. Two DAGs that pin conflicting versions of the same package coexist without interference. This is the bit that made me file the issue against Airflow for the first time, ten years ago. We finally have it.

Map-reduce, as a Python list comprehension

Hyperparameter search. K-fold cross-validation. Ensemble training. Monte Carlo. Every parallel ML workload is map-reduce. Most orchestrators make you build it: an operator per fan-out, a broker for the intermediate values, shared storage for the artifacts, a custom reducer that knows how to find them all.

Leoflow expresses the whole pattern in two lines of Python:

from airflow.sdk import DAG, task

@task
def trial(lr: float) -> dict:
    return train_one(lr)                            # map

@task
def select_best(trials: list[dict]) -> dict:
    return max(trials, key=lambda r: r["score"])    # reduce

with DAG("hparam_search", schedule=None):
    select_best([trial(lr) for lr in [0.001, 0.01, 0.05, 0.1, 0.5]])

That [trial(lr) for lr in …] is the whole map. trials: list[dict] is the whole reduce. No XCom plumbing, no broker, no shared filesystem, no special operator. The parser captures the list shape at compile time; the runtime assembles upstream XComs in declaration order and delivers them as a real Python list. Per-trial isolation (own pod, own process, own venv if you want). Per-trial retry. Deterministic ordering. A 256 KB cap per upstream value. A null slot for any upstream that legitimately produced no result.

If you have ever written a Celery chord by hand, take a moment.

Architecture

┌─────────────────────────────────────────────────────────────────┐
│                          Author / CI                             │
│  leoflow.yaml  +  dag.py  +  (auto-generated) Dockerfile         │
└───────────────────────────────┬─────────────────────────────────┘
                                │  leoflow compile / push
                                ▼
┌─────────────────────────────────────────────────────────────────┐
│                     Control plane — Go                           │
│ ┌───────────────────────────────────────────────────────────┐   │
│ │ HTTP API  /api/v2  ·  JWT · RBAC · multi-tenant           │   │
│ ├───────────────────────────────────────────────────────────┤   │
│ │ Scheduler   ·  state machine · cron · catchup             │   │
│ │             ·  PG-advisory-lock leader election           │   │
│ │             ·  retries with backoff                       │   │
│ ├───────────────────────────────────────────────────────────┤   │
│ │ Agent gRPC service  ·  task spec · state · XCom · logs     │   │
│ └───────────────────────────────────────────────────────────┘   │
│       │                                  │                       │
│       │ Postgres (metadata)              │ Redis (XCom + log)    │
└───────┼──────────────────────────────────┼──────────────────────┘
        │                                  │
        │     dispatch: one pod per task   │
        ▼                                  │
┌───────────────────────────────────────┐ │
│              Kubernetes               │ │
│  ┌─────────────────────────────────┐  │ │
│  │  Worker pod = your DAG image    │  │ │
│  │  leoflow-agent (15 MB Go bin)   │  │ │
│  │     ⇅ gRPC                      │  │ │
│  │  your Python / Bash code        │──┼─┘
│  └─────────────────────────────────┘  │
└───────────────────────────────────────┘

Short-lived http_api tasks skip the pod and run inline as goroutines (capped). Everything else runs pod-per-task, every time. Concurrency is goroutines and pods — no Celery, no triggerer process, no shared worker pool.

A few specifics worth calling out:

Leader election is a Postgres advisory lock. No external coordinator. No ZooKeeper, no etcd, no Raft library. It is the kind of decision you can explain to a new hire in 30 seconds.
XCom lives in Postgres on Lite (small, no Redis required for laptop dev) and Redis on Pro. 256 KB cap, optional schema validation, last-write-wins.
Connections are encrypted at rest with AES-256-GCM and delivered to tasks via Airflow's standard AIRFLOW_CONN_<ID> env var. Postgres / MySQL / SQLite / MSSQL / Redis / HTTP / GCS connectors ship with chain-of-custody-tested integration tests.
The agent is a static Go binary, ~15 MB. PID 1 of the task pod. Talks gRPC back to the control plane. Forks one process per task. Does not buffer Python output (-u plus PYTHONUNBUFFERED=1), because watching a SIGKILL race steal half of the user's print() output is its own kind of torment.

The numbers (the only honest part of any orchestration README)

We are not going to claim "1000× faster" because nobody who has run real pipelines believes you. Here is what falls out of replacing the control plane:

	Airflow today	Leoflow
Scheduler decision latency	3–10 s per task	<200 ms — native Go, no GIL
Sensor concurrency	~500 (asyncio Triggerer)	100,000+ — each sensor is a 2 KB goroutine
DAG parsing cost	Re-parsed every scheduler loop	Zero — `dag.json` is precompiled, immutable
Worker lifecycle	Long-lived, leak-prone	Ephemeral pod per task — spawn, run, die
Worker image size	1.5 GB+ Airflow base	~200 MB typical — each DAG is its own slim image
Dependency isolation	Workaround via `KubernetesPodOperator`	Native — every DAG is a container
Cold start	15–45 s	2–5 s target — agent is a 15 MB static binary
Observability	Retrofitted with effort	Native — Prometheus + OpenTelemetry + structured logs from commit one

These are the structural wins. The marketing-grade "X× faster" depends on your DAG. The scheduler latency drop is universal.

What it is not (because we have all read those launch posts)

It is not v1.0. Per ADR 0037, v0.0.1 ends the pre-alpha series; every release after is vX.Y.Z-rc.N → vX.Y.Z. The HTTP API, CLI surface, and Helm values may change between minor versions until v1.0.0 locks them.
The UI is still Airflow 3.2.x. It is a tactical choice (your team's muscle memory). A purpose-built Leoflow UI is on the roadmap (ADR 0018).
Pro is Kubernetes-only. Lite runs anywhere. Pro means a real cluster, external Postgres + Redis, the Helm chart. There is deliberately no Docker-Compose "Pro" path; we explained why in ADR 0015.
It is not a drop-in for every Airflow plugin. The Airflow operator catalog has 30+ years of accreted Python; we ship a closed set (python, bash, http_api) plus first-party connectors. ADR 0036 defines a runtime shim for from airflow.providers.<X>.hooks.<Y> import <Z>Hook so the common cases keep working — but if your DAG depends on three obscure providers we have not vendored, you will hit a wall today. File an issue; we are gating the next batch by demand.

Lite: the zero-deploy path

Here is the part that surprises people. To run Leoflow on your laptop you do not need a Kubernetes cluster. You do not need Docker. You do not need a container registry, a compile, a push, or a single line of deploy YAML. You need one shell command:

curl -fsSL https://raw.githubusercontent.com/neochaotic/leoflow/main/install.sh | sh
leoflow lite                # → http://localhost:8088 (LITE badge, top-center)

The installer is a single shell script: three static Go binaries into ~/.leoflow/bin, then leoflow setup provisions a managed CPython, the parser, and a managed local Postgres — nothing touches your system Python or your global packages. Then leoflow lite boots a full control plane (scheduler, API, UI) against that managed Postgres. No system services, no Compose file, no cluster. Close the terminal and it's gone.

There is no "dags/" folder — there is your folder

This trips up everyone coming from Airflow, so let's be explicit. Leoflow has no magic dags/ directory. During leoflow setup you pick a workspace folder (default ~/leoflow) — that folder is the runtime. Every subdirectory that contains a leoflow.yaml is a DAG project; the watcher scans them and hot-reloads on save. Your tree looks like what you'd actually keep in git:

~/leoflow/                     ← the workspace you chose at install
├── etl_sales/
│   ├── leoflow.yaml           ← makes this folder a DAG
│   └── dag.py
└── hparam_search/
    ├── leoflow.yaml
    └── dag.py

No central registry file, no dag_folder setting to fight, no "why isn't my DAG showing up." A folder with a leoflow.yaml is a DAG. That's the whole rule.

Edit DAGs from the browser — and get examples in one click

Lite ships a small embedded web editor so you can go from install to a running DAG without leaving the browser. Click the < > IDE button (bottom-right of the UI) and you get a real Monaco editor — the engine behind VS Code — scoped to your workspace:

Python + YAML syntax highlighting, a workspace file tree, open/save (⌘S), create/rename/delete with a "create target" chip that always tells you where a new file will land, collapse/expand carets that remember their state, and a recursive folder delete that says so out loud before it nukes a tree.
A "Download examples" button in the header. Click it and Leoflow materializes the bundled example DAGs straight into your workspace — fan-out/aggregate, Monte Carlo π, an HTTP-load DAG, a daily-sales ETL — so you have real, runnable DAGs in the UI in seconds instead of staring at an empty home screen.
Every save hits disk, the watcher picks it up, and the DAG hot-reloads. (One gotcha: the Airflow tab doesn't auto-refresh DAG structure — reload it.)

It is deliberately not a full IDE — no extensions, no terminal, no debugger. For those, point your own editor at the same workspace folder; it's just files on disk. The editor is a Lite convenience and is never registered in Pro.

Recover the admin password any time with leoflow lite reset-password. The Lite web-editor guide and the Lite cookbook cover the rest.

Pro: when you outgrow the laptop

For production, the Helm chart deploys against external Postgres 13+ and Redis 6+ (Cloud SQL / RDS / Memorystore / ElastiCache / Azure Cache all work):

kubectl create namespace leoflow
helm install lf oci://ghcr.io/neochaotic/leoflow -n leoflow \
  --version v0.0.1 \
  --set database.url='postgres://USER:PASS@HOST:5432/leoflow?sslmode=verify-full' \
  --set redis.url='rediss://HOST:6380/0' \
  --set auth.jwtSecret="$(openssl rand -base64 64)" \
  --set secretKey="$(openssl rand -hex 16)"

Read the chart docs for every value.

Engineering discipline (the stuff you would actually want to know)

If you are evaluating Leoflow as a load-bearing piece of your data platform, the answers are in the repo, but the short version:

Strict TDD. Every line of production code is preceded by a failing test (ADR 0011). Per-package coverage floors enforced in CI.
Go Report Card A+ as the quality floor (ADR 0012). gocyclo ≤ 15 per function. GoDocs on every exported identifier.
Supply chain from commit one. govulncheck, gosec, Trivy, CodeQL, gitleaks, OpenSSF Scorecard, OpenSSF Best Practices badge. Releases signed with cosign. SBOMs published per platform.
Every architectural decision is an ADR. 37 of them at the time of writing. They are the single best place to start if you want to understand the why.

The repo also runs an end-to-end install smoke against seven Linux distros (ubuntu:24.04, debian:12, fedora:41, alpine:3.20, opensuse/leap:15, archlinux:latest, rockylinux:9) on every release, plus a prealpha.N → v0.0.1 upgrade smoke. The v0.0.1 release got the green light because every one of those passed.

Help us ship the next milestone

We are at v0.0.1. The next steps are visible — pick any:

Star the repo at github.com/neochaotic/leoflow. It is the cheapest way to tell us this matters to you.
Open an issue with a chronic Airflow pain we have not closed yet. Pre-1.0 is the time to shape the API.
File a PR. Strict TDD applies. The CONTRIBUTING guide is the entry point. We review fast.
Try the Lite quick start (60 seconds, above). If something does not work, that is an issue worth filing.
Run the Helm chart against a real cluster and tell us what bit you. Pro alpha needs real-world miles before it gets a Pro v1.0.0.

What is on the post-`v0.0.1` table

Visible work toward v0.1.0:

Optimized backfill (parallel execution with throttling)
UI scaling for 10,000+ DAGs (caching, server-side pagination)
Dynamic task mapping
OIDC authentication (Google, Azure AD, Keycloak, Okta)
Deferrable tasks (ADR 0016) — efficient dispatch + long-poll pattern, native Go, no separate Triggerer process
A purpose-built Leoflow UI (ADR 0018)

We are deliberately keeping the surface small until it is boring and reliable. Workflow orchestrators have to be boring to be useful.

License & credits

Apache 2.0. We stand on the shoulders of Airflow — the team behind it defined the vocabulary, proved the architecture, and built the UI we reuse without modification today. We also studied Argo Workflows, Prefect, and Dagster carefully; each made decisions worth borrowing, and we did.

If you have ever waited five seconds for an Airflow task to start, you know why we built this.

→ github.com/neochaotic/leoflow — the v0.0.1 release notes are at /releases/tag/v0.0.1.

We rewrote Apache Airflow's control plane in Go (and kept the UI)

Alisson Rosa — Wed, 03 Jun 2026 16:22:27 +0000

TL;DR — Leoflow v0.0.1 just shipped. It speaks the Airflow API, runs the Airflow 3.2.x UI unmodified, but replaces the Python control plane with Go. Pod-per-task is the only execution mode. Each DAG is its own container image. Fan-in (map-reduce) is a Python list comprehension. Install: curl -fsSL https://raw.githubusercontent.com/neochaotic/leoflow/main/install.sh | sh. GitHub: neochaotic/leoflow — stars and issues warmly accepted.

The 3 AM pager

So we did the only thing left: we kept everything Airflow got right and replaced everything that bleeds.

What "kept" means

We also did not invent a new UI. The Airflow 3.2.x React SPA ships embedded inside the Leoflow server binary. Your team's muscle memory survives the migration.

What we kept:

The pod-per-task execution model
The Airflow 3.2.x UI (literally the same React build, served from /)
The HTTP API shape (/api/v2/dags/..., /api/v2/dagRuns/..., etc.)
The vocabulary: DAG, TaskInstance, DagRun, XCom, Trigger Rules
The DAG-authoring dialect — from airflow.sdk import DAG, task, TaskFlow, classic operators

What we threw out:

The Python scheduler. It's Go now.
The Python triggerer. Sensors are 2 KB goroutines.
The shared /dags folder. Each DAG is its own immutable container image.
The "long-lived Celery worker" model. Every task is an ephemeral pod.

What it looks like to write a DAG

Two files. That's the whole project.

# leoflow.yaml — your deploy concerns
dag_id: etl_sales
python_version: "3.11"
dependencies:
  - pandas==2.1.0
  - requests==2.31.0

# dag.py — your DAG, in real Airflow SDK 3.2.x
from airflow.sdk import DAG, task

@task
def fetch() -> list[dict]:
    import requests
    return requests.get("https://api.example.com/orders").json()

@task
def transform(orders: list[dict]) -> list[dict]:
    return [{"id": o["id"], "value": o["amount"] * 1.1} for o in orders]

with DAG("etl_sales", schedule="0 5 * * *", catchup=False):
    transform(fetch())

leoflow compile .              # generates Dockerfile, builds image, emits dag.json
leoflow push ./dag.json        # registers a new versioned DAG

Map-reduce, as a Python list comprehension

Leoflow expresses the whole pattern in two lines of Python:

from airflow.sdk import DAG, task

@task
def trial(lr: float) -> dict:
    return train_one(lr)                            # map

@task
def select_best(trials: list[dict]) -> dict:
    return max(trials, key=lambda r: r["score"])    # reduce

with DAG("hparam_search", schedule=None):
    select_best([trial(lr) for lr in [0.001, 0.01, 0.05, 0.1, 0.5]])

If you have ever written a Celery chord by hand, take a moment.

Architecture

┌─────────────────────────────────────────────────────────────────┐
│                          Author / CI                             │
│  leoflow.yaml  +  dag.py  +  (auto-generated) Dockerfile         │
└───────────────────────────────┬─────────────────────────────────┘
                                │  leoflow compile / push
                                ▼
┌─────────────────────────────────────────────────────────────────┐
│                     Control plane — Go                           │
│ ┌───────────────────────────────────────────────────────────┐   │
│ │ HTTP API  /api/v2  ·  JWT · RBAC · multi-tenant           │   │
│ ├───────────────────────────────────────────────────────────┤   │
│ │ Scheduler   ·  state machine · cron · catchup             │   │
│ │             ·  PG-advisory-lock leader election           │   │
│ │             ·  retries with backoff                       │   │
│ ├───────────────────────────────────────────────────────────┤   │
│ │ Agent gRPC service  ·  task spec · state · XCom · logs     │   │
│ └───────────────────────────────────────────────────────────┘   │
│       │                                  │                       │
│       │ Postgres (metadata)              │ Redis (XCom + log)    │
└───────┼──────────────────────────────────┼──────────────────────┘
        │                                  │
        │     dispatch: one pod per task   │
        ▼                                  │
┌───────────────────────────────────────┐ │
│              Kubernetes               │ │
│  ┌─────────────────────────────────┐  │ │
│  │  Worker pod = your DAG image    │  │ │
│  │  leoflow-agent (15 MB Go bin)   │  │ │
│  │     ⇅ gRPC                      │  │ │
│  │  your Python / Bash code        │──┼─┘
│  └─────────────────────────────────┘  │
└───────────────────────────────────────┘

A few specifics worth calling out:

Leader election is a Postgres advisory lock. No external coordinator. No ZooKeeper, no etcd, no Raft library. It is the kind of decision you can explain to a new hire in 30 seconds.
XCom lives in Postgres on Lite (small, no Redis required for laptop dev) and Redis on Pro. 256 KB cap, optional schema validation, last-write-wins.
Connections are encrypted at rest with AES-256-GCM and delivered to tasks via Airflow's standard AIRFLOW_CONN_<ID> env var. Postgres / MySQL / SQLite / MSSQL / Redis / HTTP / GCS connectors ship with chain-of-custody-tested integration tests.
The agent is a static Go binary, ~15 MB. PID 1 of the task pod. Talks gRPC back to the control plane. Forks one process per task. Does not buffer Python output (-u plus PYTHONUNBUFFERED=1), because watching a SIGKILL race steal half of the user's print() output is its own kind of torment.

The numbers (the only honest part of any orchestration README)

We are not going to claim "1000× faster" because nobody who has run real pipelines believes you. Here is what falls out of replacing the control plane:

	Airflow today	Leoflow
Scheduler decision latency	3–10 s per task	<200 ms — native Go, no GIL
Sensor concurrency	~500 (asyncio Triggerer)	100,000+ — each sensor is a 2 KB goroutine
DAG parsing cost	Re-parsed every scheduler loop	Zero — `dag.json` is precompiled, immutable
Worker lifecycle	Long-lived, leak-prone	Ephemeral pod per task — spawn, run, die
Worker image size	1.5 GB+ Airflow base	~200 MB typical — each DAG is its own slim image
Dependency isolation	Workaround via `KubernetesPodOperator`	Native — every DAG is a container
Cold start	15–45 s	2–5 s target — agent is a 15 MB static binary
Observability	Retrofitted with effort	Native — Prometheus + OpenTelemetry + structured logs from commit one

These are the structural wins. The marketing-grade "X× faster" depends on your DAG. The scheduler latency drop is universal.

What it is not (because we have all read those launch posts)

It is not v1.0. Per ADR 0037, v0.0.1 ends the pre-alpha series; every release after is vX.Y.Z-rc.N → vX.Y.Z. The HTTP API, CLI surface, and Helm values may change between minor versions until v1.0.0 locks them.
The UI is still Airflow 3.2.x. It is a tactical choice (your team's muscle memory). A purpose-built Leoflow UI is on the roadmap (ADR 0018).
Pro is Kubernetes-only. Lite runs anywhere. Pro means a real cluster, external Postgres + Redis, the Helm chart. There is deliberately no Docker-Compose "Pro" path; we explained why in ADR 0015.
It is not a drop-in for every Airflow plugin. The Airflow operator catalog has 30+ years of accreted Python; we ship a closed set (python, bash, http_api) plus first-party connectors. ADR 0036 defines a runtime shim for from airflow.providers.<X>.hooks.<Y> import <Z>Hook so the common cases keep working — but if your DAG depends on three obscure providers we have not vendored, you will hit a wall today. File an issue; we are gating the next batch by demand.

Lite: the zero-deploy path

curl -fsSL https://raw.githubusercontent.com/neochaotic/leoflow/main/install.sh | sh
leoflow lite                # → http://localhost:8088 (LITE badge, top-center)

There is no "dags/" folder — there is your folder

~/leoflow/                     ← the workspace you chose at install
├── etl_sales/
│   ├── leoflow.yaml           ← makes this folder a DAG
│   └── dag.py
└── hparam_search/
    ├── leoflow.yaml
    └── dag.py

No central registry file, no dag_folder setting to fight, no "why isn't my DAG showing up." A folder with a leoflow.yaml is a DAG. That's the whole rule.

Edit DAGs from the browser — and get examples in one click

Python + YAML syntax highlighting, a workspace file tree, open/save (⌘S), create/rename/delete with a "create target" chip that always tells you where a new file will land, collapse/expand carets that remember their state, and a recursive folder delete that says so out loud before it nukes a tree.
A "Download examples" button in the header. Click it and Leoflow materializes the bundled example DAGs straight into your workspace — fan-out/aggregate, Monte Carlo π, an HTTP-load DAG, a daily-sales ETL — so you have real, runnable DAGs in the UI in seconds instead of staring at an empty home screen.
Every save hits disk, the watcher picks it up, and the DAG hot-reloads. (One gotcha: the Airflow tab doesn't auto-refresh DAG structure — reload it.)

Recover the admin password any time with leoflow lite reset-password. The Lite web-editor guide and the Lite cookbook cover the rest.

Pro: when you outgrow the laptop

For production, the Helm chart deploys against external Postgres 13+ and Redis 6+ (Cloud SQL / RDS / Memorystore / ElastiCache / Azure Cache all work):

kubectl create namespace leoflow
helm install lf oci://ghcr.io/neochaotic/leoflow -n leoflow \
  --version v0.0.1 \
  --set database.url='postgres://USER:PASS@HOST:5432/leoflow?sslmode=verify-full' \
  --set redis.url='rediss://HOST:6380/0' \
  --set auth.jwtSecret="$(openssl rand -base64 64)" \
  --set secretKey="$(openssl rand -hex 16)"

Read the chart docs for every value.

Engineering discipline (the stuff you would actually want to know)

If you are evaluating Leoflow as a load-bearing piece of your data platform, the answers are in the repo, but the short version:

Strict TDD. Every line of production code is preceded by a failing test (ADR 0011). Per-package coverage floors enforced in CI.
Go Report Card A+ as the quality floor (ADR 0012). gocyclo ≤ 15 per function. GoDocs on every exported identifier.
Supply chain from commit one. govulncheck, gosec, Trivy, CodeQL, gitleaks, OpenSSF Scorecard, OpenSSF Best Practices badge. Releases signed with cosign. SBOMs published per platform.
Every architectural decision is an ADR. 37 of them at the time of writing. They are the single best place to start if you want to understand the why.

Help us ship the next milestone

We are at v0.0.1. The next steps are visible — pick any:

Star the repo at github.com/neochaotic/leoflow. It is the cheapest way to tell us this matters to you.
Open an issue with a chronic Airflow pain we have not closed yet. Pre-1.0 is the time to shape the API.
File a PR. Strict TDD applies. The CONTRIBUTING guide is the entry point. We review fast.
Try the Lite quick start (60 seconds, above). If something does not work, that is an issue worth filing.
Run the Helm chart against a real cluster and tell us what bit you. Pro alpha needs real-world miles before it gets a Pro v1.0.0.

What is on the post-`v0.0.1` table

Visible work toward v0.1.0:

Optimized backfill (parallel execution with throttling)
UI scaling for 10,000+ DAGs (caching, server-side pagination)
Dynamic task mapping
OIDC authentication (Google, Azure AD, Keycloak, Okta)
Deferrable tasks (ADR 0016) — efficient dispatch + long-poll pattern, native Go, no separate Triggerer process
A purpose-built Leoflow UI (ADR 0018)

We are deliberately keeping the surface small until it is boring and reliable. Workflow orchestrators have to be boring to be useful.

License & credits

If you have ever waited five seconds for an Airflow task to start, you know why we built this.

→ github.com/neochaotic/leoflow — the v0.0.1 release notes are at /releases/tag/v0.0.1.

DEV Community: Alisson Rosa

Leoflow Lite: a local development environment for Apache Airflow

What it is

What's supported

Your local cloud credentials just work

Leoflow Lite vs the usual local-Airflow setups

FAQ

Try it

How we parse Apache Airflow DAGs without importing Airflow

Where the shim sits

The constraint that forces the design

The weight you don't carry

The shim: a structural stand-in for airflow

The long tail: capture, don't reimplement

What the shim won't do (on purpose)

The seam: the real operator runs in the pod

Fidelity: golden tests are the contract

Why it matters

Leoflow v0.1.0 — run your Airflow DAGs on a Go control plane (no Airflow in the hot path)

The bet

You write real Airflow DAGs

How the shim works (dag.py → dag.json, no Airflow imported)

…but the real operator runs in the pod

Operators, sensors & 86 connectors

Lite: zero to a local grid, Docker-free

Leoflow vs an Airflow control plane

Why use it

Status

We rewrote Apache Airflow's control plane in Go (and kept the UI)

The 3 AM pager

What "kept" means

What it looks like to write a DAG

Map-reduce, as a Python list comprehension

Architecture

The numbers (the only honest part of any orchestration README)

What it is not (because we have all read those launch posts)

Lite: the zero-deploy path

There is no "dags/" folder — there is your folder

Edit DAGs from the browser — and get examples in one click

Pro: when you outgrow the laptop

Engineering discipline (the stuff you would actually want to know)

Help us ship the next milestone

What is on the post-v0.0.1 table

License & credits

We rewrote Apache Airflow's control plane in Go (and kept the UI)

The 3 AM pager

What "kept" means

What it looks like to write a DAG

Map-reduce, as a Python list comprehension

Architecture

The numbers (the only honest part of any orchestration README)

What it is not (because we have all read those launch posts)

Lite: the zero-deploy path

There is no "dags/" folder — there is your folder

Edit DAGs from the browser — and get examples in one click

Pro: when you outgrow the laptop

Engineering discipline (the stuff you would actually want to know)

Help us ship the next milestone

What is on the post-v0.0.1 table

License & credits

The shim: a structural stand-in for `airflow`

How the shim works (`dag.py → dag.json`, no Airflow imported)

What is on the post-`v0.0.1` table

What is on the post-`v0.0.1` table