DEV Community: Evgenii Timofeev

Self-Hosting Datanika with Docker Compose

Evgenii Timofeev — Sun, 20 Sep 2026 21:17:18 +0000

Most data platforms make self-hosting a footnote — a "contact sales for the on-prem option" link that goes nowhere, or an open-source core so stripped-down it's really just a demo for the paid cloud. Datanika's open-source core is the actual product: extract (dlt), transform (dbt-core), scheduling, a visual pipeline builder, every connector, multi-org RBAC, nine languages. It runs from one Docker Compose file, and self-hosting it costs $0 forever — the AGPL-3.0 core has no license key, no seat count, and no GB meter.

This is the walkthrough: what you get, the five-minute quick start, the configuration that actually matters, and — because this is the part most tutorials skip — an honest checklist for running it in production, where you own the pager.

Why self-host at all

The managed version at app.datanika.io exists because plenty of teams would rather not run infrastructure. Self-hosting is the right call when:

Data residency / compliance. Your customer data never leaves your VPC. No third-party processor, no data-processing addendum to negotiate.
Cost at volume. The managed plan meters bytes processed; self-hosted meters nothing. If you're moving terabytes a month, a $12 VPS beats any per-GB bill. (We did that math in The Real Cost of Your Modern Data Stack.)
No vendor lock-in. It's dlt + dbt-core under an open UI. If Datanika vanished tomorrow, your pipelines are standard dlt sources and dbt models — they keep running.
You already have infra. A spare box, a Kubernetes cluster, a managed Postgres — drop Datanika next to them.

The tradeoff is real and we'll be straight about it below: self-hosting means you own upgrades, backups, and the 3 AM page when a disk fills up. For a lot of teams that's a fair trade. For some it isn't.

Prerequisites

Docker Engine 24+ and Docker Compose v2 (docker compose, not the old docker-compose)
4 GB RAM minimum, 8 GB recommended
That's it. Postgres 16 and Redis 7 ship inside the Compose file — you don't install them separately unless you want to bring your own.

The five-minute quick start

git clone https://github.com/datanika-io/datanika-core.git
cd datanika-core
cp .env.example .env
# edit .env — at minimum set SECRET_KEY and ENCRYPTION_KEY (see below)
docker compose up -d

That's the whole thing. Compose pulls the images, starts four containers, and runs the database migrations on first boot. Give it a minute, then open:

http://localhost:3000 — the app (frontend)
http://localhost:8000 — the API

Create your account on the first-run screen and you're in. Your first pipeline — a source, a destination, a run — takes about five more minutes.

What's actually running

Four containers, and it's worth knowing what each one does before you put it in production:

Service	Image	Port	Job
`app`	datanika	3000, 8000	The Reflex app — frontend + API in one process
`celery`	datanika	—	Background worker: runs your extracts, loads, and dbt builds
`postgres`	postgres:16	5432	Application database (your orgs, connections, run history — not your warehouse)
`redis`	redis:7	6379	Task broker for Celery + cache

One thing that trips people up: postgres here is Datanika's own metadata database, not your data warehouse. Your extracted data lands wherever you point it — BigQuery, Snowflake, a separate Postgres, DuckDB on the same box. This container just holds Datanika's bookkeeping.

The configuration that actually matters

Most of .env has sane defaults. Two variables you must set to real random values before anything touches production:

# JWT signing key — anyone who has this can forge login tokens
SECRET_KEY=$(openssl rand -hex 32)

# Fernet key — encrypts every stored connector credential at rest
ENCRYPTION_KEY=$(python -c "from cryptography.fernet import Fernet; print(Fernet.generate_key().decode())")

ENCRYPTION_KEY is the one to guard: it's the Fernet key Datanika uses to encrypt every source and destination credential in the metadata DB. If you lose it, every stored credential becomes unrecoverable and you'll re-enter them all. Back it up somewhere that isn't the same box.

The rest, with their defaults:

Variable	What it's for	Default
`DATABASE_URL`	Metadata Postgres connection	`postgresql+asyncpg://datanika:datanika@postgres:5432/datanika`
`REDIS_URL`	Celery broker	`redis://redis:6379/0`
`SMTP_HOST` / `SMTP_PORT` / `SMTP_USER` / `SMTP_PASSWORD` / `EMAIL_FROM`	Email for run-failure alerts and invites	—
`RECAPTCHA_SITE_KEY` / `RECAPTCHA_SECRET_KEY`	Bot protection on signup (leave empty to disable)	—

Configure SMTP if you want failure notifications — a self-hosted pipeline that fails silently is worse than no pipeline. Everything else can wait.

Migrations and first boot

Migrations run automatically the first time the app container starts. If you ever need to run them by hand — say after pulling a new version — it's:

docker compose exec app alembic upgrade head

Taking it to production (the honest part)

docker compose up -d gets you a working instance. It does not get you a production-grade one. Here's the checklist we'd actually run through, in order of how much it'll hurt to skip:

Put a reverse proxy in front and terminate TLS. Nginx or Caddy (Caddy does automatic Let's Encrypt certs in about four lines). Bind the app containers to 127.0.0.1 and let the proxy be the only thing on :443. Never expose :3000/:8000 to the internet directly.
Back up the metadata Postgres. A nightly pg_dump is the difference between "restore in ten minutes" and "rebuild every connection by hand." Ship it off the box:

   docker compose exec -T postgres pg_dump -U datanika datanika | gzip > datanika-$(date +%F).sql.gz

Back up ENCRYPTION_KEY alongside it — a database dump full of credentials you can no longer decrypt is not a backup.

Lock down the network. Change the default Postgres/Redis passwords, restrict container ports to localhost, and put a firewall (ufw / security group) in front. Redis with no password on a public interface is a classic way to get owned.
Point monitoring at it. The Compose file ships optional Grafana + Prometheus profiles; wire them up or point your existing stack at the app's metrics. You want to know a pipeline is failing before your stakeholders tell you the dashboard is stale — the Slack-alerts setup is the fastest win here.
Right-size the box. 8 GB RAM / 4 vCPU is a comfortable floor for real workloads. Extract jobs are memory-hungry in bursts; Celery is where that shows up.

None of this is Datanika-specific — it's the standard "I now run a stateful service" checklist. But it's real work, and it's the honest cost of the $0 license.

Bring your own database

The bundled Postgres and Redis are convenient for getting started and fine for a small single-box deploy. For anything you care about, point Datanika at managed instances instead — set DATABASE_URL and REDIS_URL to your managed endpoints and the bundled containers become dead weight you can remove from the Compose file. Managed Postgres gets you backups, failover, and point-in-time recovery without you building any of it.

Upgrading

cd datanika-core
git pull origin master
docker compose up -d --build

The app container runs alembic upgrade head on startup, so migrations apply themselves. Pin to a tagged release rather than tracking master if you want change control — and take a pg_dump before every upgrade, because "roll back the database" is a lot easier than "figure out what the half-applied migration did."

Kubernetes, if that's your world

A minimal Helm chart ships in-tree at deploy/helm/datanika/ — same image, one app Deployment, one celery Deployment, optional ingress. A few sharp edges to know before you helm install: it needs a ReadWriteMany storage class (the app and celery pods share a dbt_projects volume), migrations run on every pod start (so keep app.replicaCount=1 until an HA migration hook lands), and you should disable the bundled single-replica Postgres/Redis in favor of managed ones. The self-hosting docs have the full values.yaml walkthrough.

Self-hosted vs. managed — the honest split

Everything in the product is in the open-source core. What you're not getting by self-hosting:

Billing / metering (Paddle integration) — irrelevant unless you're reselling
Managed infrastructure and automatic updates — you run git pull
Priority support with an SLA
SSO (SAML/OIDC) — gated to the Enterprise plan on managed cloud, though the SSO code itself lives in the open-source core

If none of those matter to you, self-hosting isn't a downgrade — it's the same platform on your terms. This blog, and the pipelines behind it, run on exactly this stack.

Where to go next

Self-hosting docs — the canonical reference: full env-var table, Helm values.yaml, production notes.
Getting Started — your first source → destination → run.
Architecture — how dlt, dbt-core, Celery, and Reflex fit together.
Browse all connectors — 31 sources and 9 destinations, all included, no plan gating.
The €12/mo stack, as it stood in April 2026 — the bill for running real software on one small VPS.

Clone it, run docker compose up -d, and you own your data pipeline stack end to end. Or if you'd rather we run it — the managed free tier is one click and includes 10 GB/month.

Four Identical Red Runs: One Was Our Watchdog Working, Three Were Its Corpse

Evgenii Timofeev — Tue, 15 Sep 2026 22:47:57 +0000

A cron in one of our repositories stopped firing on 21 June and nobody noticed for ten weeks. Its only job was to rebuild this site each morning so that blog posts whose publish date had arrived would actually appear. So for ten weeks, scheduled posts did not publish. The failure mode of a cron is silence, and silence is the one thing no dashboard renders.

We built a watchdog for it. The watchdog has four scheduled runs in its history and all four are red.

Here is what makes this worth writing down: the first red was the watchdog working exactly as designed, and the other three were it dying before it checked anything. From the outside they are indistinguishable — not merely the same colour, but the same run conclusion and the same step-by-step breakdown, line for line.

The four runs

33330840430  2026-08-30T19:25:44Z  failure
33442245077  2026-08-31T21:36:52Z  failure
33550257729  2026-09-01T19:34:08Z  failure
33673601655  2026-09-02T19:29:21Z  failure

Expand any of them and you get the same two lines that matter:

step 5  Check every scheduled workflow in both public repos  = success
step 6  File an issue when a schedule has stopped            = failure

That is run 1. It is also run 2, run 3 and run 4. We checked all four against the API rather than trusting the screen; they agree to the character.

On 30 August, step 6 was red because the watchdog had found a stopped cron and was filing an issue about it. The issue exists, machine-authored, timestamped 19:26:02Z, and it is correct in every particular:

datanika-landing :: .github/workflows/daily-rebuild.yml last ran on a schedule event at 2026-06-21T09:54:45+00:00 — 70.4 days ago. Its cron (0 6 * * *) should have fired within 38h. It is active, so this is NOT the 60-day disable; something else is stopping it.

On 31 August, 1 September and 2 September, step 6 was red because the watchdog had crashed in step 5 and there was nothing to report. It verified nothing on any of those nights.

What broke it, and it was not a commit

Nothing in our repository changed between 30 and 31 August. What changed was a repository setting: Dependabot became active on the repo. And when Dependabot is on, GitHub starts listing an extra workflow.

$ gh api repos/OWNER/REPO/actions/workflows --jq '.workflows[].path'
.github/workflows/ci.yml
.github/workflows/deploy-pointer.yml
...
dynamic/dependabot/update-graph

That last one has state: active and a display name of "Dependency Graph". It is not a file. We did not write it, it is not in the tree, and it is not in any branch:

$ gh api "repos/OWNER/REPO/contents/dynamic/dependabot/update-graph?ref=master"
{
  "message": "Not Found",
  "documentation_url": "https://docs.github.com/rest/repos/contents#get-repository-content",
  "status": "404"
}

GitHub synthesises it. The workflows API returns it; the contents API has never heard of it. It is not documented as an exception anywhere we could find, and it appears in both of our public repositories.

Our watchdog walks every workflow the API returns and reads each one's YAML off the default branch to extract its cron:

def _gh(*args: str) -> str:
    return subprocess.run(["gh", *args], check=True, capture_output=True, text=True).stdout

def _parse_crons(repo: str, path: str, ref: str) -> list[str]:
    raw = _gh("api", f"repos/{repo}/contents/{path}?ref={ref}", "--jq", ".content")

check=True turns the 404 into CalledProcessError. Nothing catches it. The script dies mid-collection — and because it collects the repos in order, it died on the first one and never reached the second, which is the repo the watchdog was built to watch.

If you have any tooling that enumerates workflows through the API and then reads their files, run this against your repos now:

gh api repos/OWNER/REPO/actions/workflows \
  --jq '.workflows[] | select(.path | startswith(".github/workflows/") | not) | .path'

Anything it prints will 404 on contents/. dynamic/pages/pages-build-deployment shows up the same way once GitHub Pages is enabled, so this is a family, not a one-off.

The fix is a whitelist, not a dynamic/ blacklist — GitHub only ever executes workflows out of .github/workflows/, so anything outside that path cannot be a workflow you own, and whatever prefix GitHub invents next year is handled without a code change.

It specifically must not become "swallow the 404". A 404 on a real .github/workflows/*.yml means a file you were asked to check is unreadable — a token scope, a rename, an API change — and that has to stay fatal. Turning it into "no crons found" would make the watchdog report health from an error.

The part that cost us three nights

The bug above is a fifteen-minute fix. The reason it survived three nights is a design problem, and it is the transferable half.

Look at the reporting step. Every path through it ends the same way:

if [ ! -f problems.md ]; then
  echo "::error::The watchdog exited non-zero but wrote no problems.md."
  echo "::error::That means it failed to RUN, not that it found a fault."
  exit 1
fi

if [ -n "$existing" ]; then
  gh issue comment "$existing" --body-file /tmp/comment.md
  exit 1
fi

gh issue create --title "Scheduled workflow stopped firing" --body-file /tmp/body.md
exit 1

Three outcomes — filed a new finding, added to an existing finding, could not run at all — collapsed into one signal. The run had to be red for the first two, because that is how a monitor gets your attention. So the third inherited the same colour, and the watchdog's own catastrophic failure was camouflaged by its success case.

We have written before about a green that proves nothing and about alerts that could not fire at all, and about the inverse case, a red that means "I found nothing". This is a fourth shape and it is the meanest of them, because the signal is not useless — it is genuinely informative, one night in four. It just does not carry which thing it means.

A monitor has three states, not two: ran and clean, ran and found something, did not run. If two of those share a colour, the pair that shares it is the pair you will conflate — and you will conflate it in the direction of "working", because that is the reading that requires no action.

The diagnostics were perfect and nobody read them

This is the detail that stings. Open the log of any of the three dead runs and the workflow tells you, in plain English, exactly what happened:

##[error]The watchdog exited non-zero but wrote no problems.md.
##[error]That means it failed to RUN, not that it found a fault.
##[error]Check the step above -- most likely the token cannot
##[error]read one of the repos, or the workflows API changed.

Whoever wrote that anticipated this precise failure and left a message that names the right layer and distinguishes the two cases the conclusion collapses. It was there all three nights. It went unread, because a red tick on a default branch that carries other standing reds is not a thing anyone clicks into.

Diagnostics one level below the signal are not diagnostics. They are an artifact you will find during the postmortem and feel bad about. If the message needs to be read, it has to travel through the same channel as the finding it disambiguates.

So that is what the fix does. The watchdog now files an issue about itself when it cannot run, under a deliberately different title, with a line stating what its own silence is now worth:

While this issue is open, the absence of a "Scheduled workflow stopped firing" issue means nothing: the detector is down, not the crons proven up.

Both paths still exit non-zero — the workflow needs that — but the two conditions are now one glance apart in the issue list instead of one log dive apart.

Twenty-six green tests, none of which touched the broken part

The watchdog had a test suite. Twenty-six tests, green throughout all three dead nights.

Every one of them exercised the comparator: given these crons and these last-run timestamps, which of them are overdue? That logic was never wrong. Not one test exercised collection — the loop that asks the API what workflows exist and fetches each one's file. Collection was the thin shell around gh api that the module's own docstring described as not the part worth testing.

There is a second, quieter version of the same mistake in there. The suite did check that collection found the expected number of workflows — but it asserted a total across both repos. Our first repo has enough workflows on its own to satisfy that total, so the count passed while the second repo was never reached. The fix asserts per repo, which is the assertion that would have failed.

Whenever you decide some part of a program is too thin to test, you have made a claim about where failure lives. Write it down as a claim, because it is one.

The last twist

Update, 15 September 2026. This section describes the morning of 3 September, when it was written, and it stopped being true that evening. The fix reached the default branch that day: the scheduled run that night completed cleanly, and so did the five after it. Since 9 September the watchdog has been red every night again, and not with the crash described here — it runs, and its most recent report names its own failed runs as the schedule that stopped. The original text follows as it was published.

schedule: triggers only ever run the copy of a workflow on the repository's default branch. Our fix is merged to the integration branch and has not been promoted. Tonight's scheduled run will crash again, in exactly the way described above, and there is no branch we could put the fix on that would change that.

Which means the fix cannot be proven by the mechanism it fixes. The only successful run in this watchdog's entire history is a workflow_dispatch we triggered by hand against the fix branch. That proves the code runs. It does not prove the schedule fires, and it does not prove the schedule fires this code.

Our own workflow header had already warned about this trap, from the last time it bit us:

Do not read a green run here as proof that it works. The thing under test is whether an unattended event=schedule run appears at all; a workflow_dispatch proves only that the dispatch step is sound.

The query that answers it honestly filters on the event:

gh api "repos/OWNER/REPO/actions/workflows/NAME.yml/runs?per_page=100" --paginate \
  --jq '.workflow_runs[] | select(.event=="schedule") | .created_at' | head -1

Run that against the cron this whole story started with, and the gap is unmissable: daily schedule runs from 16 April to 21 June, then nothing until 31 August. Seventy-one days. During that window a hand-triggered dispatch went green — which is precisely why a green dispatch was never evidence.

One honest footnote while we are counting. The remedy we applied for that cron was to move it off the top of the hour, on the theory that GitHub's queue is shortest away from :00. Three scheduled runs since: 4h49m, 5h17m and 7h48m late. Three samples is not a refutation, but it is not the improvement we told ourselves we were buying either, and we would rather print that than quietly drop it.

Four things to check in your own repositories

List every workflow whose path is not under .github/workflows/. If anything prints, every tool you have that reads workflow files by path is one API call from a crash.
For each monitor you run, ask what a red means. If "found a problem" and "could not run" share a conclusion, you own one signal doing two jobs, and it will fail in the direction that looks fine.
Give the clean path something to emit. A monitor that is only ever heard from when it has bad news is indistinguishable from one that has died — the useful assertion is on a heartbeat going stale, not on an alarm being absent.
Ask when each scheduled workflow last ran on an event=schedule, not when it last ran. Manual and dispatched runs will happily paper over a cron that has been dead for two months.

Then take the monitor you trust most and break it on purpose — not the system it watches, the monitor itself — and see whether anything looks different.

Datanika is an open-source data platform that runs dlt extract-and-load and dbt-core transformations behind one UI, with scheduling, run history and a REST API. Every workflow quoted here lives in our public repositories, defects included. It is AGPL-3.0 and self-hostable — see the architecture, or browse the connectors.

The Check That Went Red for Doing Its Job

Evgenii Timofeev — Tue, 15 Sep 2026 13:26:07 +0000

We have written twice recently about checks that could not fail — four alerts that could never fire and a nightly suite that passed for eight nights while twelve tests failed. Both are the same defect: a green signal that would have looked identical had the thing it watches been broken.

This one is the mirror image, and it is worse in a way that took us a while to articulate.

One run, three levels, three answers

Here is a single GitHub Actions run from one of our repositories, read three ways.

The run conclusion:

failure

The two jobs in it:

JOB parity        = success
JOB config-fields = failure

The steps inside the failing job:

1 Set up job                                    = success
2 Run actions/checkout@v6                       = success
3 Fetch core's shipped connection schema        = success
4 Compare documented fields against the form    = success
5 File an issue on drift                        = failure      <-- the only red

Every step that does real work succeeded. The one that failed was the step that reports. And here is its entire log:

Already tracked in #449; not filing again.
##[error]Process completed with exit code 1.

The check went red because it found nothing new to say.

The bug is four lines and looks completely reasonable

The workflow compares two catalogues that live in different repositories and files an issue when they disagree. Filing the same issue every morning would be useless, so it suppresses duplicates:

existing=$(gh issue list --state open --search "in:title Connector config field drift" \
  --json number --jq '.[0].number // empty')
if [ -n "$existing" ]; then
  echo "Already tracked in #$existing; not filing again."
  exit 1
fi

Read that in review and nothing jumps out. exit 1 after "we found drift" feels right — drift is a problem, and a red check is how a cron gets anyone's attention.

But this branch is not "we found drift." It is "we found drift, and it is the same drift as yesterday, already filed, already assigned." Nothing changed. Nothing needs doing. And because the issue will be open for as long as it takes to fix a 36-page documentation mismatch, this branch runs every day until then, painting the check red every time.

Why a permanent red is worse than a permanent green

This is the part worth taking away, and it is not symmetric with the "green that proves nothing" story.

A green that proves nothing fails to inform. A red that repeats forever destroys the channel.

A useless green leaves you no worse off than having no check. A permanent red actively trains every human near the repository to stop reading that signal — and the steps above it in the same job are the reason the cron exists. In our case those steps fetch a schema from another repository and parse a data file by regex. Both are exactly the kind of thing that breaks quietly when someone renames a field. The job even has explicit guards for it:

::error::Parsed $slugs slugs out of connectors.ts — the file shape changed.

That guard reports through the channel the duplicate-suppression branch was jamming. Had the file shape actually changed, the message would have arrived on a check that everyone had already learned to ignore, and it would have looked exactly like yesterday.

There is a second cost, which we hit within the hour. The red job sat next to a succeeding one and a stale auto-filed issue titled "Connector count drift." Glanced at, the repository said the connector count was broken. It was not — the counts agree, on both sides, in the workflow's own log, and on the live site. We nearly spent a session re-fixing something that was already fixed.

The line we drew

The distinction that resolves it is not "is there a problem" but "did anything change that needs a human?"

branch	exit	reasoning
filed a new issue	`1`	something changed today; a red run is a standing signal that is easy to see
an issue already exists	`0`	nothing changed, nothing to do, and the open issue is already the tracking mechanism

Note that we did not simply make the cron always green. The exit 1 after a genuine gh issue create stays exactly as it was. A rule that removes the check's ability to ever go red would be the original defect arriving from the other direction, so the test we added asserts the narrow version and asserts that both jobs still exit non-zero after really filing something.

Read the level that answers your question

The same run demonstrates a second thing, and it is free.

Three conclusions were available — run, job, step — and they were failure, success/failure, and one red step among five. Every one is a true statement about a different question. If you ask the run whether your drift check is healthy, the answer is a confident no, for a reason that has nothing to do with drift.

Fetch step-level data whenever the question is about a particular thing:

gh api repos/<owner>/<repo>/actions/runs/<id>/jobs \
  --jq '.jobs[] | {name} + {steps: [.steps[] | {number, name, conclusion}]}'

Two related traps we have hit: a job's own conclusion can be red because of an artifact upload rather than anything it tested, and a fetched log can silently omit steps entirely while the API reports them as success. Read outcomes from the API, and never conclude from an absence.

Three checks you can run on your own repositories

Grep your automation for exit 1 in a de-duplication or "already handled" branch. Anywhere a script's happy path for "nothing new" is an error exit. This includes retry guards, lockfile checks, and "skip because it is already deployed".
List the checks that have been red for more than a week. For each, ask what would happen if a different thing in that job broke tomorrow. If the answer is "the colour would not change", the job has no remaining signal.
For every scheduled workflow, ask what its steady state should be. A cron that watches for drift is supposed to be green almost always and red on the day something moves. If yours is red most days by design, the design is wrong.

One note on writing the test

We pinned the fix with a test that parses the workflow file, and the first run of its mutation harness reported:

M1 :: MUTATION DID NOT APPLY
M2 :: MUTATION DID NOT APPLY
M3 :: MUTATION DID NOT APPLY

The working copy of the workflow is CRLF and the harness's multi-line anchors used bare \n, so they matched nothing. That is worth stating because of what the alternative looks like: a mutation tool that silently changes nothing will report every mutation as "the test caught it", and you will believe you have a proven guard when you have an untested one. Make a harness assert that its mutation actually landed — we compare the file's hash before and after — and treat "did not apply" as a distinct outcome from "passed".

With the anchors corrected, all four mutations went red: both duplicate-suppression branches restored to exit 1, the scope control that deletes a legitimate exit 1, and a positive control that renames the log line the matcher keys on.

Datanika is an open-source data platform — extraction, loading, transformation and scheduling in one UI. The workflows described here are in our public repositories, defects included. Try it free, or read the self-hosting guide.

Four Alerts That Could Never Fire, and How We Found Them

Evgenii Timofeev — Tue, 15 Sep 2026 13:21:05 +0000

We spent a week auditing our own alerting, and the useful question turned out not to be "is anything alerting?" It was "would this alert look any different if the thing it watches had failed?"

For four of them the answer was no. Not "it fires late", not "the threshold is loose" — these could not produce a red signal at all, in the exact circumstance each was written for. Every dashboard was green the entire time, and the green was not evidence of anything.

Here they are with the mechanisms, because each one is a shape you can go and check for in your own stack in about ten minutes.

1. The rule that could only fire while the system was healthy

We wanted an alert for "a scheduled maintenance task has stopped arriving." The natural expression:

increase(celery_tasks_total{task="datanika.run_maintenance"}[2h]) < 1

Read it in English and it is obviously right: fire when fewer than one run happened in two hours. It evaluates cleanly. It shows up in the rule list. Its health reads ok.

It cannot fire.

In PromQL, a bare comparison between a vector and a scalar is a filter, not a boolean. X < 1 does not return true or false. It returns X's own value, for every series where the condition holds, and drops the rest. So the number that reaches the alerting threshold is not 1 and not true — it is whatever increase() computed, which by the definition of the filter is strictly less than 1.

Our rules pair that with a Grafana threshold of gt [0]: fire when the reduced value is greater than zero. Now do the arithmetic for the case the rule exists to catch. A task that has completely stopped arriving has increase(...) == 0. Zero passes the < 1 filter, so a series is produced — and then 0 > 0 is false, so nothing fires.

The rule is live only in the open interval (0, 1). It detects a task that has partially stopped and is structurally blind to one that has entirely stopped.

Demonstrated against the engine rather than argued:

vector(0) < 1        -> series=1 value=0     # gt [0] on 0 is FALSE -> dead rule
vector(0) < bool 1   -> series=1 value=1     # gt [0] on 1 is TRUE  -> fires

bool is the fix: it converts the filter into the 1/0 comparison everyone assumed they were writing.

The part worth stealing is how it was found. The engineer writing that rule used < bool 1 deliberately, and then mutated their own correct code to the naive < 1 to check whether the linter would have caught the mistake had they made it.

It did not. 196 tests passed.

Four other mutations run the same way — a dropped scrape job, a dropped deploy step, a drifted subquery step, a bare staleness comparison — all went correctly red, which is what proves the harness was armed and this one check simply does not look. A satisfiability check existed; it recognised the == N shape and skipped silently on everything else, and a skip is a pass.

2. The exporter that was scraped, and received nothing

Our task metrics were collected by code that had been in the repo since the start:

celery_tasks_total.labels(task=name, status="failure").inc()

There was an alert on that counter. It had never fired. There had also never been a task failure it should have caught, so nobody thought about it.

Two things were wrong, and each on its own is fatal.

The counter incremented in one process and was served from another. celery_tasks_total is incremented inside the Celery worker. /metrics is a route in the web process. Two processes, two prometheus_client registries, no shared state. The counter the worker maintains is not in the payload the web process serves, and never was.

And Prometheus was not scraping Celery at all. Seven targets configured, none of them the worker. So even a correctly located counter had no path to the time series database.

The alert's query returned zero series, forever. And here is the part that makes this a false-negative rather than a visible outage: our alert rules run with noDataState: OK, which is deliberate and correct for filtering expressions — a healthy system genuinely produces no rows. So "the metric does not exist" and "nothing is wrong" arrive at the alert engine as the same signal.

The general form, which we now have written down:

An exporter that is scraped but has stopped receiving events looks exactly like a quiet system.

The fix was not to move the counter. It was to stop asking the application to report on itself and read the broker's own event stream instead, with a dedicated exporter and the worker started with -E. Which introduced its own trap, so it goes in the list too: an exporter pointed at a worker without -E still emits worker-liveness metrics and zero task metrics — indistinguishable from a worker that simply has not run a task yet. You tell them apart by waiting for a scheduled firing and re-querying, never by reading the exporter's own health.

3. `curl -sf` succeeds on an HTML error page

This one is the cheapest to reproduce and probably the most widespread.

A verification runbook, gating a pricing change, contained this step:

curl -sf https://app.example.com/metrics | grep -E "bytes_processed|bytes_quota"

Under it, three checkboxes to tick when the metrics appear. Measured:

http_code=200   size=5106   content_type=text/html;charset=utf-8
<!DOCTYPE html><html lang="en">…

/metrics had no entry in the reverse-proxy config. Our proxy routes a specific list of paths to the backend and sends everything else to the single-page app, so an unrouted backend path does not 404 — it silently resolves to the frontend and returns the app shell with a cheerful 200.

curl -f fails on HTTP status codes at or above 400. A 200 carrying an HTML page is not a status error, so -f does not trigger and curl exits 0. The pipe then hands grep five kilobytes of HTML, grep matches nothing, and the operator sees no output and no error.

The checkboxes under that command could never be ticked, no matter what the application did. And the runbook's own troubleshooting table sent the reader to the wrong layer — "no metrics after 2h → check the worker logs" — for a problem that was one line of proxy config.

Two rules came out of it:

-f is a status check, not a content check. If you are grepping a response, assert on the Content-Type or on a string you know must be present, and fail the step when it is absent. curl -sf … | grep -q 'expected' at least fails on the pipe's own exit status.
Any backend route outside your proxied prefixes needs its own entry, or it becomes your SPA. The failure is silent, returns 200, and is invisible to anything that only checks status codes.

4. The pager that rang for something that had not happened

The other direction belongs in the same audit, because an instrument that manufactures incidents costs you the same credibility as one that hides them — and it burns it faster.

Our end-to-end suite has an auto-filer: on failure it opens a tracker issue and pages. It fires on a job-level failure().

One night it filed and paged for a failed end-to-end run. Reading the step outcomes from the API rather than the log:

step	conclusion
Run gating E2E specs	✅ success
Detect flaky gating specs	✅ success
Run informational E2E specs	✅ success
Assert the specs were actually collected	✅ success
Upload test report	🔴 failure
Telegram alert on failure	fired
File issue on failure	fired

The only error anywhere in the job:

Attempt 1..4 of 5 failed with error: Request timeout:
  /twirp/github.actions.results.api.v1.ArtifactService/CreateArtifact

Every spec passed. The job was red because an artifact upload timed out.

What makes this worse than a cosmetic false alarm is what the filer writes into the issue it creates: a sentence stating that only the gating tier can open this report. That sentence became affirmatively false at the moment it was filed — and it is exactly the line a reader uses to decide how much to trust what they are reading. The tracker asserted a test failure that had not occurred, in a thread whose entire purpose is to be believed.

if: failure() at job level means "any step failed". If you want "the tests failed", key the alert on the test step's own outcome:

- name: Run specs
  id: specs
  run: pytest ...

- name: Page on real failure
  if: steps.specs.outcome == 'failure'

The pattern under all four

Every one of these measured the instrument rather than the system:

what was green	what it actually recorded
the alert rule's health	that the expression parses and evaluates
the scrape target being `up`	that an HTTP endpoint answered, not that it carried our data
`curl -sf` exiting 0	that some server returned a status below 400
the E2E job's red	that some step in the job failed

None of them is broken in the sense of throwing an error. Each answers a real question accurately. The question is just not the one anybody thought was being asked.

So the check we now run on every new alarm is a single sentence:

Would this signal look different if the thing it watches had failed?

And the only honest way to answer it is to make the thing fail. Not in a synthetic fixture built to satisfy the assertion — against the real artifact, in the real failure mode. Break the rule and watch it go red. Stop the worker and watch the metric vanish, then check what your noDataState does with a vanished series. Point the curl at a path you know is unrouted and confirm the step fails.

A passing check is not evidence until you have seen it fail. We keep relearning that in a new costume roughly once a fortnight — most recently as a nightly CI job that reported success over twelve failing tests for eight nights, because a pipe to tee discarded the exit code and GitHub Actions' default shell does not set pipefail. Same family, different layer.

Four things to go and check in your own stack

Grep your alert expressions for a bare < or > against a scalar, and check what value actually reaches your threshold evaluator. In PromQL, add bool unless you specifically want the filter.
For every counter you alert on, confirm the process that increments it is the process that serves it. With any pre-fork or multi-process server, that is not automatic.
Find every curl -sf in your runbooks and CI, and make each one assert on content, not just status. Then point one at a deliberately wrong path and confirm it fails.
Read your alert conditions for scope. if: failure() and on: failure are usually broader than the thing you meant.

Then take one working alert and break the underlying system on purpose. If nothing turns red, you have found the fifth one.

Datanika is an open-source data platform that runs dlt extract-and-load and dbt-core transformations behind one UI, with scheduling, run history and a REST API. It is AGPL-3.0 and self-hostable with a single docker compose up — see the architecture, or browse the connectors.

Our Nightly CI Passed for Eight Nights While Twelve Tests Failed

Evgenii Timofeev — Wed, 09 Sep 2026 13:40:46 +0000

Our nightly connector smoke suite reported success eight nights running. Here is what it actually printed on each of those nights:

12 failed, 9 passed, 4 warnings in 44.01s

Twelve of twenty-one probes failing, every night, for at least eight consecutive nights — the log retention window is the only reason the count stops at eight. The step reported completed/success. The job reported success. The workflow reported success. The Telegram alert on failure step, guarded by if: failure(), was skipped every single night, because failure() never became true.

Nobody had broken anything that week. The suite had been lying since long before.

The mechanism is four words of shell

Here is the step, near enough verbatim:

- name: Run connector smoke tests
  run: |
    pytest tests/test_connector_smoke/ -v --tb=short -rs | tee /tmp/smoke.log
    if grep -qE '[0-9]+ skipped' /tmp/smoke.log; then
      echo "::error::Smoke probes were SKIPPED"
      exit 1
    fi

A shell pipeline's exit status is the exit status of its last command. The last command here is tee, and tee succeeded — it wrote the file it was asked to write. pytest exited non-zero into a void.

That much is ordinary POSIX shell, and most people who have written a CI pipeline know it. The part that catches you is the second half.

GitHub Actions gives you `pipefail` only if you ask for it, and asking looks like a no-op

Bash has a flag for exactly this. set -o pipefail makes a pipeline return the rightmost non-zero exit status, so pytest ... | tee fails when pytest fails.

GitHub Actions runs run: steps on Linux and macOS with a default shell of:

bash -e {0}

-e but no -o pipefail. Now write the step as:

- name: Run connector smoke tests
  shell: bash
  run: |
    ...

and the runner invokes:

bash --noprofile --norc -eo pipefail {0}

Naming the shell you were already using turns on pipefail. That is a real asymmetry in the product, it is documented, and it reads as a no-op in a diff. A reviewer looking at shell: bash on a step that was already running bash sees tidying, not a behavioural change — which cuts both ways: it is easy to add without argument, and easy for someone to delete later as noise.

We had no shell:, no defaults: and no pipefail anywhere in that workflow file. So the only thing left that could fail the step was the grep.

The part actually worth writing down

The grep is not a mistake. It exists for a good reason and it was working.

Our connector probes need live credentials. A probe with no credentials, or one whose client library will not import, used to skip — and a suite that skips everything passes loudly while testing nothing. So we changed conftest.py to convert both cases from skip into fail, on the reasoning that a healthy run should have zero skips, which makes "any skip at all" a usable alarm. Then we added the grep to enforce it.

Read those two decisions in isolation and both are right. Read them together and this falls out:

the outcome the guard watches for — skip — is now the rare one, by construction;
the outcome that actually happens — fail — is the one the pipe throws away.

The guard was aimed at the hole that existed before the conftest change. The conftest change moved every real failure into the blind spot. No commit introduced the bug; the second correct change walked the failure mode into the first correct change's shadow.

That is the shape to look for, and it is not rare. When you tighten one behaviour, the checks written against the old behaviour do not fail — they go quiet, and quiet and healthy look identical from outside.

What was underneath

The twelve failures were all real, and none was caused by this bug. A couple of trial accounts had lapsed and were returning 401 and 403. A message-broker probe was pointed at a cluster that no longer matched the one our credentials were minted against. Several probes were missing environment variables entirely, because the credential bundle handed to CI had drifted from the copy on disk — two copies of the same list, edited independently, which is its own recurring lesson.

Every one of those is a five-minute fix once you can see it. They had been invisible for over a week, and the instrument that was supposed to show them was reporting green the whole time.

Fixing it

Any one of these is sufficient. They are not equivalent.

1. Name the shell. The smallest diff, and it fixes every pipeline in the step at once:

- name: Run connector smoke tests
  shell: bash
  run: |
    pytest tests/test_connector_smoke/ -v --tb=short -rs | tee /tmp/smoke.log

2. Set the flag explicitly. More obvious to a reader, and it survives someone deleting shell: bash as redundant:

run: |
  set -o pipefail
  pytest ... | tee /tmp/smoke.log

3. Capture the status you care about. Verbose, but it is the only form that lets you act on pytest's code specifically — useful when a tool distinguishes "tests failed" (1) from "the run was misconfigured" (2, 4):

pytest ... | tee /tmp/smoke.log
rc=${PIPESTATUS[0]}

4. Stop piping. Write the log to a file and cat it afterwards, or emit machine-readable output and read that:

pytest ... --junitxml=/tmp/smoke.xml | tee /tmp/smoke.log

Keep the grep, in every case. It still catches the case it was written for. It just must not be the only thing that can fail the step.

Do not trust the green afterwards

This is the step that gets skipped, and it is the one that matters.

Once you have applied the fix, re-run the job unchanged, against the still-broken system, and confirm it goes red. If it goes green, your fix did not land — a check that has never been observed failing has never been shown able to fail. Our own rule, from a file of rules that each cost us an incident, is blunt about it: a passing check is not evidence until you have seen it fail.

We have paid for that rule more than once. A monthly database restore drill asserted that a table of seed rows survived the restore — while pg_dump writes tables alphabetically and puts users at the very end of the file, which makes it the first thing a truncation destroys and the last thing that check would notice. It printed PASS beside an empty user table. Alert rules that were structurally unable to fire. A test suite that mocked the very unit under test. In an earlier post we wrote up 2,300 passing tests sitting beside a CSV import that loaded exactly one row.

The through-line is one question, and it is worth asking of any signal before you rely on it:

Would this look different if the thing it watches had failed?

If you cannot answer that from the code, the green tells you nothing.

Audit your own workflows in one command

If you pipe test output anywhere in CI — to tee, grep, head, jq, a log shipper — check whether your exit code survives:

for f in .github/workflows/*.yml; do
  grep -q 'pipefail' "$f" && continue
  grep -q 'shell: bash' "$f" && continue
  grep -Hn -E '\|\s*(tee|grep|head|tail|jq|awk|sed|sort|uniq)' "$f"
done

Every line it prints is a pipeline whose left-hand exit code is being discarded. Some of those are deliberate. The ones that are not are the ones running your tests.

Two things worth knowing while you read the output:

set -e does not save you. -e aborts on a failing command; a pipeline whose last command succeeded has not failed, so there is nothing for -e to abort on. Having -e is what makes this feel safe when it is not.
Windows runners differ. The default shell there is PowerShell, which has its own rules for $LASTEXITCODE and native-command failure. A fix applied to a Linux matrix leg does not necessarily apply to a Windows one.

Alerting on the pipelines you actually run

The uncomfortable part of this story is not that a workflow was misconfigured. It is that we had an alert wired up, pointed at the right workflow, with the right condition — and it was silent for eight nights because the condition it tested was never reached. The alert was fine. The thing it was reading was wrong.

That generalises past CI. If you run scheduled data pipelines, the same question applies to every notification you have configured: does your failure alert read the pipeline's real outcome, or something downstream of a step that always succeeds?

Datanika records the outcome of every run in its own ledger rather than inferring it from a log line, and failure notifications fire off that record. If you want them in Slack, that setup takes about two minutes. If you drive pipelines from CI, the REST API hands back a run id and a status you can poll and assert on — which is a better thing to gate a deploy on than an exit code you have to hope survived a pipe.

The platform is open source and self-hostable. Start here, or read the docs.

Triggering Data Pipelines from CI/CD via the REST API

Evgenii Timofeev — Mon, 07 Sep 2026 12:44:43 +0000

The most common thing people want from a data platform's API is boring: run this pipeline, tell me if it worked. Usually right after a deploy, a dbt seed change, or a nightly job that has to finish before something else starts.

Here's how to do that with Datanika — and, more usefully, the one place this is easy to get wrong in a way your CI won't notice. We got it wrong ourselves; the fix is at the end of that section.

The short version

curl -sS --fail-with-body -X POST \
  "https://app.datanika.io/api/v1/pipelines/1/run?wait=true&timeout=300" \
  -H "Authorization: Bearer $DATANIKA_API_KEY"

One call. Blocks until the pipeline finishes, returns the run as JSON, and exits non-zero if the pipeline failed. In a set -e CI step, that is the whole integration.

The reason that last clause is worth a sentence is the rest of this section.

The status code carries the run's outcome

With ?wait=true, the endpoint polls the run until it reaches a terminal state, then answers with a status code that describes the run, not just the request:

What happened	Status
The run finished successfully	200
Still pending or running when your timeout expired	408
Terminal, but not successful (`failed`, `cancelled`)	422

The body is the serialized run in all three cases, so status and error_message are still there when you want detail:

{
  "id": 43,
  "target_type": "pipeline",
  "target_id": 1,
  "status": "failed",
  "started_at": "2026-08-30T14:00:02Z",
  "finished_at": "2026-08-30T14:00:44Z",
  "rows_loaded": 0,
  "error_message": "relation \"public.orders\" does not exist",
  "created_at": "2026-08-30T14:00:00Z"
}

That response is a 422. curl --fail trips on it, raise_for_status() raises on it, and your CI job goes red — which is what you wanted when you asked the API to wait.

The trap this replaced, because you will meet it elsewhere

Until August 2026, that same failed run came back as HTTP 200.

The reasoning was defensible and wrong. Your request succeeded — the server did its job, found the run, waited, serialized it, and handed it to you. The run is what failed, and the body says so plainly in status. Textbook HTTP.

The problem is that nothing in CI reads the body:

# The reflex every CI script has
curl --fail -X POST ".../pipelines/1/run?wait=true" -H "Authorization: Bearer $KEY"

--fail trips on 4xx and 5xx. A failed pipeline was a 200. Exit code 0. The job goes green, the dashboard downstream is stale, and nobody finds out until someone notices the numbers are yesterday's. A step that reports success on a failed load is worse than no step at all, because it converts a loud failure into a silent one.

What settled it was noticing the endpoint had already decided the question. It returned 408 when the run was still going at the timeout — and that isn't a transport failure either; the request was served perfectly. So 200 == failed wasn't a competing philosophy, it was an inconsistency with the endpoint's own behaviour. ?wait=true is the caller explicitly opting into "block until you know the outcome." If the outcome doesn't reach the status line, the option is half-built.

So if you are integrating some other pipeline API and it offers a "wait" mode: check what a failed run returns before you trust --fail. Trigger something you know is broken and look at the exit code. It is a two-minute experiment that this post exists because we ran late.

Why 422 and not 500

A 5xx means "our API broke, retry the request." Retrying this request would start a second pipeline run — a second extract, a second load, a second set of rows. A failed pipeline is not a transport failure and must not be retried like one.

422 says the opposite: the request was fine, and the thing you asked about did not succeed. The failure is in your pipeline — your credentials, your SQL, your source — not in our server. Retrying blindly is exactly the wrong move, and the status code should say so.

cancelled is a 422 as well. The check is "not success" rather than a list of failure names, so a terminal status added later cannot quietly rejoin the success branch.

Keep the body when you fail

Plain curl --fail discards the response body on an HTTP error, which throws away error_message — the one thing you want in the log.

Use --fail-with-body (curl 7.76+, so every current GitHub runner) to get the non-zero exit and the body. If you're on something older, capture the status code explicitly, as the full workflow below does.

The three endpoints

Runs are triggered on the resource, not on a runs collection. There is no POST /api/v1/runs.

What you're running	Endpoint	Key scope needed
A pipeline	`POST /api/v1/pipelines/{id}/run`	`pipelines:write`
A file upload	`POST /api/v1/uploads/{id}/run`	`uploads:write`
A dbt transformation	`POST /api/v1/transformations/{id}/run`	`transformations:write`

All three behave identically with respect to everything below.

Two modes: fire-and-forget, or wait

Without wait you get an immediate 202 Accepted:

{"run_id": 43, "status": "pending"}

Use this when CI's job is to kick off work — a nightly ingest that takes 40 minutes and nothing downstream is blocking on it. A 202 is about dispatch only; it says nothing about the outcome, and there is nothing to check with --fail.

With ?wait=true the request blocks until the run is terminal, then answers 200 / 408 / 422 as above.

timeout is in seconds, defaults to 120, and is clamped to 1–300. Passing timeout=3600 doesn't get you an hour — you get 300 seconds, then a 408. Status is polled every 2 seconds, and waiting doesn't occupy a worker, so a waiting request costs you nothing but the open connection.

A 408 is not a failure. The run is still going; you just stopped waiting. The body carries "timed_out": true, and the run's own status is still pending or running. That distinction matters because the remedy is different — a 422 means fix your pipeline, a 408 means wait longer or stop blocking on it. Treating them the same is how a slow Tuesday becomes a red build.

Runs longer than five minutes

Because of that 300-second ceiling, anything longer needs the async shape: trigger, then poll.

run_id=$(curl -sS --fail-with-body -X POST \
  "https://app.datanika.io/api/v1/pipelines/1/run" \
  -H "Authorization: Bearer $DATANIKA_API_KEY" | jq -r '.run_id')

deadline=$(( $(date +%s) + 3600 ))
while [ "$(date +%s)" -lt "$deadline" ]; do
  run=$(curl -sS "https://app.datanika.io/api/v1/runs/$run_id" \
    -H "Authorization: Bearer $DATANIKA_API_KEY")
  status=$(jq -r '.status' <<<"$run")
  case "$status" in
    success)             echo "done"; exit 0 ;;
    failed|cancelled)    jq -r '.error_message // .status' <<<"$run"; exit 1 ;;
  esac
  sleep 15
done

echo "Still running after 1h — check run $run_id in the app"
exit 1

GET /api/v1/runs/{id} is a plain read: it returns 200 with the run whatever its status, because here you are asking about the row rather than asking "did it work?". The .status field is the answer in this shape.

Note the case covers every terminal status, not just success. A loop that only watches for success runs until your CI timeout and then reports the wrong cause.

If you want the run's output while debugging, GET /api/v1/runs/{id}/logs returns {"run_id": …, "logs": "…"}.

Idempotent retries: the header that stops double-runs

CI reruns. Someone clicks "Re-run failed jobs", a runner gets evicted mid-step, a network blip makes curl retry. Any of those can fire the same trigger twice — and by default, twice means two runs, two loads, and two sets of rows.

Every POST endpoint accepts an optional Idempotency-Key header. Replay the same key and you get the original response back instead of a second run. Keys are cached for 24 hours, and it's opt-in — no header, no deduplication.

The natural key in GitHub Actions is the run identity itself:

curl -sS --fail-with-body -X POST \
  "https://app.datanika.io/api/v1/pipelines/1/run?wait=true&timeout=300" \
  -H "Authorization: Bearer $DATANIKA_API_KEY" \
  -H "Idempotency-Key: gha-${GITHUB_RUN_ID}-${GITHUB_RUN_ATTEMPT}"

Include GITHUB_RUN_ATTEMPT if a manual re-run should start a fresh pipeline, and leave it out if it shouldn't. That's a real decision, not a formality — decide it deliberately.

A complete GitHub Actions job

name: Refresh analytics

on:
  push:
    branches: [main]
    paths: ["dbt/**"]
  workflow_dispatch:

jobs:
  refresh:
    runs-on: ubuntu-latest
    steps:
      - name: Trigger the Datanika pipeline
        env:
          DATANIKA_API_KEY: ${{ secrets.DATANIKA_API_KEY }}
        run: |
          set -euo pipefail

          response=$(curl -sS -X POST \
            "https://app.datanika.io/api/v1/pipelines/1/run?wait=true&timeout=300" \
            -H "Authorization: Bearer $DATANIKA_API_KEY" \
            -H "Idempotency-Key: gha-${GITHUB_RUN_ID}-${GITHUB_RUN_ATTEMPT}" \
            -w "\n%{http_code}")

          code=$(tail -n1 <<<"$response")
          body=$(sed '$d' <<<"$response")

          case "$code" in
            200) echo "Loaded $(jq -r '.rows_loaded' <<<"$body") rows" ;;
            408) echo "::warning::Still running after 300s — run $(jq -r '.id' <<<"$body")"
                 exit 1 ;;
            422) echo "::error::Pipeline run $(jq -r '.status' <<<"$body") — $(jq -r '.error_message // "no message"' <<<"$body")"
                 exit 1 ;;
            *)   echo "::error::API returned $code"; echo "$body"; exit 1 ;;
          esac

Four branches, because there are four genuinely different situations: it worked, it's still going, your pipeline broke, or our API did. --fail-with-body collapses the middle two into one non-zero exit, which is fine when you only need pass/fail — but the messages above are what you'll want at 2am, and the 408 branch is the one people most often want to handle differently.

Cancelling a run

Update, 15 September 2026. One clause in this section is no longer true. A run cancelled while it is running now keeps the status cancelled when its task finishes; it is no longer overwritten back to success. Everything else here still holds: cancelling does not stop the worker, the extract and the load run to the end, and the run is billed for everything it processed. The original text follows as it was published.

If your workflow is cancelled, the pipeline it started is not. A 202 handed the work to a background worker, and CI walking away doesn't reach it.

And the API cannot currently clean it up for you — plan around that rather than around the endpoint. POST /api/v1/runs/{id}/cancel exists and will return 200 with "status": "cancelled" — but that call only writes the status onto the run row. It does not stop the worker. The extract keeps reading, the load keeps writing, and when the task finishes it overwrites the status back to success. Tracked as core#657, along with the reason there is no cancel button in the app either: shipping a control onto that behaviour would spread the wrong impression to every user instead of only to API callers.

So don't wire a cleanup step and believe it. Until cancellation actually stops work:

Bound the blast radius instead of the run. Smaller, more frequent pipelines beat one long job you might want to kill.
Assume anything you triggered will finish. Check the app rather than your CI log for what a cancelled workflow left behind.
On usage-based plans, a run you "cancelled" keeps metering until it completes on its own. That's the practical reason this is a limitation worth stating rather than a detail.

The one accurate part of the old advice: the endpoint returns 409 with "not_cancellable" when the run has already finished, which is a perfectly normal outcome and not something to treat as an error.

Give CI its own key, scoped down

API keys are created in Settings → API Keys in the app, carry the etf_ prefix, and are shown once. They're hashed with SHA-256 before storage, so a lost key can't be recovered — you create a new one and revoke the old. Full details on the API keys page.

For a CI key, set the scopes explicitly rather than leaving them empty (empty means full access):

pipelines:write — to trigger
runs:read — to poll status and read logs

That's a key that can start and observe one kind of work and cannot delete a connection, read your credentials, or create a schedule. If it leaks into a build log, the blast radius is a pipeline someone can already trigger from the UI.

Set an expiry on it too, and rotate it on a calendar rather than after an incident.

Rate limits

Each key is rate-limited independently, per plan, and exceeding it returns 429 with a Retry-After header. Current per-plan limits are in the API reference — a fan-out matrix build that triggers one pipeline per shard is the realistic way to hit them, so back off on 429 rather than retrying immediately.

The polling loop above is well inside every tier: at sleep 15 it costs 4 requests a minute, and ?wait=true polls server-side, so a 300-second wait is one request against your budget, not 150.

Note that 429 is a genuine transport-level "try again", unlike the 422 above. It's the one 4xx here you should retry.

Why this is easier here than in a three-tool stack

The reason this post is short is architectural. In a Fivetran + dbt Cloud + Airflow stack, "run the pipeline and tell me if it worked" is three APIs, three auth schemes, three status vocabularies, and a decision about which failure counts. Here, extract, load, and transform are the same run object with one status field, so CI asks one question once.

That argument, with numbers attached, is in Datanika vs the Modern Data Stack.

What this post does not cover

Scheduling. If you want a pipeline to run nightly, use a schedule instead of a cron-triggered CI job. Cron in CI gives you the worst of both — an extra dependency and no visibility in the app.
Webhooks back into CI. There is no outbound "run finished" callback into a workflow today; poll, or use a notification channel to tell a human.
Every endpoint. The API reference has the full surface, including connections, schedules, and bulk import.

Every status code, header, scope name and default here was read out of the live OpenAPI document and the shipped route handlers, not out of our own docs. That matters more than usual for this post: the 200-on-failure behaviour in "the trap this replaced" was **found while writing the first draft of it, filed as core#663, and fixed before this went out. Writing the tutorial is what surfaced the bug — so if you find a discrepancy, open an issue and we'll fix the post, or the API.

The Real Cost of Your Modern Data Stack in 2026

Evgenii Timofeev — Mon, 07 Sep 2026 09:18:12 +0000

Everyone has an opinion about the "modern data stack." Very few people publish the bill.

The standard pitch — Fivetran for extract, Snowflake for warehouse, dbt Cloud for transform, Looker for BI, Monte Carlo for observability — is sold as the default for any company with more than three engineers. In 2020 that bundle cost a small team maybe $1,500/mo. By 2024 it was closer to $3,000. In 2026, with Fivetran counting deletes as MAR and enforcing a $5 per-connection minimum, it's worse.

Meanwhile, something genuinely new happened in the last eighteen months: open-source ELT caught up to the commercial tools, and AI coding agents got good enough to own the "glue work" that used to justify half the price tag. The honest cost of running a growing team's data stack in 2026 isn't $3,000/mo. If you're willing to self-host and let an agent do the connector wiring, it's closer to $15.

This post is the math, not the pitch. If you disagree, the numbers are there to argue with.

The 2024 baseline — a growing team on managed everything

Let's pick a realistic "growing team" profile and price it out in 2024:

10 sources — a couple of production Postgres databases, Stripe, HubSpot, Segment, Shopify, Salesforce, Google Ads, a couple of internal APIs
10M rows/mo ingested across those sources
20 dbt models on top — staging, intermediate, marts
3 analytics engineers using the transform tool

Here's what that looked like on a "default" managed stack eighteen months ago:

Layer	Tool	Monthly cost
Extract + Load	Fivetran Standard	~$500 (10M MAR + $5/connection minimum × 10)
Warehouse	Snowflake (XS, ~40hr compute/mo)	~$120
Transform	dbt Cloud Starter	$300 (3 seats × $100)
BI	Looker / Metabase Cloud	$200–$400
Observability	Monte Carlo Entry / Elementary Cloud	$200–$500
Total		$1,320 – $1,820/mo

Where these numbers come from, because it is the weakest part of the post and worth saying plainly: they are our estimates for the profile above, not quotes. Checked in August 2026, most of these vendors publish no rate card you can compute a bill from — Fivetran points at an estimator and a consumption table with no per-MAR rate, Airbyte prices its paid tiers on compute capacity rather than data moved, and Snowflake's pricing-options page carries no dollar figure at all. dbt Cloud Starter at $100/seat/month is the one line here taken from a published price. Your own bill will differ, sometimes by a lot; run the numbers through Fivetran's estimator and Airbyte's pricing page for a binding figure. The Datanika rows below are exact, because our rate is published.

Nobody budgets for this up front. It creeps in one invoice at a time until someone in finance prints all five of them on the same page.

A few things to notice:

The "extract" layer is the most expensive single line. Fivetran's MAR-based pricing rewards narrow pipelines and punishes the long tail of small sources. Adding a sixth source you only sync once a day still costs $5 minimum — before any data moves.
Transform is priced per seat, not per workload. dbt Cloud charges you $100/month for the third analyst, whether they push one commit or a hundred.
Connector count is a lock-in metric, not a utility metric. Every vendor brags about "700+ connectors." Most teams use 8–12.

What changed between 2024 and 2026

Two things, mostly.

First: open-source ELT became legitimately production-ready. dlt (the Python library, not "data loading tool" the consulting company) went from "interesting library" to "the thing you'd actually pick" over about eighteen months. Its incremental-load primitives, schema evolution, and native typing are good enough that the main reason to pay for Fivetran — "we don't want to maintain extractors" — weakened considerably. Pair that with dbt-core (still free, still open source) and you have the whole EL-and-T loop in two Python packages.

The blocker used to be orchestration and the UI. You'd end up with a half-finished Airflow deployment, a YAML graveyard, and a Grafana dashboard nobody checks. That's the piece that actually burned out small data teams.

Second: coding agents. In 2024, writing a new source extractor with dlt meant reading the docs, writing a test, debugging pagination, and babysitting the first few runs. In 2026, you ask Claude or GPT to scaffold it, give it the API spec, and the first draft compiles. The "agent-shaped work" — wiring up credentials, mapping columns, writing the five lines of Jinja dbt needs — is the cheapest it has ever been. (We wrote about this shift in more depth in our AI-agent-native positioning post; the short version is that the API surface you expose to agents now matters more than the connector count.)

The implication is uncomfortable for the managed-ELT vendors: the thing you were paying Fivetran to avoid doing is now the thing an agent does in eight minutes for the cost of a few tokens.

The 2026 line-item budget for the same team

Same 10 sources, same 10M rows/mo, same 20 dbt models, same 3 analysts. Here's a stack built in 2026 with open-source ELT, a single VPS, and an AI agent doing the glue work:

Layer	Tool	Monthly cost
VPS (Hetzner CPX31 — 4 vCPU, 8 GB RAM, 160 GB NVMe)	Hetzner	€11.49 (~$12.50)
Snapshots / backups	Hetzner	€0.20
Extract + Load	`dlt` (open source)	$0
Warehouse	DuckDB / Postgres on the same box, or BigQuery on-demand (~$5–$20 depending on volume)	$0–$20
Transform	`dbt-core` (open source)	$0
Orchestration + UI	Datanika (open source)	$0
BI	Metabase OSS (same box)	$0
Observability	Grafana + Prometheus (same box)	$0
DNS / CDN / SSL	Cloudflare free tier	$0
Transactional email	Resend free tier	$0
AI agent (pipeline build + dbt model scaffolding, ~20 hours/mo)	Claude or similar	~$20–$40
Total		~$15 – $75/mo

The range comes from two variables: (a) whether you run the warehouse locally (DuckDB/Postgres, $0) or reach for BigQuery on-demand for heavier workloads, and (b) how much you let an agent drive. At the low end, with DuckDB and light agent use, you're paying €15/mo to run what used to cost $1,500. That's a 100× cost reduction for the same team profile.

If that sounds too good, it is and isn't. The cost of the tools dropped by 100×. The cost of operating the thing — someone who understands how pipelines fail at 3 AM — didn't drop at all. We'll come back to that.

Apples-to-apples: where each layer actually lands in 2026

Here's the same 10-source, 10M-row profile priced across everything a 2026 data team could actually pick:

Stack	EL + T + Orch	Warehouse	BI + Obs	Total / mo
Fivetran + dbt Cloud + Snowflake + Looker	$900	$120	$400	~$1,420
Airbyte Cloud + dbt Cloud + BigQuery + Metabase OSS	$550	$20	$0	~$570
Hevo + dbt Cloud + Snowflake + Metabase OSS	$540	$120	$0	~$660
Datanika Pro ($79) + BigQuery + Metabase OSS	$79	$20	$0	~$99
Self-hosted Datanika + DuckDB + Metabase OSS + agent	€12	$0	$0	~$15

Datanika's $79 Pro tier includes EL, T, scheduling, notifications, and multi-tenant dbt projects in a single app — which is why it lands where it does. The self-hosted tier is a straight VPS bill.

Before anyone writes me an angry email: yes, these are rounded. Yes, "it depends." Yes, the Snowflake bill for any team that actually writes meaningful SQL is going to be more than $120. But the order-of-magnitude gap between the top row and the bottom row is the real story and it's not a rounding error.

"But you're paying with your time"

Here is the honest objection. Managed stacks aren't expensive because the vendors are greedy. They're expensive because someone, somewhere, is being paid to keep the pipelines running at 3 AM when the Shopify API changes its pagination without telling anyone. That someone is a real cost, and when you self-host you inherit their job.

Two things have changed about that math:

The operational surface got smaller. A decade ago self-hosting "the data stack" meant running Airflow (a distributed scheduler you will fight), a warehouse you provision yourself, and a half-dozen custom extractors. In 2026 it means one Docker Compose file with six containers, running on a $12 box, with dlt handling the boring parts of extract and dbt-core handling transform. If the box falls over, you restore a snapshot. That's not zero work, but it's far from a full-time job.

Agents absorb the worst part. The failure mode that used to eat analytics engineers alive wasn't writing the pipeline — it was modifying it. Stripe adds a field. Shopify deprecates an endpoint. HubSpot changes its OAuth scopes. In 2024 this was half your calendar. In 2026 you open Claude, paste the error, and it hands you back the patch. The schema-drift tax didn't disappear, but its price collapsed.

So yes, you're "paying with your time" — but the bill is roughly one afternoon a month for a team of three, versus zero afternoons a month but $1,500. Plenty of teams should still pick managed. Plenty shouldn't. The default changed.

The three questions that actually decide it

If you're sitting on a 2024-era bill and wondering whether to rip it out, don't start with the tools. Start with these:

How many of your sources are in the long tail? If seven of your ten sources are small and only syncing once a day, Fivetran's $5-per-connection minimum is eating you alive. Self-hosting wins on long-tail pipelines and loses on high-volume ones. Look at your distribution of row volume, not the total.
Do you have anyone who can debug a Python stack trace? Not "love doing it" — just "can." The agent-assisted version of self-hosting is dramatically easier than the 2020 version, but there's still a floor. If nobody on the team can read a traceback, pay for managed and move on.
Is your warehouse bill the biggest line item? If yes, this post doesn't help you. Nothing in the EL/T layer moves the needle compared to rewriting a few nested CTEs. Go fix the warehouse first.

If the answers are "yes long-tail, yes Python, no warehouse isn't the biggest" — the 2026 stack at $15/mo is a real option, not a thought experiment.

Where Datanika lands in this

You can do everything in this post without us. dlt is on PyPI, dbt-core is on PyPI, DuckDB is on PyPI, Metabase ships a Docker image. Nothing about "open-source ELT + an agent" requires buying anything from anyone.

What Datanika is, honestly, is the UI and orchestration layer that keeps the six-container Docker Compose from turning into Airflow. We wrap dlt for extract, dbt-core for transform, APScheduler + Celery for scheduling, and add a multi-tenant Reflex UI so you can point-and-click the boring parts and let an agent drive the rest via our Agent API. The core is open source under AGPL-3.0; the billing layer is a paid plugin. If self-hosting scares you, Pro is $79/mo. If it doesn't, the core runs on the same €12 VPS class as everything else in this post.

We are not the cheapest thing on the table — dlt + dbt-core hand-rolled is. We are the cheapest thing with a UI and scheduling and notifications that an agent can talk to.

The punchline

The modern data stack in 2024 was an unbundling story: one tool per layer, each one priced for the Fortune 500, all of them assuming you had a dedicated data team. The 2026 version is a rebundling story. dlt + dbt-core + one VPS + one agent collapses five vendor invoices into a Hetzner bill with a euro sign on it.

None of this is theoretical. We publish our own infrastructure bill, line by line, including what changed when we moved hosts. Our Agent API exists because we use agents to build pipelines on Datanika, not just because other people might.

If the number on your current data-tools invoice has three commas in it and you haven't looked at this math in twelve months, look again. A lot changed while you weren't looking.

Datanika is an open-source data pipeline platform — dlt for extract, dbt-core for transform, APScheduler for orchestration, Reflex for the UI. Start free, self-host it, or read the Agent API docs if you want an agent to drive the whole thing.

dbt Incremental Models Duplicate Rows When `unique_key` Is NULL — and the Fix Is Off by Default

Evgenii Timofeev — Thu, 03 Sep 2026 12:29:01 +0000

You have an incremental model. It has a unique_key. You ran it twice on overlapping data, and the row count went up instead of staying flat:

{{ config(materialized='incremental', unique_key='order_id') }}

select order_id, customer_id, status, updated_at
from {{ source('shop', 'orders') }}
{% if is_incremental() %}
  where updated_at > (select max(updated_at) from {{ this }})
{% endif %}

Most of the table updates correctly. A small, stubborn subset duplicates on every single run, and it is always the same rows.

Check whether order_id is NULL in exactly those rows. It almost certainly is.

Why NULL keys never match

dbt's incremental merge strategy compiles to a MERGE statement whose join predicate compares the incoming batch to the existing table. In the default macro that predicate is, at heart:

merge into my_model as DBT_INTERNAL_DEST
    using my_model__dbt_tmp as DBT_INTERNAL_SOURCE
    on (DBT_INTERNAL_SOURCE.order_id = DBT_INTERNAL_DEST.order_id)

when matched then update set ...
when not matched then insert ...

Now recall what SQL does with NULL = NULL. It does not return TRUE. It does not return FALSE. It returns UNKNOWN, and MERGE treats anything that is not TRUE as "no match."

So for every row whose key is NULL:

when matched is never reached.
when not matched then insert fires.
A second copy is appended.

Run it again tomorrow and you get a third. The model is not broken in a way that errors — it is broken in a way that quietly accumulates. This is the same three-valued logic that makes where status != 'shipped' silently drop NULL statuses, arriving in a place where it costs you row counts instead of filter results.

Two consequences worth internalising:

unique_key is not a uniqueness constraint. dbt never enforces it. It is a join hint, and a join hint that is NULL matches nothing.
The bug is proportional to your NULL rate, not your data volume. A key that is NULL in 0.1% of rows produces a slow leak that no one notices until a dashboard's totals drift.

dbt fixed it — in 2025, behind a flag, off by default

The upstream issue is dbt-core#7597 / dbt-adapters#159, "[CT-2563] [Bug] Incremental updates result in duplicates if values for any unique_key are null". It was opened in May 2023, went stale twice, and was closed as resolved on 2026-03-03 by dbt-adapters#744, which merged on 2025-02-04.

If you find that thread while debugging, it reads as fixed. It is fixed only if you opt in.

Here is the shipped equals macro on dbt-adapters' main, in macros/utils/equals.sql:

{% macro default__equals(expr1, expr2) -%}
{%- if adapter.behavior.enable_truthy_nulls_equals_macro.no_warn %}
    case when (({{ expr1 }} = {{ expr2 }}) or ({{ expr1 }} is null and {{ expr2 }} is null))
        then 0
        else 1
    end = 0
{%- else -%}
    ({{ expr1 }} = {{ expr2 }})
{%- endif %}
{% endmacro %}

The NULL-aware comparison is real, and it is on the if branch. The else branch — the one you get by default — is still the plain = that started the problem.

The flag's registration in dbt/adapters/base/impl.py is worth reading in full:

{
    "name": "enable_truthy_nulls_equals_macro",
    "default": False,
    "docs_url": "",
},

"default": False, and an empty docs_url. There is no documentation page to link to, which is a fair part of why a fix that has been available since early 2025 is still an unfamiliar flag in 2026.

Turn it on in dbt_project.yml:

flags:
  enable_truthy_nulls_equals_macro: true

Then recompile a model and read the generated SQL in target/run/ before you trust it. If the predicate still says (DBT_INTERNAL_SOURCE.order_id = DBT_INTERNAL_DEST.order_id) with no is null branch, the flag has not taken effect — which is your signal that your installed dbt-adapters predates PR #744.

The flag does not cover composite keys

This is the part that will bite people who enable the flag and assume they are done.

A unique_key can be a single column or a list. Those two take different paths through default__get_merge_sql, and only one of them goes near the equals macro:

{% if unique_key is sequence and unique_key is not mapping and unique_key is not string %}
    {% for key in unique_key %}
        {% set this_key_match %}
            DBT_INTERNAL_SOURCE.{{ key }} = DBT_INTERNAL_DEST.{{ key }}
        {% endset %}
        {% do predicates.append(this_key_match) %}
    {% endfor %}
{% else %}
    {% set unique_key_match = get_merge_unique_key_match(source_unique_key, target_unique_key) %}
    {% do predicates.append(unique_key_match) %}
{% endif %}

The list branch writes a bare = for each column and never calls get_merge_unique_key_match, which is the only route to equals(). So in the default merge macro, unique_key: ['tenant_id', 'order_id'] stays NULL-unsafe whether or not you set the flag.

One honest caveat on that: this is the default macro from dbt's global project. Adapters are free to override get_merge_sql and get_merge_unique_key_match, so your warehouse's adapter may behave differently. Compile the model and read the emitted predicate — that is the only answer that is true for your stack.

What to do instead

Fix the data, not the comparison. A NULL in a key column is usually telling you something upstream is wrong. Before reaching for a workaround, find out why order_id is missing.

When the NULL is legitimate, give the merge something real to match on. A sentinel keeps the predicate on the fast path:

{{ config(
    materialized='incremental',
    unique_key='order_key'
) }}

select
    coalesce(order_id, -1)                      as order_key,
    ...

For composite keys, collapse them into one surrogate column so you are back on the scalar branch — and so that a single NULL component does not silently disable the whole match:

{{ config(materialized='incremental', unique_key='row_key') }}

select
    {{ dbt_utils.generate_surrogate_key(['tenant_id', 'order_id']) }} as row_key,
    ...

generate_surrogate_key coalesces NULLs to a string literal before hashing, which is exactly the property you want here. It also means one column to test rather than two to reason about.

Catch it before a dashboard does

The reason this bug survives so long in real projects is that nothing fails. Add the tests that turn silence into a red run:

models:
  - name: orders
    columns:
      - name: order_id
        tests:
          - not_null
          - unique

not_null on the key column is the one that matters — it fails at the source of the problem rather than at the symptom. unique is the backstop that catches duplication arriving by any other route.

If you use a composite key, test the surrogate:

      - name: row_key
        tests:
          - not_null
          - unique

A model whose unique_key has no not_null test is a model that has assumed something it never checked. That assumption is free until the day a source system starts emitting partial rows.

Where this fits

We run dbt-core in production, one project per tenant, so compiled SQL is something we read rather than something we take on faith. This bug is a good argument for a habit that generalises well beyond it: when a tool gives you a green run, ask what the green actually compared. dbt's incremental run succeeded. It emitted valid SQL. It inserted rows. Every layer reported success, and the model was still wrong — because nothing in the chain was checking the thing that broke.

If you would rather run dlt extraction, dbt transformation and scheduling in one place instead of gluing them together, that is what Datanika is. It is open source, self-hostable with docker compose up, and the docs start with a working pipeline rather than a concepts tour.

Further reading: How we run a dbt project per tenant · Building a Stripe revenue dashboard with dbt

Changelog: password reset and password change are live

Evgenii Timofeev — Wed, 02 Sep 2026 13:11:44 +0000

Until this release, Datanika had no way to reset a forgotten password and no way to change one. Not a hidden flow, not an admin-only flow — it did not exist. That is now fixed, and this is the changelog entry rather than a launch announcement, because a missing password reset is not a feature you get to celebrate shipping.

What shipped

/forgot-password — enter your email, receive a reset link.
/reset-password — set a new password from that link.
Change your password from Settings, with your current password as confirmation.

How the reset token works

The interesting part of a password reset is not the form; it's the token. Six decisions went into ours, and each one is there because the obvious alternative fails in a specific way.

The token is stored hashed, not signed. 32 bytes from secrets.token_urlsafe — about 256 bits — SHA-256'd at rest. At that entropy the hash needs no stretching, and a database dump yields hashes of dead capabilities rather than live ones.

It expires after 60 minutes. Long enough to walk away from your desk, short enough that a mail sitting in a synced archive stops being a key.

Validating a token never consumes it. This is the one that bites people. Corporate mail scanners and link-preview bots fetch every URL in an incoming message — so if the reset page burned the token on load, the scanner would burn it and the user's own click would always land on "this link is invalid." Loading the page validates; only submitting a new password consumes.

A rejected password doesn't burn the token either. Type a password that's too short and you get an error, not another round trip through your mailbox.

Requesting a new link invalidates the old one. Two live tokens for one account doubles the window in which an intercepted email is useful — and people request a second link precisely when they think the first went astray, which is when both are floating around.

Consumption is atomic. The token is claimed with a conditional UPDATE, so two submissions racing on the same link both pass validation and exactly one wins.

There's also no user-enumeration oracle: requesting a reset for an address with no account renders exactly what a real one renders. And requests are rate-limited three ways — per email address, per IP, and separately on the consume step.

Password rules

Minimum 8 characters. Maximum 72 bytes, which is bcrypt's limit, not a preference — so a passphrase in a non-Latin script hits the ceiling sooner than its character count suggests. No composition rules: no forced symbol, no forced digit, no forced capital. Length beats punctuation, and the rules that force a ! on the end mostly produce passwords ending in !.

Changing your password requires your current one. If your account was created through OAuth and has never had a password, you're setting one rather than changing it, and there is no current password to prove — the flow works that out from the account rather than asking you.

Self-hosting: this needs SMTP

Worth being explicit, because it will otherwise look like a bug. Password reset requires an outbound mail server. If SMTP_HOST isn't configured on your instance, /forgot-password shows an "unavailable" notice instead of the form.

That notice is deliberately instance-level rather than per-account — it tells you the instance can't send mail, not whether the address you typed exists. Configure SMTP and the flow lights up. Everything else here works the same self-hosted as it does on our cloud.

What this does not cover

This is one narrow thing, and it would be easy to read it as more than it is.

It's password recovery, not account management. You still cannot change the email address on an account, and there is no self-serve account or organization deletion. Both are real gaps, both are on the list, and neither ships in this release. If you need either, mail us and a human will do it.

This is not a "we're ready" post. We're pre-launch with no paying customers, and there is a list of things a mature product has that we don't. Password reset was near the top of it. It is now off the list. The list is not empty.

Why it took this long

Honestly: because we built the interesting parts first. Connectors, a metering system, an MCP surface, a benchmark — all more fun than a form that emails you a link. It took an audit that asked "what happens when a user forgets their password?" to notice the answer was "nothing, forever."

That's the recurring failure mode of building fast, and it's the same one we wrote about in the 109 security tests we ran before launch: the gap isn't in the code you wrote, it's in the code you never thought to write. A test suite can only fail on paths that exist.

Datanika is an open-source data pipeline platform — self-host it or start free. The account docs live under Organizations.

My SaaS Ran on €12 a Month (April 2026). Here's the Full Stack.

Evgenii Timofeev — Wed, 02 Sep 2026 13:06:43 +0000

Update — 3 September 2026. This is the stack as it stood in April 2026, and the hosting line has since changed. Production moved off Hetzner in July 2026 and now runs on a VPS with Pointer (pointer.gr), in Athens — still one box, still Docker Compose, still the EU. So every mention of Hetzner below is historical — the cost rows, the snapshot-retention row, the DNS table, and the paragraph about what happens if that data centre goes down: €11.49 is Hetzner's price, not what we pay today, and Nuremberg is not where your data is. What we pay today is €22.20 a month for the Pointer Gold VPS — the figure on the renewal invoice dated 6 August 2026, covering 15 August to 15 September 2026. Everything else — the container layout, the free tiers, the reasoning about when this stops working — still holds.

The current provider, country and backup arrangement are listed on our trust page, and that page is the one to believe if it ever disagrees with this one.

We have left the original figures in the table rather than quietly restating them. A cost post that edits its own numbers after the fact is not worth reading.

People assume you need hundreds of dollars in cloud bills to run a SaaS. Here's what Datanika actually costs to operate, line by line, with no hand-waving.

The Bill

Item	Monthly cost	What it does
Hetzner CPX31 (Nuremberg)	€11.49	4 vCPU, 8 GB RAM, 160 GB NVMe — runs the entire platform
Hetzner snapshot retention	€0.20	Weekly snapshots, 30-day retention
Aweb VPS (landing site)	€0 marginal	Sits on a shared server I already pay for
Cloudflare DNS + CDN + SSL	€0	Free tier covers everything
Resend (transactional email)	€0	Free tier (3,000 emails/month)
Paddle (billing)	€0	Pre-revenue. 5% + $0.50 per transaction once we're not
Operational total	€11.69	What hits my card every month
Porkbun domains (3 × ~€10/year)	€2.50 amortized	datanika.io, datanika.cloud, datanika.pro
All-in total	~€14.20	Including amortized one-time costs

The "12 euros" headline is the operational number — compute and bandwidth. Domains are a once-a-year purchase amortized into a monthly figure. I'm calling that out because most "cheap stack" posts conveniently leave it out, then someone notices in the comments and the post loses credibility.

What Runs on the €11.49 Box

The Hetzner CPX31 runs eight Docker containers via a single docker-compose.yml:

Container	Purpose
`datanika-app`	Reflex frontend + Starlette backend (ports 3000 / 8000)
`datanika-celery`	Background task worker for pipeline runs
`datanika-postgres`	PostgreSQL 16 — app metadata + pipeline data
`datanika-redis`	Celery broker + APScheduler job store + session cache
`datanika-grafana`	Internal-only monitoring dashboards
`datanika-prometheus`	Metrics scraper
`datanika-cadvisor`	Per-container resource metrics
`datanika-node-exporter`	Host-level metrics (CPU, disk, network)

That's roughly 2 GB resident, leaving 6 GB headroom for query workloads, build cache, and Celery task spikes. The box is dramatically over-provisioned for current usage — but the next size down is half the RAM and the savings would be ~€5/month, not worth the migration friction.

No managed Postgres. No managed Redis. No Kubernetes. No CDN paid tier. Just Docker Compose, nginx as reverse proxy, and Cloudflare in front.

The Landing Site Costs Nothing Extra

The marketing site at datanika.io sits on a separate VPS (Aweb) that I already pay for, hosting other unrelated projects. Adding the Datanika landing site to that box was zero marginal cost — nginx serves a static Astro build from /var/www/datanika.io/. If I had to spin up a dedicated VPS, it'd add maybe €5/mo. Currently it doesn't.

I'm being deliberate about saying "marginal cost" instead of "free". Free implies no underlying cost exists. Marginal cost reflects the fact that adding one more static site to an already-running nginx is genuinely zero new spending — but the box itself is real.

DNS and SSL: Cloudflare Free Tier

Three domains (datanika.io, datanika.cloud, datanika.pro) on Porkbun with Cloudflare DNS. Redirect rules handle the routing between them:

datanika.io → landing site on Aweb
app.datanika.io → SaaS app on Hetzner (Pointer, in Athens, since July 2026 — see the update at the top)
datanika.cloud → 301 redirects to app.datanika.io (and legal pages redirect to datanika.io)

SSL via Cloudflare Origin Certificates on both servers. Full strict mode. Zero cost, zero maintenance.

Why This Works (And When It Will Stop)

The trick is not overengineering early. No Kubernetes orchestration, no managed database, no Redis cluster, no Datadog, no PagerDuty. One VPS, Docker Compose, and a monitoring stack that runs on the same box it's monitoring.

Yes, that's a single point of failure. Yes, if Hetzner Nuremberg goes down, so does the SaaS. The probability is low and the alternative — multi-region active-active — would 5x the cost for an app with zero paying customers.

When the bottleneck becomes infrastructure rather than users, here's what gets added (in order):

Managed Postgres (~€15/mo) — when point-in-time recovery becomes a real need
Separate Celery worker (~€11/mo) — when one box can't run pipelines and serve UI on the same cores
Read replica (~€11/mo) — when SQL Editor queries start blocking app traffic
Cloudflare paid tier (~€20/mo) — only if image optimization or edge caching becomes load-bearing

That's ~€50/mo of growth headroom, all deferred until usage justifies it. Right now the bottleneck is users, not infrastructure. Every euro saved on hosting is one more month of runway.

What's Not on the Bill

A few things I get for free that would otherwise cost money:

GitHub — free for public repos. The open-source core lives there
GitHub Actions — free CI minutes for public repos
Error tracking — none. Errors go to docker logs and Grafana. Will add Sentry's free tier when I outgrow tail-the-logs debugging
Staging server — none. The deploy pipeline runs build + tests in CI before touching production

If I were spending €200/month with no paying users, every one of those services would feel "necessary". At €12/mo, the discipline of "buy nothing until it hurts" stays sharp.

The Self-Hosting Story

Because the infrastructure is this simple, self-hosting is genuinely straightforward. The same docker-compose.yml that runs production runs your self-hosted instance:

git clone https://github.com/datanika-io/datanika-core.git
cd datanika-core
cp .env.example .env
docker compose up -d

That's it. App on port 3000, no Kubernetes, no cloud provider lock-in. The self-hosting guide covers configuration, backups, and upgrades.

Once the containers are up, the fastest way to verify everything works is the CSV to DuckDB template — zero credentials, zero cloud accounts, first rows loaded in two minutes. No egress fees to trip over, no service-account JSON to provision.

The open-core architecture means self-hosted gets all features — the billing plugin simply doesn't load. No crippled "community edition".

If You're Launching

If you're launching a SaaS and spending more than €20/mo on infrastructure before you have paying users, ask yourself why. Most "what stack should I use" discussions get answered with whatever's trending on Hacker News. The boring answer — one VPS, one database, Docker Compose, Cloudflare — works for an enormous range of indie SaaS scale. It'll get you to your first 100 paying customers without rearchitecting anything.

Building in public. What does your SaaS infrastructure cost you each month? Open a discussion on GitHub — I'm collecting cost breakdowns.

How I Split Open-Source Core From Paid Cloud — the architecture that lets self-hosted users run the full platform for free
Self-Hosting Guide — Docker Compose setup, environment variables, and upgrade flow
Pricing — what the cloud edition adds on top (Free / Pro / Enterprise)
I Built an ETL Platform Solo — how scope control kept the infrastructure this simple

Introducing Datanika: Open-Source Data Pipelines for Everyone

Evgenii Timofeev — Wed, 02 Sep 2026 13:01:41 +0000

Why We Built Datanika

Building data pipelines shouldn't require stitching together five different tools. Most teams today need a data loader (Fivetran, Airbyte), a transformation tool (dbt Cloud), a scheduler (Airflow, Dagster), and a monitoring solution — each with its own learning curve, billing, and maintenance burden.

Datanika takes a different approach: one platform for the entire pipeline.

What Datanika Does

Extract & Load — Connect to 30 sources (databases, APIs, SaaS tools, files) and load data into your warehouse using dlt.
Transform — Write SQL models powered by dbt-core, with a built-in editor, tests, and snapshots.
Orchestrate — Build visual pipelines with a DAG editor, set up schedules with cron expressions, and define dependencies.
Monitor — Track every run with streaming logs, error details, and execution history.

Open Source First

Datanika's core is open source under AGPL-3.0. You can self-host it with Docker Compose or use our managed platform at app.datanika.io.

Get Started

Getting Started Guide — first pipeline in 5 minutes
35 Connectors — databases, SaaS APIs, files, and streaming
Pricing — Free / Pro / Enterprise tiers
Architecture Overview — how the pieces fit together

Datanika is AI-Agent Native: Build Data Pipelines with Autonomous Agents

Evgenii Timofeev — Mon, 31 Aug 2026 19:01:35 +0000

The Problem: AI Agents Can't Use Your Data Pipeline Tool

AI agents are getting good at writing code, answering questions, and managing workflows. But try pointing one at Fivetran, Airbyte, or Stitch and asking it to build a data pipeline. It won't get far.

These platforms weren't designed for machines. Their APIs lack discovery endpoints, config schemas are undocumented blobs, and there's no way for an agent to validate its work before executing it. The agent guesses, fails, and loops.

We decided to fix this. (If you missed it, we shipped the REST API v1 last week — the agent stack builds on top of it.)

What We Shipped: 5 Tiers of Agent Compatibility

Starting today, Datanika's API is fully agent-native. We built five tiers of capability that give an autonomous agent everything it needs to build a complete data pipeline from scratch — no human intervention required.

Tier 1 — Discover & Introspect

An agent needs to know what's available before it can build anything. Our meta endpoints return full JSON Schema for every connection type, dlt config option, dbt test type, and materialization strategy. The agent doesn't guess — it reads the schema and builds valid configs.

GET /api/v1/meta/connection-types     → 32 types with config schemas
GET /api/v1/meta/dlt-config-schema    → full upload config schema
GET /api/v1/meta/dbt-tests            → test types with parameter schemas
POST /api/v1/connections/{id}/introspect → list source tables
POST /api/v1/connections/{id}/columns    → column types and metadata

Tier 2 — Build

Standard CRUD for every resource: connections, uploads, pipelines, transformations, schedules, and notification channels. The agent builds the pipeline piece by piece using the schemas it discovered in Tier 1.

Tier 3 — Validate Before Executing

This is where most competitor APIs fall short. Datanika lets the agent compile a dbt transformation (catching Jinja and ref errors) and preview the output — all without touching production data. Typed error codes (compilation_error, execution_error, unsafe_sql) let the agent branch on failures instead of regex-matching error messages.

POST /api/v1/transformations/{id}/compile   → validate SQL + Jinja
POST /api/v1/transformations/{id}/preview   → sample output rows

Tier 4 — Execute with Control

Update, 15 September 2026. "Cancel stuck runs" below says more than the endpoint does. POST /api/v1/runs/{id}/cancel marks a run cancelled, but it does not stop the work: the run continues to the end and is billed for everything it processed. If a run looks stuck, read GET /api/v1/runs/{id} and GET /api/v1/runs/{id}/logs instead of cancelling it and triggering it again, because a second trigger starts the same job beside the first, and an upload that appends would load its rows twice. The original text follows as it was published.

Trigger runs and wait for completion in a single request (?wait=true). Cancel stuck runs. Retry safely with Idempotency-Key headers — same key within 24 hours returns the cached response instead of creating duplicates.

POST /api/v1/uploads/{id}/run?wait=true     → run and wait
POST /api/v1/runs/{id}/cancel               → cancel if stuck

Tier 5 — Machine-Readable Discovery

Two unauthenticated documents that tell any LLM how to use the platform:

/llms.txt — plain-text discovery file at the site root. Contains the API summary, base URL, auth format, rate limits, and the 5-tier capability list. Point your agent here first.
/api/v1/agent-guide.md — a 17-step golden-path loop that walks the agent through the entire process: discover → introspect → build → validate → execute → monitor.

The Golden-Path Loop

Here's what an agent does after reading our discovery docs:

Read /llms.txt to learn the API surface
Read /api/v1/agent-guide.md for the strategy
Discover available connection types with full config schemas
Create source + destination connections
Test both connections
Introspect the source to list tables and columns
Create an upload (extract + load job)
Run the upload and wait for completion
Browse the data catalog to see loaded tables
Write SQL transformations referencing those tables
Compile to validate — fix errors if any
Preview the output
Schedule the pipeline
Monitor runs

No guessing. No hallucination loops. Every step uses typed schemas and returns typed errors.

How Competitors Compare

Capability	Datanika	Fivetran	Airbyte	Stitch	Hevo
`/llms.txt` discovery	Yes	No	No	No	No
Agent strategy guide	Yes	No	No	No	No
Typed config schemas in API	Yes	No	Partial	No	No
Source introspection via API	Yes	No	Yes	No	No
Compile-before-run validation	Yes	No	No	No	No
Typed error codes for agents	Yes	No	No	No	No
Idempotent retries	Yes	Yes	No	No	No
Wait mode on run triggers	Yes	No	No	No	No

Fivetran has no agent API. Airbyte has no /llms.txt. dbt Cloud has no compile-before-run via API. None of them publish a strategy guide that tells an agent how to use the platform end-to-end.

Try It: Build a Pipeline with Your AI Agent

Point Claude, GPT, or any LLM at your Datanika instance:

import httpx

BASE = "https://app.datanika.io/api/v1"
HEADERS = {"Authorization": "Bearer etf_your_key"}

# 1. Discover what's available
types = httpx.get(f"{BASE}/meta/connection-types", headers=HEADERS).json()

# 2. Create a Stripe source
stripe = httpx.post(f"{BASE}/connections", headers=HEADERS, json={
    "name": "Stripe Production",
    "connection_type": "stripe",
    "config": {"api_key": "sk_live_..."},
}).json()

# 3. Create a BigQuery destination
bq = httpx.post(f"{BASE}/connections", headers=HEADERS, json={
    "name": "Analytics Warehouse",
    "connection_type": "bigquery",
    "config": {"project": "my-project", "dataset": "raw_stripe",
               "service_account_json": "..."},
}).json()

# 4. Introspect, build upload, run, transform...
# The agent guide at /api/v1/agent-guide.md has the full 17-step loop.

Or just tell your agent: "Read https://app.datanika.io/llms.txt and build me a Stripe-to-BigQuery pipeline."

For a head start, point the agent at our public pipeline templates — the Stripe → Postgres and Postgres → BigQuery pages ship pre-validated schemas, prerequisites, and example transformations that collapse the Tier 1 discovery loop into a single read.

What's Next

Update: the MCP server shipped. uvx datanika-mcp hands Claude Desktop, Claude Code, or Cursor 25 typed tools over the same API described above — read-only until you pass --allow-write. Setup is in the MCP server guide.

Still on the list: agent-specific observability — tracking which pipelines were built by agents versus humans.