DEV Community: Evgenii Timofeev

Google Sheets to Your Warehouse: the Share Step Is the One That Fails

Evgenii Timofeev — Sat, 26 Sep 2026 00:00:41 +0000

Google Sheets is the most common shadow database in any company. Marketing keeps campaign spend in one, finance maintains the budget model in another, ops runs a lightweight CRM in a third. None of it is in the warehouse, so none of it can be joined to anything, so somebody copy-pastes it into a dashboard once a month and everyone quietly agrees not to ask how fresh it is.

Landing a sheet in your warehouse is genuinely quick — a service account, four fields, a cron. This post is about the one step in the middle that fails for almost everybody the first time, and about a button that will refuse to tell you whether it worked.

The setup, briefly

Google Sheets is source-only in Datanika: you land it somewhere, you do not write back to it. You need a destination warehouse already connected — PostgreSQL is the quickest if you do not have one yet — and a Google Cloud project with the Google Sheets API enabled.

If you already run BigQuery with Datanika, reuse that same project and its service account; you only need to enable the Sheets API on it and do the sharing step below.

Create a service account — IAM & Admin → Service accounts — and give it no IAM roles at all. This trips people up, because granting a role feels like the responsible thing to do. It is not relevant here: the service account reaches Sheets through the Sheets API, not through GCP resources, so a role grant would widen its access without enabling anything you need. Create a JSON key, download it, and note the account's email address:

datanika-sheets-reader@your-project.iam.gserviceaccount.com

That address is the whole story of this post.

The step that fails

A Google service account can only read spreadsheets that have been explicitly shared with it. Not sheets your account can see. Not sheets in a Drive folder it can reach. The specific spreadsheet, shared with that specific address, the same way you would share it with a colleague.

So: open the sheet, click Share, paste datanika-sheets-reader@your-project.iam.gserviceaccount.com, set it to Viewer — Datanika never writes to Google Sheets — and, the part that catches people, untick "Notify people" before you confirm. The service account has no mailbox, so there is nobody to notify.

Repeat for every spreadsheet you want this connection to read. There is no folder-level shortcut.

The button that says "not tested"

Now add the connection. Open /connections — the form is already on the page, there is no "New Connection" button to hunt for — pick google_sheets from the type dropdown, and fill in three fields:

Connection Name — something you will recognise later, like gsheets-marketing-budget.
Spreadsheet URL — the full URL, https://docs.google.com/spreadsheets/d/<ID>/edit, not just the ID.
Service Account JSON — the entire contents of the key file. It is encrypted at rest with Fernet.

Then click Test Connection, and watch it decline to give you a verdict. It returns a neutral not tested, carrying the reason.

This is deliberate, and it is worth explaining, because a lot of tools would have shown you a green tick here.

Verifying a service-account credential means minting an OAuth token. That is a real check, and it would pass — the JSON is well-formed, the key is valid, the API is enabled. But it would tell you nothing about the failure you are actually going to hit, because the sharing permission is checked per upload, not per connection. A green tick at this point would be an accurate statement about the credential and a misleading one about the pipeline, and the person reading it cannot tell those apart.

The alternative — a red cross — is worse in the other direction: a connection that is fine, reported as broken. Reporting an unverified thing as working and reporting it as failed are the same lie told in two directions. So the button says what is true: it has not been tested here, and the first real verification is the first run.

Click Create Connection and go get one.

Configure the upload

Extract-load lives at /uploads, not on the connection. Connection rows offer Test / Edit / Copy / Delete and nothing else, and /pipelines is the dbt builder — a different thing entirely.

Open /uploads, and note the first quirk while you type: the upload name accepts letters, digits and spaces, and strips everything else as you type. sheets-daily-sync becomes sheetsdailysync in front of you, while Sheets Daily Sync is kept verbatim. This matters more than it looks, because the schedule you create later references the upload by name, exactly as saved.

Pick your source and destination connections — the pickers list entries as 16 — myconnection (postgres), so id, name, type — and then set:

Sheet Names (optional, comma-separated) — name the tabs you want. Leave it empty to load every tab in the spreadsheet.
Batch size — 10000 by default.
Schema Contract — three dropdowns, Tables / Columns / Data Type, deciding whether a changed incoming shape evolves the destination or fails the run. For a spreadsheet that humans edit, this is the setting worth thinking about hardest: somebody will add a column.

There is no write disposition, load mode, source schema or table-name field, and that is not an omission — those controls are rendered only when the source is a SQL database.

The first run is the actual test

Hit Run on the upload's row. There is no "Run now" on a pipeline page; the trigger lives on the upload's own row. Watch /runs: status badge, start and finish timestamps, a Rows count, and a Logs icon for the detail.

If the share step went wrong, this is where you find out.

When it finishes, open Models (/models). Your tables land in a schema derived from the upload's name: whitespace runs become single underscores and the whole thing is lower-cased. So sheetsdailysync creates schema sheetsdailysync, while Sheets Daily Sync — the name this post told you is kept verbatim — creates schema sheets_daily_sync. dlt also writes its own _dlt_loads, _dlt_pipeline_state and _dlt_version bookkeeping tables into that schema, and Models does not list them; seeing only your own tables there is correct, not a partial load. There is no target-schema field to choose.

Then do the thing the badge cannot do for you: spot-check the row count against the sheet. A green run means the load finished. It does not mean it moved what you expected — an empty tab and a tab you forgot to share are very different problems that can produce the same colour.

Put it on a cron

Schedules are at /schedules, and reference the upload by name:

Target type — upload.
Target name — exactly as saved, so sheetsdailysync.
Cron expression — a real five-field cron string. There is no cadence picker and no "manual only" option; leaving an upload unscheduled is manual-only. 0 * * * * hourly, 0 */6 * * * every six hours, 0 3 * * * nightly at 03:00.
Timezone — UTC by default, and the cron is evaluated in it, which matters for daily and weekly cadences.

Then wire failure alerts in Settings → Notifications, because a spreadsheet pipeline breaks for a reason no other pipeline has: somebody renames a tab.

The one-line version

Everything in a Sheets sync is easy except remembering that the service account is a stranger to your spreadsheet until you introduce them. If a run fails and the credential is fine, you already know which step to go back to.

Full field-by-field reference: the Google Sheets setup guide and the connector page.

A Customer 360 from HubSpot and Stripe: the Join Is the Work

Evgenii Timofeev — Fri, 25 Sep 2026 08:07:46 +0000

Every "customer 360" tutorial shows you the easy half. Sync your CRM, sync your billing system, write a sources.yml, and then — with no ceremony at all — this line:

join hubspot_contacts hc on hc.email = sc.email

That line is where the tutorial ends and the work starts. HubSpot and Stripe share no identifier. Not one. There is no HubSpot ID in Stripe and no Stripe ID in HubSpot unless somebody deliberately put it there. The only overlapping value is an email address, typed by a human into two different forms, possibly years apart, possibly not the same address.

So the question this post answers is not "how do I join these." It is what fraction of your customers actually match, how do you find out, and what do you do with the ones that don't — because in every real dataset there are some, and the difference between a trustworthy customer 360 and a misleading one is entirely in how honestly you handle them.

If you want the Stripe-only revenue modelling first — MRR, churn, LTV — that is a separate post. This one assumes you have Stripe landed and adds HubSpot beside it.

What the two systems actually give you

Worth being precise, because half the tutorials out there join against tables that don't exist.

In Datanika, HubSpot and Stripe are SaaS sources, so the upload form shows Select endpoints to load — a checkbox per resource. The full lists:

Source	Endpoints
HubSpot	`companies`, `contacts`, `deals`
Stripe	`charges`, `customers`, `invoices`, `prices`, `products`, `subscriptions`

Nine tables. Scan them for a shared key and you will not find one. hubspot.contacts has an email and an associated company; stripe.customers has an email and a name. That is the entire overlap, and both sides are free text.

Where the tables land. A SaaS source has no write-disposition, load-mode or target-schema field — those are rendered only for SQL database sources. On BigQuery / Snowflake / Databricks the schema is the Dataset/Schema field on the destination connection. On Postgres / MySQL / DuckDB it is the upload's own name, snake-cased — and upload names accept letters, digits and spaces only, so you name the upload Raw Hubspot and get the schema raw_hubspot. Check the Schema column in Models — the sidebar entry for Datanika's data catalog, at /models — for the real name before writing sources.yml; a wrong schema name is the most common reason the models below don't resolve.

Two uploads, two schemas:

# models/staging/sources.yml
version: 2

sources:
  - name: raw_stripe
    tables: [{name: customers}, {name: subscriptions}, {name: invoices}]
  - name: raw_hubspot
    tables: [{name: contacts}, {name: companies}, {name: deals}]

HubSpot returns default properties only. If email or domain is missing from a landed table, it is not a Datanika bug — HubSpot's API returns a default property set unless the pipeline asks for more. Add the custom properties you need to the resource config before you go hunting through dbt.

Step 1 — Normalize the identifiers before you join anything

Never join on a raw email column. Email is case-insensitive in the domain part, effectively case-insensitive at every provider anyone uses, frequently has trailing whitespace from a paste, and Gmail ignores dots and +tags. Two staging models, one job each: produce a match_email you can trust and a email_domain you can fall back to.

-- models/staging/stg_hubspot__contacts.sql
{{ config(materialized='view') }}

select
    hs_object_id                              as contact_id,
    associatedcompanyid                       as company_id,
    email                                     as email_raw,
    lower(trim(email))                        as match_email,
    split(lower(trim(email)), '@')[offset(1)] as email_domain,   -- BigQuery
    firstname,
    lastname,
    createdate                                as created_at
from {{ source('raw_hubspot', 'contacts') }}
where email is not null

-- models/staging/stg_stripe__customers.sql  (extends the version in the revenue post)
{{ config(materialized='view') }}

select
    id                                        as customer_id,
    email                                     as email_raw,
    lower(trim(email))                        as match_email,
    split(lower(trim(email)), '@')[offset(1)] as email_domain,
    name,
    -- the exact join key, when someone was disciplined enough to write it
    json_value(metadata, '$.hubspot_contact_id') as hubspot_contact_id_from_metadata,
    timestamp_seconds(created)                as created_at
from {{ source('raw_stripe', 'customers') }}

Dialect. split(...)[offset(1)] and json_value are BigQuery. Snowflake: split_part(lower(trim(email)), '@', 2) and metadata:hubspot_contact_id::string. Postgres/DuckDB: split_part(...) and metadata ->> 'hubspot_contact_id'. Everything after staging is plain SQL.

The two remaining HubSpot models are thin. Note that companies.domain gets the same lower(trim(...)) treatment — it is the Tier C join key, and a stray capital there costs you matches for no reason:

-- models/staging/stg_hubspot__companies.sql
{{ config(materialized='view') }}

select
    hs_object_id        as company_id,
    name,
    lower(trim(domain)) as domain
from {{ source('raw_hubspot', 'companies') }}
where domain is not null

-- models/staging/stg_hubspot__deals.sql
{{ config(materialized='view') }}

select
    hs_object_id       as deal_id,
    associatedcompanyid as company_id,   -- see the caveat below
    dealname,
    amount,
    dealstage,
    closedate
from {{ source('raw_hubspot', 'deals') }}

Check the deal→company column before you trust it. In HubSpot, a deal's link to a company is an association, not a plain property, and which column (if any) lands depends on the properties your pipeline requests. Open Models and look at the landed deals table. If there is no company column, either add the association to the resource config or drop the two deal columns from the final model — the customer 360 is still worth building without them. Do not join deals on company name.

Step 2 — Three joins, in descending order of trust

This is the part worth reading twice. There is not one join. There are three, they have very different reliability, and a customer 360 that treats them as interchangeable is quietly wrong.

Tier A — an explicit key in Stripe `metadata` (exact)

Stripe lets you attach arbitrary metadata to a customer. If your signup flow writes the HubSpot contact ID there, you have a real foreign key and none of the rest of this post applies to those rows.

Almost nobody has this on day one, because it has to be decided before the customers exist. If you take one action from this post, make it this one: start writing hubspot_contact_id into Stripe customer metadata today. It does nothing for your backlog and it makes every future row exact.

Tier B — normalized email (good, and incomplete)

The workhorse. Matches whenever the human typed the same address into both systems.

It fails in ways worth knowing, because each one is a real customer in your unmatched pile:

signed up with dana@acme.com, billing goes through accounts-payable@acme.com
signed up with a personal address, expensed it later
the company was acquired and the domain changed
one side has a typo

Tier C — email domain → company domain (recovers real customers, over-matches on public domains)

When the emails differ but both live at acme.com, HubSpot's companies.domain bridges them. This is how you catch the accounts-payable@ case, and it is the tier that needs a guard:

gmail.com is not a company. Join two consumer addresses on their domain and you will merge unrelated people into one "customer" with a straight face. Exclude free providers explicitly, and treat a domain matching many distinct Stripe customers as a signal to stop rather than a bigger match.

-- models/marts/customer_identity_map.sql
{{ config(materialized='table') }}

with stripe as (select * from {{ ref('stg_stripe__customers') }}),
     contacts as (select * from {{ ref('stg_hubspot__contacts') }}),
     companies as (select * from {{ ref('stg_hubspot__companies') }}),

free_domains as (
    select domain from unnest([
        'gmail.com','googlemail.com','yahoo.com','outlook.com','hotmail.com',
        'live.com','icloud.com','me.com','proton.me','protonmail.com','aol.com'
    ]) as domain
),

-- Tier C is only trustworthy where the domain identifies ONE company. A domain
-- shared by many Stripe customers is either a free provider we missed or an
-- agency billing for several clients; both should fall through to unmatched.
safe_domains as (
    select email_domain
    from stripe
    where email_domain not in (select domain from free_domains)
    group by email_domain
    having count(distinct customer_id) = 1
),

tier_a as (
    select s.customer_id, c.contact_id, 'A_metadata' as match_tier
    from stripe s
    join contacts c on c.contact_id = s.hubspot_contact_id_from_metadata
),

tier_b as (
    select s.customer_id, c.contact_id, 'B_email' as match_tier
    from stripe s
    join contacts c on c.match_email = s.match_email
    where s.customer_id not in (select customer_id from tier_a)
),

tier_c as (
    select s.customer_id, c.contact_id, 'C_domain' as match_tier
    from stripe s
    join safe_domains d on d.email_domain = s.email_domain
    join companies co on co.domain = s.email_domain
    join contacts c on c.company_id = co.company_id
    where s.customer_id not in (select customer_id from tier_a)
      and s.customer_id not in (select customer_id from tier_b)
),

unmatched as (
    select s.customer_id, cast(null as string) as contact_id, 'UNMATCHED' as match_tier
    from stripe s
    where s.customer_id not in (select customer_id from tier_a)
      and s.customer_id not in (select customer_id from tier_b)
      and s.customer_id not in (select customer_id from tier_c)
)

select * from tier_a
union all select * from tier_b
union all select * from tier_c
union all select * from unmatched

Two design choices worth stating, because they are the difference between this and the one-liner:

Every Stripe customer appears exactly once, including the ones that matched nothing. A map that silently drops rows turns "we couldn't match 18% of your revenue" into "revenue is 18% lower than Stripe says," and someone will spend a day on that.
match_tier is a column, not a comment. It travels downstream, so any number built on this map can be recomputed for exact matches only.

Step 3 — Measure the match rate; don't assume it

Here is the model that makes this a system instead of a query. It answers "how much of this do I believe?" and it answers it every run.

-- models/marts/customer_identity_coverage.sql
{{ config(materialized='table') }}

select
    match_tier,
    count(*)                                                    as customers,
    round(100.0 * count(*) / sum(count(*)) over (), 1)          as pct_of_customers
from {{ ref('customer_identity_map') }}
group by match_tier

Then turn the number into a gate. dbt tests fail builds; that is the point:

# models/marts/schema.yml
version: 2

models:
  - name: customer_identity_map
    columns:
      - name: customer_id
        tests: [not_null, unique]        # a fan-out here means a duplicate contact
      - name: match_tier
        tests:
          - accepted_values:
              values: ['A_metadata', 'B_email', 'C_domain', 'UNMATCHED']

-- tests/assert_identity_coverage_above_threshold.sql
-- Fails when unmatched customers exceed 25%. Set the threshold to a little worse
-- than today's real number, then tighten it as you fix the data. A threshold you
-- have never been near is not a test — it is decoration.
select 1
from (
    select
        countif(match_tier = 'UNMATCHED') / count(*) as unmatched_rate
    from {{ ref('customer_identity_map') }}
)
where unmatched_rate > 0.25

Run it the first time and read the number before you set the threshold. Whatever it is, it is your number, and it is the most useful single fact in this entire pipeline: nobody who joins on email has any idea what theirs is.

Step 4 — Now the 360 is boring, which is the goal

With a map that every row goes through, the customer view is a straightforward assembly:

-- models/marts/customer_360.sql
{{ config(materialized='table') }}

select
    m.customer_id,
    m.match_tier,
    sc.email_raw                    as billing_email,
    sc.name                         as billing_name,
    hc.firstname, hc.lastname,
    co.name                         as company_name,
    co.domain                       as company_domain,

    -- billing side
    count(distinct sub.subscription_id)                          as subscriptions,
    countif(sub.status = 'active')                               as active_subscriptions,
    coalesce(sum(inv.amount_paid), 0)                            as lifetime_revenue,
    min(sc.created_at)                                           as billing_since,

    -- CRM side
    count(distinct d.deal_id)                                    as deals,
    max(d.closedate)                                             as last_deal_closed_at

from {{ ref('customer_identity_map') }} m
left join {{ ref('stg_stripe__customers') }}    sc using (customer_id)
left join {{ ref('stg_hubspot__contacts') }}    hc on hc.contact_id = m.contact_id
left join {{ ref('stg_hubspot__companies') }}   co on co.company_id = hc.company_id
left join {{ ref('stg_stripe__subscriptions') }} sub using (customer_id)
left join {{ ref('stg_stripe__invoices') }}     inv on inv.customer_id = m.customer_id
                                                   and inv.status = 'paid'
left join {{ ref('stg_hubspot__deals') }}       d  on d.company_id = co.company_id
group by 1,2,3,4,5,6,7,8

Every join is a left join from the map, so an unmatched Stripe customer still produces a row with real billing figures and null CRM fields. That is the honest shape: you know what you know.

Step 5 — Ship the unmatched list to the humans

The unmatched bucket is not an error state. It is a work queue, and it is the highest-value output of the whole build — every row is a paying customer your CRM cannot see.

-- models/marts/unmatched_paying_customers.sql
select
    c.customer_id,
    c.billing_email,
    c.billing_name,
    c.lifetime_revenue
from {{ ref('customer_360') }} c
where c.match_tier = 'UNMATCHED'
  and c.lifetime_revenue > 0
order by c.lifetime_revenue desc

Sorted by revenue, because the ten unmatched customers who pay you the most are worth an afternoon of manual reconciliation and the long tail is worth an automation. Every row someone fixes moves permanently into Tier A if they write the metadata back.

What this does not do

It is not identity resolution as a product. No fuzzy name matching, no phonetic keys, no probabilistic scoring. Deterministic tiers with an explicit unmatched bucket beat a similarity threshold you cannot explain to finance.
It does not deduplicate within a system. Two HubSpot contacts for the same person will both match the same Stripe customer and trip the unique test on customer_id — deliberately. Fix the CRM; don't paper over it in SQL.
It has no opinion on which side is right when HubSpot and Stripe disagree about a name or a company. The model keeps both columns and lets you choose.
Tier C is a heuristic. The count(distinct customer_id) = 1 guard and the free-domain list make it defensible, not correct. If a number would embarrass you at 5% error, compute it on Tier A and B only — match_tier is right there.

Wire it up

Two connections — HubSpot (private app token, crm.objects.{contacts,companies,deals}.read) and Stripe (a restricted read key, never a secret key).
Two uploads. Name them so the schemas come out right, and confirm in Models.
The models above under Transformations, then schedule the transform to depend on both uploads — a 360 built from a fresh Stripe and yesterday's HubSpot is a 360 that disagrees with itself. See the scheduling guide.
Point a dashboard at customer_360 and customer_identity_coverage. Put the coverage number on the dashboard, not in a runbook. A match rate nobody looks at drifts.

Datanika meters bytes processed; both of these are narrow JSON sources and the Free plan includes 10 GB/month. Since 2026-09-24 the dashboard's Plan Usage panel carries a bytes processed dimension against your plan's included volume (datanika-core#1513); it appears once a pipeline has written volume data, so there is nothing to read there before your first run. Size the first estimate from your own data — the meter counts what an upload writes after normalization.

Next steps

Stripe revenue dashboard with dbt — MRR, churn and LTV on the Stripe half.
HubSpot setup guide · Stripe setup guide
Transformations — how models, tests and schedules fit together.
All connectors — the same identity map takes a third source the day you add one.

The join is the work. Measure it, publish the number, and hand the leftovers to a human.

Correction, 2026-09-23. The cost line told you to check Usage for your own byte figures. That card counted model runs, not bytes, and at the time no screen in the app showed a byte count, so the line was changed to size it from your data instead. The same sentence was corrected in the Stripe revenue post a day earlier; tracked in datanika-core#1513.

Update, 2026-09-24. datanika-core#1513 shipped, and the dashboard's usage card now carries a bytes-processed dimension — verified on the serving container across that day's deploy. The cost line names the screen again, together with the condition that makes the figure appear.

We're Adding a Volume Dimension to Our Pricing. Here's the Math, and Here's Why.

Evgenii Timofeev — Thu, 24 Sep 2026 09:01:47 +0000

Our v1 pricing had a known hole. We're closing it now.

Specifically: we charged $79/mo for "Pro" with a 15,000-runs-per-month quota and no cap on the volume of data those runs moved. On paper, fine. In practice, a single customer running one pipeline that shovels 1 TB of Postgres into BigQuery per month would cost us $50–$150 in infrastructure to serve — on a $79 bill. Repeat that across ten customers and the business is upside-down.

We don't have ten of those customers yet. We don't have one of them yet. Which is exactly why we're fixing this now, before the first one arrives — not after, when it turns into an apology post.

What changed

Plan	Old	New
Free	500 runs, no volume cap	10 GB processed/mo + 500 runs
Pro ($79/mo)	15,000 runs, no volume cap	100 GB processed/mo, $0.50/extra GB, 15,000 runs
Enterprise (from $399/mo)	50,000 runs, $0.01/run overage, no volume cap	1 TB processed/mo, $0.25/extra GB, 50,000 runs

Seats, connections, and schedules stay the same. SSO stays on Enterprise. The tiers haven't moved; the meter has.

Every GB on this page — and every number derived from one below — is the binary GB our meter counts: 1,073,741,824 bytes (2³⁰), which is 7.4% more data than the decimal GB most warehouse consoles report. Worth knowing before you check our arithmetic against your own, because counting in decimal makes your usage look larger than we meter it.

Why "volume" was missing

Because we shipped v1 with Paddle's default-shape subscription plans — a flat monthly fee plus a secondary usage meter (model runs). That captured the obvious cost driver (orchestration, scheduler CPU, log storage) but ignored the expensive one (the actual bytes that hit disk, get normalized, get re-read by dbt, and get written to the destination).

Run counts correlate with some costs. They don't correlate with the cost that scales with your success as a customer. A pipeline running 30×/day with 500 rows per run is cheap for us. A pipeline running once a day with a full-history dump of your production database is not. v1 charged both the same.

That's the hole.

The math that broke it

Take one customer running one pipeline: a nightly export of a Postgres table with a million wide rows. Each row is ~1 KB, so the raw export is ~1 GB. Nested JSON columns flatten to ~3 GB after normalization. A dbt model aggregates that to a 100 MB summary. Total bytes touched per run: roughly 3.1 GB. Repeat nightly for 30 days: ~93 GB/mo.

At v1, we billed $79 for this. At cloud-provider rates for storage + CPU + egress, we spent $12–$25 on infrastructure. That's fine. We're profitable on this shape.

Now scale it to the same customer adding three more pipelines: a Stripe export, a HubSpot sync, a Segment event feed. Each one amplifies 2–5× after normalization. Total bytes/mo: ~400–600 GB. Infrastructure cost: $50–$100. Still $79 on the bill.

Scale once more to a customer with a real data footprint — 1 TB/mo across 8 pipelines — and we're spending $120+ on a $79 subscription. That's the bill that convinced us the pricing was wrong.

If you're processing more than 740 GB/mo — binary GB, as above; about 795 GB as a warehouse console counts them — Enterprise's $0.25/GB rate saves you more than the subscription difference. The pricing calculator auto-picks the cheaper tier for you — no mental math required.

Why GB and not MAR

Fivetran's "monthly active rows" pricing is the obvious alternative. We looked at it; we're not doing it.

MAR punishes schema choices you didn't make. A table with 10M narrow rows and a table with 100K wide rows can hit an identical disk-and-CPU bill but a 100× MAR bill. MAR also re-counts edits — update one row ten times this month and Fivetran charges you for ten rows. The customer has no way to predict the bill until the sync runs.

GB measures the cost driver directly. A wide JSON blob and a narrow normalized row land on the same gram of disk for the same price. Updates don't inflate the count. You can predict your bill by looking at du -sh on your source and multiplying by a constant.

Fivetran publishes no per-MAR rate, so its side of this comparison is an illustrative estimate rather than a quote. Converting at ~200K rows/GB, 100 GB on Starter lands in the region of $3,800/mo and 1 TB somewhere around $22,000/mo. On Datanika Pro at 100 GB you pay $79, and on Enterprise at 1 TB, $399 + nothing (it's included) — those two are exact. The calculator on /why-cheaper/ shows the comparison side-by-side with a slider, and Fivetran's own estimator is where a binding number comes from.

That's what we mean by "you pay for bytes, not tables."

How we meter honestly

We count output bytes after normalization — the amplified number, not the raw input. We do this because the amplified number is what our infrastructure actually touches, and pretending otherwise creates a gap between the sticker and the bill.

Worked example: you have a 1 GB HubSpot JSON export. Our ingestion flattens nested objects into a wide table — that's ~3 GB of post-normalization data, and ~3 GB is what counts against your quota, not 1 GB. A dbt model that aggregates it to a 100 MB summary adds nothing to that: only uploads are metered in bytes, and a model run counts as a model run.

The reason we publish the amplification rather than hiding it is that it makes the bill something you can work out in advance yourself. A gigabyte is a unit you can count; the rate is on the pricing page; the multiplication is yours. MAR is the opposite kind of unit — it is defined and counted by the vendor, so the invoice is the first place you get to see it.

What a month costs

The meter reads what an upload wrote: your data is normalized on our side, so a 1 GB JSON export becomes ~3 GB of flat tables and the meter counts ~3 GB.

At $0.50/GB overage (Pro), each nightly run past your included 100 GB adds 3 GB to the month's metered volume — $1.50 worth. Over 30 nightly runs that's 90 GB: $45. (Overage is totalled once per billing cycle and rounded up to the next whole GB, so that per-run figure is the monthly bill decomposed — not a charge you're billed run by run.)

What stays

Self-host is still $0 forever. The AGPL-3.0 open-source core has zero pricing dimensions. Run it on your own hardware, ingest 10 TB/mo, pay us nothing.
10 GB Free is real headroom. That's enough for a real side project or a 3-source trickle-volume evaluation stack — a genuine test with production-shaped data, not a crippled sandbox. Fivetran Free tops out at 500K MAR (~1–2 GB equivalent); Hevo Free at 1M events. We offer 5–10× more.
All 36 connectors on every tier. Free users don't get a crippled connector list. Fivetran adds a $5 base charge to each standard connection using under 1M MAR a month; we don't, and we're not going to.
Pro's 5 seats, Enterprise's 10 seats, schedules unlimited on paid tiers. Seat economics are unchanged.
Annual discount at 17% (Pro $79 → $66/mo billed annually; Enterprise from $399 → $333). We'll revisit after 90 days of real signup data — if the math says 20% works, we'll adjust and blog about it.

What we'll do if we got the numbers wrong

We picked 10 GB / 100 GB / 1 TB based on the cost model in price_insights.md §7 and a napkin-margin target of 80–95% on variable cost. We don't yet have real signup data to validate those numbers against customer reality.

Commitment: 90 days from now we look at actual usage distributions against actual infrastructure cost. If 100 GB on Pro is the wrong number — too tight for the real median customer, or too generous for our margin — we adjust the Pro tier's included volume. We blog about it when we do. The overage rate won't move in year one; we picked $0.50/GB and $0.25/GB to sit comfortably above our variable cost at every volume we've modeled.

If you're already on Datanika when we adjust: your current tier honors its current numbers until your next renewal, and we email you the change at least 30 days before it hits. Not that there's anyone on paid Datanika yet — this is the policy for when there is.

What this doesn't mean

We are not becoming a per-row-pricing company. We are not adding event fees. We are not going to count deletes. "Processed GB" is the only new meter. Everything else on your bill — seats, connections, schedules, support — stays on the subscription.

If you've been evaluating Datanika on the v1 pricing page and waiting to decide: the economics on your bill are now predictable against your data volume, which is probably the thing you actually wanted.

The numbers, measured

On a standard Hetzner CPX32 (4 vCPU, 8 GB RAM, €13/mo), dlt — the library a Datanika upload runs — processed 17,704 rows/second on a 10.1M-row Postgres → DuckDB pipeline, full extract, normalize and load, with a p95 of 571s across 3 runs (569.9s, 570.5s, 571.4s). The benchmark script calls dlt directly rather than going through a Datanika upload, so it measures the library, not the product. Full benchmark log and methodology in Datanika vs. the Modern Data Stack.

The /why-cheaper/ calculator lets you drag a slider from 1 GB to 10 TB and see the cost side-by-side with Fivetran Starter, with Datanika auto-picking the cheapest tier: Free up to 10 GB, then Pro-with-overage or Enterprise-flat, whichever costs less.

Try it free at app.datanika.io — 10 GB/mo on Free, no credit card, and one published per-GB rate with no MAR arithmetic to reverse-engineer.

Correction, 2026-08-31. An earlier version of this post said Pro and Enterprise pipelines show a pre-run predicted_bytes estimate, derived from a moving average of the last five runs, before you click "Run." That was written from the pricing spec rather than from the product: the estimate is not computed today, on any plan. The paragraph has been replaced with what the meter actually does. Tracked in landing#375.

Correction, 2026-09-22. Three more passages described something other than the product. The post had a section titled "Pick ELT, pay less". That section said every pipeline has an ETL/ELT mode selector, and that ELT is metered at ~0.8 GB where ETL is metered at ~3 GB. No pipeline can be switched to ELT: the app shows no mode selector, and nothing saves a mode, so every run takes the normalizing path described above. The worked example also counted a dbt model's 100 MB against your quota. It does not: only uploads are metered in bytes. And the benchmark was credited to Datanika when it measured dlt called directly by a script. All three passages now say what the product does. Separately, the post said Fivetran charges $5 per connection on top of MAR; Fivetran's own pricing page applies that base charge only to standard connections using between 1 and 1M MAR a month, and not on its Free plan, so the sentence now says so. Tracked in landing#656.

How We Built a 5.8x Faster Data Pipeline with DLT + Arrow

Evgenii Timofeev — Wed, 23 Sep 2026 12:35:11 +0000

Before Datanika existed, the founding team ran production data pipelines at Whisk — a food-tech platform processing 54 million MySQL rows and 8.6 million MongoDB documents daily into ClickHouse. We went through three generations of architecture, and the numbers tell the story better than any pitch.

The three generations

v1: Custom Airflow operators (99 min for 54M rows)

Each source type had its own operator. MySqlToClickHouseTransfer, PostgresToClickHouseOperator, MongoToCH — four files with duplicated logic, no shared schema management, no execution logging, no incremental state.

It worked. For small tables it was fine. But the main table — user_recipe_rel, 54 million rows — took 99 minutes as a batch INSERT over the native ClickHouse protocol. Acceptable, not fast.

v2: DLT + JSON normalization (174 min — slower)

We unified everything behind DLT's sql_database source and a single DltRunOperator. YAML-driven table configs, automatic schema evolution, centralized execution logging, incremental state tracking. The operator improvement was night and day.

The performance was not.

DLT's default mode processes rows as Python dicts: fetch from source, convert each row to a dict, run schema inference, write to JSONL, convert to Parquet, upload. For 54 million rows, this JSON normalization pipeline took 174 minutes — 75% slower than the legacy batch INSERT.

The operator was better. The throughput was worse.

v3: DLT + Arrow mode (17 min — 5.8x faster)

Arrow mode changes the data flow completely:

MySQL --> Server-side cursor --> PyArrow Tables --> Parquet Files --> ClickHouse
          (chunk_size rows)     (in memory)        (file_max_bytes)   (HTTP insert)

No dict conversion, no schema inference, no JSONL intermediate step. The source yields pa.Table objects directly, DLT writes them to Parquet files, and the Parquet files upload to ClickHouse via HTTP.

54 million rows in 17 minutes. 5.8x faster than legacy, 10x faster than DLT + JSON.

The numbers

MySQL: `user_recipe_rel` (54M rows, production)

Metric	Legacy operator	DLT + JSON	DLT + Arrow
Fetch from MySQL	~99 min (bundled)	~72 min	~10 min
Normalize + Load	bundled	~102 min	~7 min
Total time	99 min	174 min	17 min
Load files	batch inserts	547	5,462
Speedup vs legacy	baseline	0.57x (slower)	5.8x faster

The Arrow fetch is 7x faster because it reads columnar batches instead of iterating row-by-row through Python dicts. The normalize step drops from 102 minutes to effectively zero because Arrow's normalizer does a direct Parquet file import — a file rename, not data processing.

MongoDB: `hostedImages` (8.6M documents, dev)

Metric	Legacy operator	DLT + JSON	DLT + Arrow
Fetch from MongoDB	~49 min (prod, 29.7M)	~11 min	~3.3 min
Normalize	N/A	~12 min (JSONL)	~1 sec
Load to ClickHouse	bundled	~1 min	~40 sec
Total	~49 min (29.7M)	~23.5 min (8.6M)	~4 min (8.6M)
Normalized: sec/M rows	99s	164s	28s
vs legacy	baseline	0.6x (slower)	3.5x faster

Extrapolated to production scale (29.7M documents), DLT + Arrow would complete in approximately 14 minutes versus the legacy operator's 49 minutes.

The `file_max_bytes` war story

Arrow mode has one critical configuration that doesn't exist in JSON mode: DATA_WRITER__FILE_MAX_BYTES.

Without it, DLT concatenates all Arrow chunks from the extract step into a single Parquet file. For 54 million rows, that's a multi-gigabyte file. And when you try to upload a 2 GB Parquet file via HTTP to ClickHouse through an nginx proxy, you get a timeout.

The fix: set DATA_WRITER__FILE_MAX_BYTES=100MB. This splits the extract output into manageable ~100 MB Parquet files — 5,462 files for the 54M-row table. Each uploads in seconds.

The gotcha: the environment variable must use the DATA_WRITER__ prefix, not NORMALIZE__DATA_WRITER__. Arrow's direct-import path skips the normalize step entirely, so normalize-scoped config has no effect. We discovered this after a week of "why is NORMALIZE__DATA_WRITER__FILE_MAX_ITEMS not splitting files" debugging.

Why this matters for Datanika

Datanika runs dlt, so the trade-off this post measured sits under every upload.

There is no ELT mode to switch a Datanika pipeline to. Every upload runs through dlt's normalize step on our side, and the meter counts what that step writes: a 1 GB JSON export that becomes ~3 GB of flat tables is metered as ~3 GB.

Our CPX32 benchmark (10.1M rows, Postgres to DuckDB) is not an Arrow number either. Its script calls dlt with the default backend, not Arrow, and completes the full extract, normalize and load in 571 seconds at 17,704 rows/s. It measures dlt called directly, not a Datanika upload.

Key takeaways

DLT's JSON normalization is the bottleneck, not the source fetch. Switching to Arrow skips it entirely.
Arrow mode requires explicit file splitting. Without DATA_WRITER__FILE_MAX_BYTES, you'll build multi-GB files that timeout on upload.
The speedup is not from Arrow being faster at reading — it's from eliminating the dict-to-JSON-to-Parquet conversion chain.

If you're running dlt at scale and haven't tried Arrow mode yet, the performance gain is significant and the migration is straightforward — set use_arrow: True in your source config and add the DATA_WRITER__FILE_MAX_BYTES env var.

Datanika is an open-source data pipeline platform built on dlt + dbt-core. Start free with 10 GB/mo, self-host it, or read about our volume-based pricing.

Correction, 2026-09-22. An earlier version of this post said this architecture powers a Datanika "ELT mode" you can select on a pipeline, metered at ~0.8 GB where ETL is metered at ~3 GB. There is no such mode to select: the app shows no mode selector and nothing saves a mode, so every upload runs the normalizing path. The post also said the CPX32 benchmark was built on this Arrow path, but that script uses dlt's default backend. The "Why this matters for Datanika" section has been rewritten to say what the product does, and a fourth takeaway that repeated the claim is gone. Tracked in landing#656.

How to Build a Stripe Revenue Dashboard with dbt

Evgenii Timofeev — Tue, 22 Sep 2026 14:38:59 +0000

Stripe's built-in dashboard is genuinely good — for operations. You can see today's charges, chase a failed payment, and eyeball gross volume. But the moment finance asks "what was net MRR in March, split by plan, excluding trials and after refunds?" you're exporting CSVs and fighting a pivot table. Stripe's reporting is deliberately shallow because Stripe is a payments processor, not an analytics warehouse.

The fix is the same one every serious SaaS team eventually lands on: get Stripe's raw objects into a warehouse, model them with dbt, and point a BI tool at the result. Once the data is modeled, MRR, churn, ARR, LTV, and cohort retention are just SQL — recomputed every morning, joinable against product usage, and auditable line by line.

This is the end-to-end tutorial. We'll land Stripe in a warehouse with Datanika (which wraps dlt for extract and dbt-core for transform in one app), write the staging and mart models, add tests, schedule the refresh, and connect a dashboard. Every SQL snippet below is real and runnable — adjust the column names to match what actually lands in your warehouse and you're done.

What we're building

Stripe API
   │  (dlt, via Datanika)
   ▼
raw_stripe.*          ← raw objects: customers, invoices, subscriptions, charges
   │  (dbt staging: clean + cast)
   ▼
stg_stripe__*         ← typed, renamed, cents → dollars, epoch → timestamp
   │  (dbt marts: business logic)
   ▼
revenue_by_month      ┐
active_subscriptions  ├→  Metabase / Looker / your BI tool
customer_ltv          ┘

Three layers, one principle: raw data lands untouched, staging cleans it, marts hold the business logic. When someone disputes an MRR number six months from now, you can trace it from the dashboard tile all the way back to a specific Stripe invoice.

Step 1 — Land Stripe in your warehouse

We won't repeat the full connector walkthrough here — the Stripe setup guide covers it end to end (create a read-only restricted key, add the connection, pick resources, run, schedule). The short version:

In Stripe, create a restricted key (Developers → API keys → Create restricted key) with Read permission on Customers, Charges, Invoices, Subscriptions, Products, and Prices. Never use a standard secret key — Datanika only ever reads.
In Datanika, open /connections, pick Stripe, and paste the key (stored encrypted at rest with Fernet).
Create the upload. Because Stripe is a SaaS source, the form shows Select endpoints to load — a checkbox each for charges, customers, invoices, prices, products and subscriptions, all ticked by default. Untick what you don't need; each ticked endpoint becomes its own table.
Run it from the /uploads row for your upload — the button is Run, and the trigger lives on the upload's own row, not on a pipeline page. Watch it in Runs; the Rows column there is one total for the whole run, not a per-endpoint breakdown.

Where it lands, and why there's no field for it. A SaaS source has no write-disposition, load-mode, source-schema or table-name control — those are rendered only for SQL database sources, and the endpoint checkboxes are the equivalent. So the destination schema is derived, not typed:

BigQuery, Snowflake, Databricks — it's the Dataset / Schema field on the destination connection, not on the upload.

Postgres, MySQL, DuckDB and the rest — it's the upload's own name, snake-cased. Upload names accept letters, digits and spaces only, so you can't type raw_stripe directly: name the upload Raw Stripe and you get the schema raw_stripe.

This tutorial assumes you did one of those two things and ended up with raw_stripe. If you named the upload something else, substitute your schema everywhere below.

When it finishes, open Models — the sidebar entry for Datanika's data catalog, at /models. You should see one row per landed table: customers, invoices, subscriptions, charges, and so on, each with its Schema and Columns. dlt's _dlt_loads / _dlt_pipeline_state / _dlt_version bookkeeping tables are created in the warehouse but filtered out of the catalog, so their absence here is expected, not a failed run. Confirm the schema name in the Schema column before you write sources.yml — it is the single most common reason the models below fail to resolve. Keep that tab open; you'll use it to check exact column names as you write models too.

Two things to know about Stripe's data before you model it. (1) Amounts are integers in the smallest currency unit — 2000 means $20.00 in USD. You divide by 100 in staging (except zero-decimal currencies like JPY — more on that below). (2) Timestamps are Unix epoch seconds — 1719792000, not 2024-07-01. dlt sometimes types these as proper timestamps during normalization and sometimes lands them as integers; check the column's type in Models and convert in staging if needed.

Step 2 — Declare the source

In Datanika, dbt models live under Transformations. Start by telling dbt where the raw data is. Create a sources.yml (or add to your existing one):

# models/staging/stripe/sources.yml
version: 2

sources:
  - name: raw_stripe
    description: "Raw Stripe objects loaded by Datanika/dlt"
    tables:
      - name: customers
      - name: invoices
      - name: subscriptions
      - name: charges
      - name: prices

Now {{ source('raw_stripe', 'invoices') }} resolves to the landed table, and dbt tracks the dependency in its DAG.

Step 3 — Staging models: clean and cast

Staging models are thin views, one per raw table. They do exactly four things: rename to your conventions, cast types, convert cents and epochs, and drop columns you'll never use. No business logic — that's the marts' job.

Stripe stores foreign keys under the object's own name (customer, subscription), so we rename them to _id for clarity.

-- models/staging/stripe/stg_stripe__invoices.sql
{{ config(materialized='view') }}

select
    id                                          as invoice_id,
    customer                                    as customer_id,
    subscription                                as subscription_id,
    status,
    currency,

    -- cents → major units. Guard zero-decimal currencies (JPY, KRW…):
    amount_paid / 100.0                         as amount_paid,
    amount_due  / 100.0                         as amount_due,
    total       / 100.0                         as total,

    -- Unix epoch → timestamp (BigQuery). See dialect notes below.
    timestamp_seconds(created)                  as created_at,
    timestamp_seconds(period_start)             as period_start,
    timestamp_seconds(period_end)               as period_end
from {{ source('raw_stripe', 'invoices') }}

-- models/staging/stripe/stg_stripe__subscriptions.sql
{{ config(materialized='view') }}

select
    id                                          as subscription_id,
    customer                                    as customer_id,
    status,                                     -- active, trialing, past_due, canceled…
    timestamp_seconds(created)                  as created_at,
    timestamp_seconds(current_period_start)     as current_period_start,
    timestamp_seconds(current_period_end)       as current_period_end,
    case when canceled_at is not null
         then timestamp_seconds(canceled_at) end as canceled_at
from {{ source('raw_stripe', 'subscriptions') }}

-- models/staging/stripe/stg_stripe__customers.sql
{{ config(materialized='view') }}

select
    id                          as customer_id,
    email,
    name,
    currency,
    timestamp_seconds(created)  as created_at
from {{ source('raw_stripe', 'customers') }}

Warehouse dialect. The timestamp_seconds() above is BigQuery. On Postgres/Redshift use to_timestamp(created); on Snowflake use to_timestamp(created); on DuckDB use to_timestamp(created) or make_timestamp(created * 1000000). If dlt already landed these as timestamps, drop the wrapper entirely. This is the one place warehouse portability actually bites — everything downstream is plain SQL.

Step 4 — The revenue marts

Now the fun part. Each mart is a table (materialized, because BI tools query it repeatedly).

Collected revenue by month

The most-asked-for chart: how much did we actually collect, by month? Paid invoices are the source of truth — a charge can succeed without an invoice, and an invoice can exist without being paid, so we filter to status = 'paid'.

-- models/marts/finance/revenue_by_month.sql
{{ config(materialized='table') }}

select
    timestamp_trunc(period_start, month) as month,  -- BigQuery TIMESTAMP_TRUNC; Postgres: date_trunc('month', period_start)
    currency,
    count(distinct invoice_id)       as invoices_paid,
    count(distinct customer_id)      as paying_customers,
    sum(amount_paid)                 as revenue
from {{ ref('stg_stripe__invoices') }}
where status = 'paid'
group by 1, 2
order by 1

That's recognized, collected revenue — the number finance reconciles against the bank. Group by period_start (the service period) rather than created_at if you want revenue attributed to the month it covers instead of the month it was billed.

Active subscriptions and churn

MRR movement starts with counting subscriptions in each state. This model gives you active counts and a monthly churn signal straight from the top-level subscription fields:

-- models/marts/finance/subscription_movement.sql
{{ config(materialized='table') }}

with subs as (
    select * from {{ ref('stg_stripe__subscriptions') }}
),

started as (
    select timestamp_trunc(created_at, month) as month,
           count(*) as new_subscriptions
    from subs
    group by 1
),

churned as (
    select timestamp_trunc(canceled_at, month) as month,
           count(*) as churned_subscriptions
    from subs
    where canceled_at is not null
    group by 1
)

select
    coalesce(s.month, c.month)                        as month,
    coalesce(s.new_subscriptions, 0)                  as new_subscriptions,
    coalesce(c.churned_subscriptions, 0)              as churned_subscriptions,
    coalesce(s.new_subscriptions, 0)
        - coalesce(c.churned_subscriptions, 0)        as net_new_subscriptions
from started s
full outer join churned c using (month)
order by month

For a point-in-time active count, filter stg_stripe__subscriptions to status in ('active', 'trialing') — that's your live subscriber base right now.

Customer lifetime value

LTV to date is just total collected per customer. Join to customers for the email/name so the dashboard is readable:

-- models/marts/finance/customer_ltv.sql
{{ config(materialized='table') }}

select
    c.customer_id,
    c.email,
    c.created_at                     as signed_up_at,
    count(distinct i.invoice_id)     as invoices_paid,
    sum(i.amount_paid)               as lifetime_revenue,
    min(i.created_at)                as first_payment_at,
    max(i.created_at)                as last_payment_at
from {{ ref('stg_stripe__customers') }} c
left join {{ ref('stg_stripe__invoices') }} i
       on i.customer_id = c.customer_id
      and i.status = 'paid'
group by 1, 2, 3
order by lifetime_revenue desc

True MRR from subscription items (the honest caveat)

Everything above uses top-level, high-confidence columns. Normalized MRR is the one metric where your schema will differ from mine, because a subscription's price lives in a nested array (items.data[]), and dlt flattens nested arrays into a child table — often something like subscriptions__items__data. Open Models and find the real name before writing this model.

Once you've located it, MRR is: for every active subscription item, take unit_amount × quantity, normalize it to a monthly figure by the price's billing interval, and sum:

-- models/marts/finance/mrr.sql  ── adjust the source table name to the one in Models
{{ config(materialized='table') }}

select
    sum(
        (item.unit_amount / 100.0) * item.quantity
        * case item.recurring_interval          -- price's recurring.interval
              when 'month' then 1.0
              when 'year'  then 1.0 / 12
              when 'week'  then 52.0 / 12
              when 'day'   then 365.0 / 12
          end
    ) as mrr
from {{ source('raw_stripe', 'subscription_items') }} item   -- ← your actual child-table name
join {{ ref('stg_stripe__subscriptions') }} s
     on s.subscription_id = item.subscription_id
where s.status in ('active', 'trialing')

If most of your subscriptions carry a single price (typical for simpler SaaS), you can skip the nested table and read the price fields directly off the subscription. Either way, the interval-normalization CASE is the part people get wrong — a yearly plan is not $1,200 of MRR, it's $100.

Step 5 — Test what you built

dbt tests turn "I think this is right" into CI. Add a schema.yml next to your models:

# models/marts/finance/schema.yml
version: 2

models:
  - name: revenue_by_month
    columns:
      - name: revenue
        tests:
          - not_null
  - name: customer_ltv
    columns:
      - name: customer_id
        tests: [not_null, unique]

Run them from the Transformations UI. A failing unique on customer_id means a fan-out bug in your join; a not_null on revenue means an invoice slipped through with a null amount. Catch it here, not in a board deck.

Step 6 — Schedule the whole pipeline

A dashboard is only as fresh as its slowest step. In Datanika, wire the extract and the transform into one dependency chain so transforms run only after the Stripe load succeeds:

Schedule the Stripe upload — hourly for revenue-ops dashboards, or daily at 03:00 if Stripe is one of many batch sources. (See the Scheduling guide for cron syntax and timezones.)
Schedule the transformation pipeline to depend on that upload. Datanika's DAG guarantees dbt won't run against half-loaded data.
Wire failure alerts in Settings → Notifications so a broken run pages your team before finance notices the numbers are stale — here's the Slack alerts walkthrough.

Step 7 — Point a dashboard at the marts

Your marts are just tables in the warehouse now, so any BI tool works. With Metabase (open source, free, runs on the same box if you self-host):

Revenue trend — line chart on revenue_by_month (month × revenue).
MRR — a single big-number tile on mrr.
Net subscriber movement — bar chart on subscription_movement (new vs churned per month).
Top customers — table on customer_ltv sorted by lifetime_revenue.

Because the logic lives in dbt, not in Metabase, every tile shares one definition of "revenue." Swap Metabase for Looker or Superset tomorrow and the numbers don't move.

What you get

Metrics Stripe won't give you — MRR, churn, net revenue retention, LTV, cohorts — defined once in SQL and recomputed every morning.
A full audit trail — dashboard tile → mart → staging → the exact Stripe invoice. No black-box reporting.
Joinable data — Stripe now sits in the same warehouse as your product usage and CRM, so "revenue by feature adoption" becomes one query.
No per-seat transform tax — it's dbt-core, running on your schedule.

What it costs

Datanika meters bytes processed, not rows or connectors. The Free plan includes 10 GB/month. Pro is $79/mo with 100 GB included and $0.50/GB beyond that. A Stripe account's daily refresh is small — Stripe objects are narrow JSON — so for most teams the bill here is the subscription, not the volume. Don't take that on trust, and don't look for the answer in the app: its Plan Usage panel counts model runs, not bytes. Size it from your data instead — the meter counts what an upload writes after normalization.

Compare that to Fivetran, which counts each Stripe object as monthly active rows and adds a per-connection minimum — the exact pricing model this whole tutorial routes around.

Next steps

Connect Stripe → the full setup guide (restricted key, resources, first run).
Stripe → BigQuery use case — the reference architecture for this pipeline.
Transformations guide — models, tests, snapshots, and materializations in Datanika.
Browse all connectors — join Stripe against HubSpot, your product Postgres, Facebook Ads, and more.

Stripe gives you the payments. dbt gives you the metrics. Datanika is the ten-minute bridge between them.

Start free at app.datanika.io

Correction, 2026-09-22. The cost section told you to check your byte usage in the dashboard's Plan Usage panel. That panel counts model runs, not bytes, and no screen in the app shows your byte count today, so the section now says to size it from your data. Tracked in datanika-core#1513.

Your Data Pipelines, Now an MCP Server

Evgenii Timofeev — Mon, 21 Sep 2026 10:36:31 +0000

Datanika now speaks MCP two ways. Paste our hosted endpoint into a client that supports remote MCP servers and approve it in the browser:

https://app.datanika.io/mcp

Or run it locally — datanika-mcp is on PyPI and listed on the official MCP registry as io.datanika/datanika-mcp:

uvx datanika-mcp

Read-only by default on both. Full setup is in the MCP server guide.

What your agent can actually do

The Model Context Protocol is how an LLM client gets typed tools instead of a blob of API documentation and a hope. We expose 25 tools over the Datanika REST API — 17 read-only, plus 8 write tools that stay switched off until you ask for them.

The interesting part isn't the count. It's that the read-only set is genuinely enough to be useful:

Discover and introspect — list connections, read the typed config schema for any connector, list schemas and tables inside a source, preview rows, run a read-only SELECT.
Validate — compile a dbt model, which resolves Jinja, ref() and source() without touching your warehouse. Then preview its output.
Observe — list uploads, pipelines, transformations and runs; fetch a specific run and its logs; browse the data catalog.

That last one changes the texture of debugging. "Why did last night's sync fail?" stops being a tab-switching expedition and becomes a question you ask in the same window where you're already working. The agent pulls the run, reads the logs, cross-references the connection config, and tells you the credential expired.

Read-only by default, and why that matters

The 8 write tools — create a connection, create a model, trigger a run — refuse to execute unless you have deliberately enabled them. How deliberate depends on which path you took:

Over the hosted endpoint, write access is granted at authorization time. When a client asks for write scopes, you approve them in the browser once, and the key that approval mints carries exactly those scopes. A client that asks for nothing gets read-only — silence is never read as consent to write — and a pasted API key stays read-only on that endpoint even if its own scopes would allow writes.

Locally, writes are an explicit opt-in:

uvx datanika-mcp --allow-write

Either way the default is read-only, and enabling writes is a decision someone has to make on purpose rather than a setting that drifts on.

This is not security theatre, but it is also not the whole story, so here is the honest version. Authentication is the real ceiling: the server can't reach another organization or escalate its own scope, query_connection is SELECT-only enforced server-side, and everything an agent does lands in your audit log. The read-only default just means the common case — "help me understand why this pipeline is broken" — needs no write access at all, so you shouldn't grant it.

If you use the local path, use a dedicated API key for agent access rather than reusing one from CI. If an agent does something surprising, you revoke one key instead of untangling which automation broke. Over the hosted path there's no key to manage: consent mints one for you and revoking the grant is the off switch.

What the agent leaves behind

There's a reason we shipped this as tools over dbt and dlt rather than as a natural-language pipeline builder with its own format.

An agent with write access can produce a great deal of work very quickly. If that work lands in a proprietary transformation format, the speed is a liability: you've automated the creation of something only one vendor can run. What create_transformation writes here is a dbt model — SQL with ref() and source(), testable, versionable, and runnable by plain dbt run on a machine that has never heard of Datanika.

We wrote about that trade-off in more detail on the AI agents page. It's the reason the MCP server is deliberately thin: it forwards to the REST API, which orchestrates dlt and dbt-core. There is no clever middle layer accumulating state you can't take with you.

Setting it up

If your client supports remote MCP servers, there is nothing to set up beyond pasting https://app.datanika.io/mcp and approving the consent screen. Everything below is the local path.

Claude Desktop, in claude_desktop_config.json:

{
  "mcpServers": {
    "datanika": {
      "command": "uvx",
      "args": [
        "datanika-mcp",
        "--url", "https://app.datanika.io",
        "--api-key", "etf_your_key_here"
      ]
    }
  }
}

Claude Code is one command:

claude mcp add datanika -- uvx datanika-mcp --api-key etf_your_key_here

Cursor reads ~/.cursor/mcp.json and takes the same shape, or environment variables if you prefer. Self-hosting? Point --url at whatever origin serves /api/v1 — with the default Docker Compose setup that's http://localhost:8000.

Get an API key from Settings → API Keys, then see the full guide for the per-client details, the complete tool table, and troubleshooting.

Where this fits

The MCP server joins the /llms.txt discovery document and the agent guide rather than replacing them. If you're driving Datanika from a script, a CI job, or your own agent framework, the REST API is still the direct path and the discovery documents still tell an LLM everything it needs. MCP is for the case where your agent already lives in a client that speaks it — which, increasingly, it does.

The server is open source under AGPL-3.0, in the datanika-mcp/ directory of the core repo. Issues and PRs welcome.

Datanika is an open-source data pipeline platform built on dlt and dbt-core. Start free with 10 GB/month, or self-host it.

Self-Hosting Datanika with Docker Compose

Evgenii Timofeev — Sun, 20 Sep 2026 21:17:18 +0000

Most data platforms make self-hosting a footnote — a "contact sales for the on-prem option" link that goes nowhere, or an open-source core so stripped-down it's really just a demo for the paid cloud. Datanika's open-source core is the actual product: extract (dlt), transform (dbt-core), scheduling, a visual pipeline builder, every connector, multi-org RBAC, nine languages. It runs from one Docker Compose file, and self-hosting it costs $0 forever — the AGPL-3.0 core has no license key, no seat count, and no GB meter.

This is the walkthrough: what you get, the five-minute quick start, the configuration that actually matters, and — because this is the part most tutorials skip — an honest checklist for running it in production, where you own the pager.

Why self-host at all

The managed version at app.datanika.io exists because plenty of teams would rather not run infrastructure. Self-hosting is the right call when:

Data residency / compliance. Your customer data never leaves your VPC. No third-party processor, no data-processing addendum to negotiate.
Cost at volume. The managed plan meters bytes processed; self-hosted meters nothing. If you're moving terabytes a month, a $12 VPS beats any per-GB bill. (We did that math in The Real Cost of Your Modern Data Stack.)
No vendor lock-in. It's dlt + dbt-core under an open UI. If Datanika vanished tomorrow, your pipelines are standard dlt sources and dbt models — they keep running.
You already have infra. A spare box, a Kubernetes cluster, a managed Postgres — drop Datanika next to them.

The tradeoff is real and we'll be straight about it below: self-hosting means you own upgrades, backups, and the 3 AM page when a disk fills up. For a lot of teams that's a fair trade. For some it isn't.

Prerequisites

Docker Engine 24+ and Docker Compose v2 (docker compose, not the old docker-compose)
4 GB RAM minimum, 8 GB recommended
That's it. Postgres 16 and Redis 7 ship inside the Compose file — you don't install them separately unless you want to bring your own.

The five-minute quick start

git clone https://github.com/datanika-io/datanika-core.git
cd datanika-core
cp .env.example .env
# edit .env — at minimum set SECRET_KEY and ENCRYPTION_KEY (see below)
docker compose up -d

That's the whole thing. Compose pulls the images, starts four containers, and runs the database migrations on first boot. Give it a minute, then open:

http://localhost:3000 — the app (frontend)
http://localhost:8000 — the API

Create your account on the first-run screen and you're in. Your first pipeline — a source, a destination, a run — takes about five more minutes.

What's actually running

Four containers, and it's worth knowing what each one does before you put it in production:

Service	Image	Port	Job
`app`	datanika	3000, 8000	The Reflex app — frontend + API in one process
`celery`	datanika	—	Background worker: runs your extracts, loads, and dbt builds
`postgres`	postgres:16	5432	Application database (your orgs, connections, run history — not your warehouse)
`redis`	redis:7	6379	Task broker for Celery + cache

One thing that trips people up: postgres here is Datanika's own metadata database, not your data warehouse. Your extracted data lands wherever you point it — BigQuery, Snowflake, a separate Postgres, DuckDB on the same box. This container just holds Datanika's bookkeeping.

The configuration that actually matters

Most of .env has sane defaults. Two variables you must set to real random values before anything touches production:

# JWT signing key — anyone who has this can forge login tokens
SECRET_KEY=$(openssl rand -hex 32)

# Fernet key — encrypts every stored connector credential at rest
ENCRYPTION_KEY=$(python -c "from cryptography.fernet import Fernet; print(Fernet.generate_key().decode())")

ENCRYPTION_KEY is the one to guard: it's the Fernet key Datanika uses to encrypt every source and destination credential in the metadata DB. If you lose it, every stored credential becomes unrecoverable and you'll re-enter them all. Back it up somewhere that isn't the same box.

The rest, with their defaults:

Variable	What it's for	Default
`DATABASE_URL`	Metadata Postgres connection	`postgresql+asyncpg://datanika:datanika@postgres:5432/datanika`
`REDIS_URL`	Celery broker	`redis://redis:6379/0`
`SMTP_HOST` / `SMTP_PORT` / `SMTP_USER` / `SMTP_PASSWORD` / `EMAIL_FROM`	Email for run-failure alerts and invites	—
`RECAPTCHA_SITE_KEY` / `RECAPTCHA_SECRET_KEY`	Bot protection on signup (leave empty to disable)	—

Configure SMTP if you want failure notifications — a self-hosted pipeline that fails silently is worse than no pipeline. Everything else can wait.

Migrations and first boot

Migrations run automatically the first time the app container starts. If you ever need to run them by hand — say after pulling a new version — it's:

docker compose exec app alembic upgrade head

Taking it to production (the honest part)

docker compose up -d gets you a working instance. It does not get you a production-grade one. Here's the checklist we'd actually run through, in order of how much it'll hurt to skip:

Put a reverse proxy in front and terminate TLS. Nginx or Caddy (Caddy does automatic Let's Encrypt certs in about four lines). Bind the app containers to 127.0.0.1 and let the proxy be the only thing on :443. Never expose :3000/:8000 to the internet directly.
Back up the metadata Postgres. A nightly pg_dump is the difference between "restore in ten minutes" and "rebuild every connection by hand." Ship it off the box:

   docker compose exec -T postgres pg_dump -U datanika datanika | gzip > datanika-$(date +%F).sql.gz

Back up ENCRYPTION_KEY alongside it — a database dump full of credentials you can no longer decrypt is not a backup.

Lock down the network. Change the default Postgres/Redis passwords, restrict container ports to localhost, and put a firewall (ufw / security group) in front. Redis with no password on a public interface is a classic way to get owned.
Point monitoring at it. The Compose file ships optional Grafana + Prometheus profiles; wire them up or point your existing stack at the app's metrics. You want to know a pipeline is failing before your stakeholders tell you the dashboard is stale — the Slack-alerts setup is the fastest win here.
Right-size the box. 8 GB RAM / 4 vCPU is a comfortable floor for real workloads. Extract jobs are memory-hungry in bursts; Celery is where that shows up.

None of this is Datanika-specific — it's the standard "I now run a stateful service" checklist. But it's real work, and it's the honest cost of the $0 license.

Bring your own database

The bundled Postgres and Redis are convenient for getting started and fine for a small single-box deploy. For anything you care about, point Datanika at managed instances instead — set DATABASE_URL and REDIS_URL to your managed endpoints and the bundled containers become dead weight you can remove from the Compose file. Managed Postgres gets you backups, failover, and point-in-time recovery without you building any of it.

Upgrading

cd datanika-core
git pull origin master
docker compose up -d --build

The app container runs alembic upgrade head on startup, so migrations apply themselves. Pin to a tagged release rather than tracking master if you want change control — and take a pg_dump before every upgrade, because "roll back the database" is a lot easier than "figure out what the half-applied migration did."

Kubernetes, if that's your world

A minimal Helm chart ships in-tree at deploy/helm/datanika/ — same image, one app Deployment, one celery Deployment, optional ingress. A few sharp edges to know before you helm install: it needs a ReadWriteMany storage class (the app and celery pods share a dbt_projects volume), migrations run on every pod start (so keep app.replicaCount=1 until an HA migration hook lands), and you should disable the bundled single-replica Postgres/Redis in favor of managed ones. The self-hosting docs have the full values.yaml walkthrough.

Self-hosted vs. managed — the honest split

Everything in the product is in the open-source core. What you're not getting by self-hosting:

Billing / metering (Paddle integration) — irrelevant unless you're reselling
Managed infrastructure and automatic updates — you run git pull
Priority support with an SLA
SSO (SAML/OIDC) — gated to the Enterprise plan on managed cloud, though the SSO code itself lives in the open-source core

If none of those matter to you, self-hosting isn't a downgrade — it's the same platform on your terms. This blog, and the pipelines behind it, run on exactly this stack.

Where to go next

Self-hosting docs — the canonical reference: full env-var table, Helm values.yaml, production notes.
Getting Started — your first source → destination → run.
Architecture — how dlt, dbt-core, Celery, and Reflex fit together.
Browse all connectors — 31 sources and 9 destinations, all included, no plan gating.
The €12/mo stack, as it stood in April 2026 — the bill for running real software on one small VPS.

Clone it, run docker compose up -d, and you own your data pipeline stack end to end. Or if you'd rather we run it — the managed free tier is one click and includes 10 GB/month.

Four Identical Red Runs: One Was Our Watchdog Working, Three Were Its Corpse

Evgenii Timofeev — Tue, 15 Sep 2026 22:47:57 +0000

A cron in one of our repositories stopped firing on 21 June and nobody noticed for ten weeks. Its only job was to rebuild this site each morning so that blog posts whose publish date had arrived would actually appear. So for ten weeks, scheduled posts did not publish. The failure mode of a cron is silence, and silence is the one thing no dashboard renders.

We built a watchdog for it. The watchdog has four scheduled runs in its history and all four are red.

Here is what makes this worth writing down: the first red was the watchdog working exactly as designed, and the other three were it dying before it checked anything. From the outside they are indistinguishable — not merely the same colour, but the same run conclusion and the same step-by-step breakdown, line for line.

The four runs

33330840430  2026-08-30T19:25:44Z  failure
33442245077  2026-08-31T21:36:52Z  failure
33550257729  2026-09-01T19:34:08Z  failure
33673601655  2026-09-02T19:29:21Z  failure

Expand any of them and you get the same two lines that matter:

step 5  Check every scheduled workflow in both public repos  = success
step 6  File an issue when a schedule has stopped            = failure

That is run 1. It is also run 2, run 3 and run 4. We checked all four against the API rather than trusting the screen; they agree to the character.

On 30 August, step 6 was red because the watchdog had found a stopped cron and was filing an issue about it. The issue exists, machine-authored, timestamped 19:26:02Z, and it is correct in every particular:

datanika-landing :: .github/workflows/daily-rebuild.yml last ran on a schedule event at 2026-06-21T09:54:45+00:00 — 70.4 days ago. Its cron (0 6 * * *) should have fired within 38h. It is active, so this is NOT the 60-day disable; something else is stopping it.

On 31 August, 1 September and 2 September, step 6 was red because the watchdog had crashed in step 5 and there was nothing to report. It verified nothing on any of those nights.

What broke it, and it was not a commit

Nothing in our repository changed between 30 and 31 August. What changed was a repository setting: Dependabot became active on the repo. And when Dependabot is on, GitHub starts listing an extra workflow.

$ gh api repos/OWNER/REPO/actions/workflows --jq '.workflows[].path'
.github/workflows/ci.yml
.github/workflows/deploy-pointer.yml
...
dynamic/dependabot/update-graph

That last one has state: active and a display name of "Dependency Graph". It is not a file. We did not write it, it is not in the tree, and it is not in any branch:

$ gh api "repos/OWNER/REPO/contents/dynamic/dependabot/update-graph?ref=master"
{
  "message": "Not Found",
  "documentation_url": "https://docs.github.com/rest/repos/contents#get-repository-content",
  "status": "404"
}

GitHub synthesises it. The workflows API returns it; the contents API has never heard of it. It is not documented as an exception anywhere we could find, and it appears in both of our public repositories.

Our watchdog walks every workflow the API returns and reads each one's YAML off the default branch to extract its cron:

def _gh(*args: str) -> str:
    return subprocess.run(["gh", *args], check=True, capture_output=True, text=True).stdout

def _parse_crons(repo: str, path: str, ref: str) -> list[str]:
    raw = _gh("api", f"repos/{repo}/contents/{path}?ref={ref}", "--jq", ".content")

check=True turns the 404 into CalledProcessError. Nothing catches it. The script dies mid-collection — and because it collects the repos in order, it died on the first one and never reached the second, which is the repo the watchdog was built to watch.

If you have any tooling that enumerates workflows through the API and then reads their files, run this against your repos now:

gh api repos/OWNER/REPO/actions/workflows \
  --jq '.workflows[] | select(.path | startswith(".github/workflows/") | not) | .path'

Anything it prints will 404 on contents/. dynamic/pages/pages-build-deployment shows up the same way once GitHub Pages is enabled, so this is a family, not a one-off.

The fix is a whitelist, not a dynamic/ blacklist — GitHub only ever executes workflows out of .github/workflows/, so anything outside that path cannot be a workflow you own, and whatever prefix GitHub invents next year is handled without a code change.

It specifically must not become "swallow the 404". A 404 on a real .github/workflows/*.yml means a file you were asked to check is unreadable — a token scope, a rename, an API change — and that has to stay fatal. Turning it into "no crons found" would make the watchdog report health from an error.

The part that cost us three nights

The bug above is a fifteen-minute fix. The reason it survived three nights is a design problem, and it is the transferable half.

Look at the reporting step. Every path through it ends the same way:

if [ ! -f problems.md ]; then
  echo "::error::The watchdog exited non-zero but wrote no problems.md."
  echo "::error::That means it failed to RUN, not that it found a fault."
  exit 1
fi

if [ -n "$existing" ]; then
  gh issue comment "$existing" --body-file /tmp/comment.md
  exit 1
fi

gh issue create --title "Scheduled workflow stopped firing" --body-file /tmp/body.md
exit 1

Three outcomes — filed a new finding, added to an existing finding, could not run at all — collapsed into one signal. The run had to be red for the first two, because that is how a monitor gets your attention. So the third inherited the same colour, and the watchdog's own catastrophic failure was camouflaged by its success case.

We have written before about a green that proves nothing and about alerts that could not fire at all, and about the inverse case, a red that means "I found nothing". This is a fourth shape and it is the meanest of them, because the signal is not useless — it is genuinely informative, one night in four. It just does not carry which thing it means.

A monitor has three states, not two: ran and clean, ran and found something, did not run. If two of those share a colour, the pair that shares it is the pair you will conflate — and you will conflate it in the direction of "working", because that is the reading that requires no action.

The diagnostics were perfect and nobody read them

This is the detail that stings. Open the log of any of the three dead runs and the workflow tells you, in plain English, exactly what happened:

##[error]The watchdog exited non-zero but wrote no problems.md.
##[error]That means it failed to RUN, not that it found a fault.
##[error]Check the step above -- most likely the token cannot
##[error]read one of the repos, or the workflows API changed.

Whoever wrote that anticipated this precise failure and left a message that names the right layer and distinguishes the two cases the conclusion collapses. It was there all three nights. It went unread, because a red tick on a default branch that carries other standing reds is not a thing anyone clicks into.

Diagnostics one level below the signal are not diagnostics. They are an artifact you will find during the postmortem and feel bad about. If the message needs to be read, it has to travel through the same channel as the finding it disambiguates.

So that is what the fix does. The watchdog now files an issue about itself when it cannot run, under a deliberately different title, with a line stating what its own silence is now worth:

While this issue is open, the absence of a "Scheduled workflow stopped firing" issue means nothing: the detector is down, not the crons proven up.

Both paths still exit non-zero — the workflow needs that — but the two conditions are now one glance apart in the issue list instead of one log dive apart.

Twenty-six green tests, none of which touched the broken part

The watchdog had a test suite. Twenty-six tests, green throughout all three dead nights.

Every one of them exercised the comparator: given these crons and these last-run timestamps, which of them are overdue? That logic was never wrong. Not one test exercised collection — the loop that asks the API what workflows exist and fetches each one's file. Collection was the thin shell around gh api that the module's own docstring described as not the part worth testing.

There is a second, quieter version of the same mistake in there. The suite did check that collection found the expected number of workflows — but it asserted a total across both repos. Our first repo has enough workflows on its own to satisfy that total, so the count passed while the second repo was never reached. The fix asserts per repo, which is the assertion that would have failed.

Whenever you decide some part of a program is too thin to test, you have made a claim about where failure lives. Write it down as a claim, because it is one.

The last twist

Update, 15 September 2026. This section describes the morning of 3 September, when it was written, and it stopped being true that evening. The fix reached the default branch that day: the scheduled run that night completed cleanly, and so did the five after it. Since 9 September the watchdog has been red every night again, and not with the crash described here — it runs, and its most recent report names its own failed runs as the schedule that stopped. The original text follows as it was published.

schedule: triggers only ever run the copy of a workflow on the repository's default branch. Our fix is merged to the integration branch and has not been promoted. Tonight's scheduled run will crash again, in exactly the way described above, and there is no branch we could put the fix on that would change that.

Which means the fix cannot be proven by the mechanism it fixes. The only successful run in this watchdog's entire history is a workflow_dispatch we triggered by hand against the fix branch. That proves the code runs. It does not prove the schedule fires, and it does not prove the schedule fires this code.

Our own workflow header had already warned about this trap, from the last time it bit us:

Do not read a green run here as proof that it works. The thing under test is whether an unattended event=schedule run appears at all; a workflow_dispatch proves only that the dispatch step is sound.

The query that answers it honestly filters on the event:

gh api "repos/OWNER/REPO/actions/workflows/NAME.yml/runs?per_page=100" --paginate \
  --jq '.workflow_runs[] | select(.event=="schedule") | .created_at' | head -1

Run that against the cron this whole story started with, and the gap is unmissable: daily schedule runs from 16 April to 21 June, then nothing until 31 August. Seventy-one days. During that window a hand-triggered dispatch went green — which is precisely why a green dispatch was never evidence.

One honest footnote while we are counting. The remedy we applied for that cron was to move it off the top of the hour, on the theory that GitHub's queue is shortest away from :00. Three scheduled runs since: 4h49m, 5h17m and 7h48m late. Three samples is not a refutation, but it is not the improvement we told ourselves we were buying either, and we would rather print that than quietly drop it.

Four things to check in your own repositories

List every workflow whose path is not under .github/workflows/. If anything prints, every tool you have that reads workflow files by path is one API call from a crash.
For each monitor you run, ask what a red means. If "found a problem" and "could not run" share a conclusion, you own one signal doing two jobs, and it will fail in the direction that looks fine.
Give the clean path something to emit. A monitor that is only ever heard from when it has bad news is indistinguishable from one that has died — the useful assertion is on a heartbeat going stale, not on an alarm being absent.
Ask when each scheduled workflow last ran on an event=schedule, not when it last ran. Manual and dispatched runs will happily paper over a cron that has been dead for two months.

Then take the monitor you trust most and break it on purpose — not the system it watches, the monitor itself — and see whether anything looks different.

Datanika is an open-source data platform that runs dlt extract-and-load and dbt-core transformations behind one UI, with scheduling, run history and a REST API. Every workflow quoted here lives in our public repositories, defects included. It is AGPL-3.0 and self-hostable — see the architecture, or browse the connectors.

The Check That Went Red for Doing Its Job

Evgenii Timofeev — Tue, 15 Sep 2026 13:26:07 +0000

We have written twice recently about checks that could not fail — four alerts that could never fire and a nightly suite that passed for eight nights while twelve tests failed. Both are the same defect: a green signal that would have looked identical had the thing it watches been broken.

This one is the mirror image, and it is worse in a way that took us a while to articulate.

One run, three levels, three answers

Here is a single GitHub Actions run from one of our repositories, read three ways.

The run conclusion:

failure

The two jobs in it:

JOB parity        = success
JOB config-fields = failure

The steps inside the failing job:

1 Set up job                                    = success
2 Run actions/checkout@v6                       = success
3 Fetch core's shipped connection schema        = success
4 Compare documented fields against the form    = success
5 File an issue on drift                        = failure      <-- the only red

Every step that does real work succeeded. The one that failed was the step that reports. And here is its entire log:

Already tracked in #449; not filing again.
##[error]Process completed with exit code 1.

The check went red because it found nothing new to say.

The bug is four lines and looks completely reasonable

The workflow compares two catalogues that live in different repositories and files an issue when they disagree. Filing the same issue every morning would be useless, so it suppresses duplicates:

existing=$(gh issue list --state open --search "in:title Connector config field drift" \
  --json number --jq '.[0].number // empty')
if [ -n "$existing" ]; then
  echo "Already tracked in #$existing; not filing again."
  exit 1
fi

Read that in review and nothing jumps out. exit 1 after "we found drift" feels right — drift is a problem, and a red check is how a cron gets anyone's attention.

But this branch is not "we found drift." It is "we found drift, and it is the same drift as yesterday, already filed, already assigned." Nothing changed. Nothing needs doing. And because the issue will be open for as long as it takes to fix a 36-page documentation mismatch, this branch runs every day until then, painting the check red every time.

Why a permanent red is worse than a permanent green

This is the part worth taking away, and it is not symmetric with the "green that proves nothing" story.

A green that proves nothing fails to inform. A red that repeats forever destroys the channel.

A useless green leaves you no worse off than having no check. A permanent red actively trains every human near the repository to stop reading that signal — and the steps above it in the same job are the reason the cron exists. In our case those steps fetch a schema from another repository and parse a data file by regex. Both are exactly the kind of thing that breaks quietly when someone renames a field. The job even has explicit guards for it:

::error::Parsed $slugs slugs out of connectors.ts — the file shape changed.

That guard reports through the channel the duplicate-suppression branch was jamming. Had the file shape actually changed, the message would have arrived on a check that everyone had already learned to ignore, and it would have looked exactly like yesterday.

There is a second cost, which we hit within the hour. The red job sat next to a succeeding one and a stale auto-filed issue titled "Connector count drift." Glanced at, the repository said the connector count was broken. It was not — the counts agree, on both sides, in the workflow's own log, and on the live site. We nearly spent a session re-fixing something that was already fixed.

The line we drew

The distinction that resolves it is not "is there a problem" but "did anything change that needs a human?"

branch	exit	reasoning
filed a new issue	`1`	something changed today; a red run is a standing signal that is easy to see
an issue already exists	`0`	nothing changed, nothing to do, and the open issue is already the tracking mechanism

Note that we did not simply make the cron always green. The exit 1 after a genuine gh issue create stays exactly as it was. A rule that removes the check's ability to ever go red would be the original defect arriving from the other direction, so the test we added asserts the narrow version and asserts that both jobs still exit non-zero after really filing something.

Read the level that answers your question

The same run demonstrates a second thing, and it is free.

Three conclusions were available — run, job, step — and they were failure, success/failure, and one red step among five. Every one is a true statement about a different question. If you ask the run whether your drift check is healthy, the answer is a confident no, for a reason that has nothing to do with drift.

Fetch step-level data whenever the question is about a particular thing:

gh api repos/<owner>/<repo>/actions/runs/<id>/jobs \
  --jq '.jobs[] | {name} + {steps: [.steps[] | {number, name, conclusion}]}'

Two related traps we have hit: a job's own conclusion can be red because of an artifact upload rather than anything it tested, and a fetched log can silently omit steps entirely while the API reports them as success. Read outcomes from the API, and never conclude from an absence.

Three checks you can run on your own repositories

Grep your automation for exit 1 in a de-duplication or "already handled" branch. Anywhere a script's happy path for "nothing new" is an error exit. This includes retry guards, lockfile checks, and "skip because it is already deployed".
List the checks that have been red for more than a week. For each, ask what would happen if a different thing in that job broke tomorrow. If the answer is "the colour would not change", the job has no remaining signal.
For every scheduled workflow, ask what its steady state should be. A cron that watches for drift is supposed to be green almost always and red on the day something moves. If yours is red most days by design, the design is wrong.

One note on writing the test

We pinned the fix with a test that parses the workflow file, and the first run of its mutation harness reported:

M1 :: MUTATION DID NOT APPLY
M2 :: MUTATION DID NOT APPLY
M3 :: MUTATION DID NOT APPLY

The working copy of the workflow is CRLF and the harness's multi-line anchors used bare \n, so they matched nothing. That is worth stating because of what the alternative looks like: a mutation tool that silently changes nothing will report every mutation as "the test caught it", and you will believe you have a proven guard when you have an untested one. Make a harness assert that its mutation actually landed — we compare the file's hash before and after — and treat "did not apply" as a distinct outcome from "passed".

With the anchors corrected, all four mutations went red: both duplicate-suppression branches restored to exit 1, the scope control that deletes a legitimate exit 1, and a positive control that renames the log line the matcher keys on.

Datanika is an open-source data platform — extraction, loading, transformation and scheduling in one UI. The workflows described here are in our public repositories, defects included. Try it free, or read the self-hosting guide.

Four Alerts That Could Never Fire, and How We Found Them

Evgenii Timofeev — Tue, 15 Sep 2026 13:21:05 +0000

We spent a week auditing our own alerting, and the useful question turned out not to be "is anything alerting?" It was "would this alert look any different if the thing it watches had failed?"

For four of them the answer was no. Not "it fires late", not "the threshold is loose" — these could not produce a red signal at all, in the exact circumstance each was written for. Every dashboard was green the entire time, and the green was not evidence of anything.

Here they are with the mechanisms, because each one is a shape you can go and check for in your own stack in about ten minutes.

1. The rule that could only fire while the system was healthy

We wanted an alert for "a scheduled maintenance task has stopped arriving." The natural expression:

increase(celery_tasks_total{task="datanika.run_maintenance"}[2h]) < 1

Read it in English and it is obviously right: fire when fewer than one run happened in two hours. It evaluates cleanly. It shows up in the rule list. Its health reads ok.

It cannot fire.

In PromQL, a bare comparison between a vector and a scalar is a filter, not a boolean. X < 1 does not return true or false. It returns X's own value, for every series where the condition holds, and drops the rest. So the number that reaches the alerting threshold is not 1 and not true — it is whatever increase() computed, which by the definition of the filter is strictly less than 1.

Our rules pair that with a Grafana threshold of gt [0]: fire when the reduced value is greater than zero. Now do the arithmetic for the case the rule exists to catch. A task that has completely stopped arriving has increase(...) == 0. Zero passes the < 1 filter, so a series is produced — and then 0 > 0 is false, so nothing fires.

The rule is live only in the open interval (0, 1). It detects a task that has partially stopped and is structurally blind to one that has entirely stopped.

Demonstrated against the engine rather than argued:

vector(0) < 1        -> series=1 value=0     # gt [0] on 0 is FALSE -> dead rule
vector(0) < bool 1   -> series=1 value=1     # gt [0] on 1 is TRUE  -> fires

bool is the fix: it converts the filter into the 1/0 comparison everyone assumed they were writing.

The part worth stealing is how it was found. The engineer writing that rule used < bool 1 deliberately, and then mutated their own correct code to the naive < 1 to check whether the linter would have caught the mistake had they made it.

It did not. 196 tests passed.

Four other mutations run the same way — a dropped scrape job, a dropped deploy step, a drifted subquery step, a bare staleness comparison — all went correctly red, which is what proves the harness was armed and this one check simply does not look. A satisfiability check existed; it recognised the == N shape and skipped silently on everything else, and a skip is a pass.

2. The exporter that was scraped, and received nothing

Our task metrics were collected by code that had been in the repo since the start:

celery_tasks_total.labels(task=name, status="failure").inc()

There was an alert on that counter. It had never fired. There had also never been a task failure it should have caught, so nobody thought about it.

Two things were wrong, and each on its own is fatal.

The counter incremented in one process and was served from another. celery_tasks_total is incremented inside the Celery worker. /metrics is a route in the web process. Two processes, two prometheus_client registries, no shared state. The counter the worker maintains is not in the payload the web process serves, and never was.

And Prometheus was not scraping Celery at all. Seven targets configured, none of them the worker. So even a correctly located counter had no path to the time series database.

The alert's query returned zero series, forever. And here is the part that makes this a false-negative rather than a visible outage: our alert rules run with noDataState: OK, which is deliberate and correct for filtering expressions — a healthy system genuinely produces no rows. So "the metric does not exist" and "nothing is wrong" arrive at the alert engine as the same signal.

The general form, which we now have written down:

An exporter that is scraped but has stopped receiving events looks exactly like a quiet system.

The fix was not to move the counter. It was to stop asking the application to report on itself and read the broker's own event stream instead, with a dedicated exporter and the worker started with -E. Which introduced its own trap, so it goes in the list too: an exporter pointed at a worker without -E still emits worker-liveness metrics and zero task metrics — indistinguishable from a worker that simply has not run a task yet. You tell them apart by waiting for a scheduled firing and re-querying, never by reading the exporter's own health.

3. `curl -sf` succeeds on an HTML error page

This one is the cheapest to reproduce and probably the most widespread.

A verification runbook, gating a pricing change, contained this step:

curl -sf https://app.example.com/metrics | grep -E "bytes_processed|bytes_quota"

Under it, three checkboxes to tick when the metrics appear. Measured:

http_code=200   size=5106   content_type=text/html;charset=utf-8
<!DOCTYPE html><html lang="en">…

/metrics had no entry in the reverse-proxy config. Our proxy routes a specific list of paths to the backend and sends everything else to the single-page app, so an unrouted backend path does not 404 — it silently resolves to the frontend and returns the app shell with a cheerful 200.

curl -f fails on HTTP status codes at or above 400. A 200 carrying an HTML page is not a status error, so -f does not trigger and curl exits 0. The pipe then hands grep five kilobytes of HTML, grep matches nothing, and the operator sees no output and no error.

The checkboxes under that command could never be ticked, no matter what the application did. And the runbook's own troubleshooting table sent the reader to the wrong layer — "no metrics after 2h → check the worker logs" — for a problem that was one line of proxy config.

Two rules came out of it:

-f is a status check, not a content check. If you are grepping a response, assert on the Content-Type or on a string you know must be present, and fail the step when it is absent. curl -sf … | grep -q 'expected' at least fails on the pipe's own exit status.
Any backend route outside your proxied prefixes needs its own entry, or it becomes your SPA. The failure is silent, returns 200, and is invisible to anything that only checks status codes.

4. The pager that rang for something that had not happened

The other direction belongs in the same audit, because an instrument that manufactures incidents costs you the same credibility as one that hides them — and it burns it faster.

Our end-to-end suite has an auto-filer: on failure it opens a tracker issue and pages. It fires on a job-level failure().

One night it filed and paged for a failed end-to-end run. Reading the step outcomes from the API rather than the log:

step	conclusion
Run gating E2E specs	✅ success
Detect flaky gating specs	✅ success
Run informational E2E specs	✅ success
Assert the specs were actually collected	✅ success
Upload test report	🔴 failure
Telegram alert on failure	fired
File issue on failure	fired

The only error anywhere in the job:

Attempt 1..4 of 5 failed with error: Request timeout:
  /twirp/github.actions.results.api.v1.ArtifactService/CreateArtifact

Every spec passed. The job was red because an artifact upload timed out.

What makes this worse than a cosmetic false alarm is what the filer writes into the issue it creates: a sentence stating that only the gating tier can open this report. That sentence became affirmatively false at the moment it was filed — and it is exactly the line a reader uses to decide how much to trust what they are reading. The tracker asserted a test failure that had not occurred, in a thread whose entire purpose is to be believed.

if: failure() at job level means "any step failed". If you want "the tests failed", key the alert on the test step's own outcome:

- name: Run specs
  id: specs
  run: pytest ...

- name: Page on real failure
  if: steps.specs.outcome == 'failure'

The pattern under all four

Every one of these measured the instrument rather than the system:

what was green	what it actually recorded
the alert rule's health	that the expression parses and evaluates
the scrape target being `up`	that an HTTP endpoint answered, not that it carried our data
`curl -sf` exiting 0	that some server returned a status below 400
the E2E job's red	that some step in the job failed

None of them is broken in the sense of throwing an error. Each answers a real question accurately. The question is just not the one anybody thought was being asked.

So the check we now run on every new alarm is a single sentence:

Would this signal look different if the thing it watches had failed?

And the only honest way to answer it is to make the thing fail. Not in a synthetic fixture built to satisfy the assertion — against the real artifact, in the real failure mode. Break the rule and watch it go red. Stop the worker and watch the metric vanish, then check what your noDataState does with a vanished series. Point the curl at a path you know is unrouted and confirm the step fails.

A passing check is not evidence until you have seen it fail. We keep relearning that in a new costume roughly once a fortnight — most recently as a nightly CI job that reported success over twelve failing tests for eight nights, because a pipe to tee discarded the exit code and GitHub Actions' default shell does not set pipefail. Same family, different layer.

Four things to go and check in your own stack

Grep your alert expressions for a bare < or > against a scalar, and check what value actually reaches your threshold evaluator. In PromQL, add bool unless you specifically want the filter.
For every counter you alert on, confirm the process that increments it is the process that serves it. With any pre-fork or multi-process server, that is not automatic.
Find every curl -sf in your runbooks and CI, and make each one assert on content, not just status. Then point one at a deliberately wrong path and confirm it fails.
Read your alert conditions for scope. if: failure() and on: failure are usually broader than the thing you meant.

Then take one working alert and break the underlying system on purpose. If nothing turns red, you have found the fifth one.

Datanika is an open-source data platform that runs dlt extract-and-load and dbt-core transformations behind one UI, with scheduling, run history and a REST API. It is AGPL-3.0 and self-hostable with a single docker compose up — see the architecture, or browse the connectors.

Our Nightly CI Passed for Eight Nights While Twelve Tests Failed

Evgenii Timofeev — Wed, 09 Sep 2026 13:40:46 +0000

Our nightly connector smoke suite reported success eight nights running. Here is what it actually printed on each of those nights:

12 failed, 9 passed, 4 warnings in 44.01s

Twelve of twenty-one probes failing, every night, for at least eight consecutive nights — the log retention window is the only reason the count stops at eight. The step reported completed/success. The job reported success. The workflow reported success. The Telegram alert on failure step, guarded by if: failure(), was skipped every single night, because failure() never became true.

Nobody had broken anything that week. The suite had been lying since long before.

The mechanism is four words of shell

Here is the step, near enough verbatim:

- name: Run connector smoke tests
  run: |
    pytest tests/test_connector_smoke/ -v --tb=short -rs | tee /tmp/smoke.log
    if grep -qE '[0-9]+ skipped' /tmp/smoke.log; then
      echo "::error::Smoke probes were SKIPPED"
      exit 1
    fi

A shell pipeline's exit status is the exit status of its last command. The last command here is tee, and tee succeeded — it wrote the file it was asked to write. pytest exited non-zero into a void.

That much is ordinary POSIX shell, and most people who have written a CI pipeline know it. The part that catches you is the second half.

GitHub Actions gives you `pipefail` only if you ask for it, and asking looks like a no-op

Bash has a flag for exactly this. set -o pipefail makes a pipeline return the rightmost non-zero exit status, so pytest ... | tee fails when pytest fails.

GitHub Actions runs run: steps on Linux and macOS with a default shell of:

bash -e {0}

-e but no -o pipefail. Now write the step as:

- name: Run connector smoke tests
  shell: bash
  run: |
    ...

and the runner invokes:

bash --noprofile --norc -eo pipefail {0}

Naming the shell you were already using turns on pipefail. That is a real asymmetry in the product, it is documented, and it reads as a no-op in a diff. A reviewer looking at shell: bash on a step that was already running bash sees tidying, not a behavioural change — which cuts both ways: it is easy to add without argument, and easy for someone to delete later as noise.

We had no shell:, no defaults: and no pipefail anywhere in that workflow file. So the only thing left that could fail the step was the grep.

The part actually worth writing down

The grep is not a mistake. It exists for a good reason and it was working.

Our connector probes need live credentials. A probe with no credentials, or one whose client library will not import, used to skip — and a suite that skips everything passes loudly while testing nothing. So we changed conftest.py to convert both cases from skip into fail, on the reasoning that a healthy run should have zero skips, which makes "any skip at all" a usable alarm. Then we added the grep to enforce it.

Read those two decisions in isolation and both are right. Read them together and this falls out:

the outcome the guard watches for — skip — is now the rare one, by construction;
the outcome that actually happens — fail — is the one the pipe throws away.

The guard was aimed at the hole that existed before the conftest change. The conftest change moved every real failure into the blind spot. No commit introduced the bug; the second correct change walked the failure mode into the first correct change's shadow.

That is the shape to look for, and it is not rare. When you tighten one behaviour, the checks written against the old behaviour do not fail — they go quiet, and quiet and healthy look identical from outside.

What was underneath

The twelve failures were all real, and none was caused by this bug. A couple of trial accounts had lapsed and were returning 401 and 403. A message-broker probe was pointed at a cluster that no longer matched the one our credentials were minted against. Several probes were missing environment variables entirely, because the credential bundle handed to CI had drifted from the copy on disk — two copies of the same list, edited independently, which is its own recurring lesson.

Every one of those is a five-minute fix once you can see it. They had been invisible for over a week, and the instrument that was supposed to show them was reporting green the whole time.

Fixing it

Any one of these is sufficient. They are not equivalent.

1. Name the shell. The smallest diff, and it fixes every pipeline in the step at once:

- name: Run connector smoke tests
  shell: bash
  run: |
    pytest tests/test_connector_smoke/ -v --tb=short -rs | tee /tmp/smoke.log

2. Set the flag explicitly. More obvious to a reader, and it survives someone deleting shell: bash as redundant:

run: |
  set -o pipefail
  pytest ... | tee /tmp/smoke.log

3. Capture the status you care about. Verbose, but it is the only form that lets you act on pytest's code specifically — useful when a tool distinguishes "tests failed" (1) from "the run was misconfigured" (2, 4):

pytest ... | tee /tmp/smoke.log
rc=${PIPESTATUS[0]}

4. Stop piping. Write the log to a file and cat it afterwards, or emit machine-readable output and read that:

pytest ... --junitxml=/tmp/smoke.xml | tee /tmp/smoke.log

Keep the grep, in every case. It still catches the case it was written for. It just must not be the only thing that can fail the step.

Do not trust the green afterwards

This is the step that gets skipped, and it is the one that matters.

Once you have applied the fix, re-run the job unchanged, against the still-broken system, and confirm it goes red. If it goes green, your fix did not land — a check that has never been observed failing has never been shown able to fail. Our own rule, from a file of rules that each cost us an incident, is blunt about it: a passing check is not evidence until you have seen it fail.

We have paid for that rule more than once. A monthly database restore drill asserted that a table of seed rows survived the restore — while pg_dump writes tables alphabetically and puts users at the very end of the file, which makes it the first thing a truncation destroys and the last thing that check would notice. It printed PASS beside an empty user table. Alert rules that were structurally unable to fire. A test suite that mocked the very unit under test. In an earlier post we wrote up 2,300 passing tests sitting beside a CSV import that loaded exactly one row.

The through-line is one question, and it is worth asking of any signal before you rely on it:

Would this look different if the thing it watches had failed?

If you cannot answer that from the code, the green tells you nothing.

Audit your own workflows in one command

If you pipe test output anywhere in CI — to tee, grep, head, jq, a log shipper — check whether your exit code survives:

for f in .github/workflows/*.yml; do
  grep -q 'pipefail' "$f" && continue
  grep -q 'shell: bash' "$f" && continue
  grep -Hn -E '\|\s*(tee|grep|head|tail|jq|awk|sed|sort|uniq)' "$f"
done

Every line it prints is a pipeline whose left-hand exit code is being discarded. Some of those are deliberate. The ones that are not are the ones running your tests.

Two things worth knowing while you read the output:

set -e does not save you. -e aborts on a failing command; a pipeline whose last command succeeded has not failed, so there is nothing for -e to abort on. Having -e is what makes this feel safe when it is not.
Windows runners differ. The default shell there is PowerShell, which has its own rules for $LASTEXITCODE and native-command failure. A fix applied to a Linux matrix leg does not necessarily apply to a Windows one.

Alerting on the pipelines you actually run

The uncomfortable part of this story is not that a workflow was misconfigured. It is that we had an alert wired up, pointed at the right workflow, with the right condition — and it was silent for eight nights because the condition it tested was never reached. The alert was fine. The thing it was reading was wrong.

That generalises past CI. If you run scheduled data pipelines, the same question applies to every notification you have configured: does your failure alert read the pipeline's real outcome, or something downstream of a step that always succeeds?

Datanika records the outcome of every run in its own ledger rather than inferring it from a log line, and failure notifications fire off that record. If you want them in Slack, that setup takes about two minutes. If you drive pipelines from CI, the REST API hands back a run id and a status you can poll and assert on — which is a better thing to gate a deploy on than an exit code you have to hope survived a pipe.

The platform is open source and self-hostable. Start here, or read the docs.

Triggering Data Pipelines from CI/CD via the REST API

Evgenii Timofeev — Mon, 07 Sep 2026 12:44:43 +0000

The most common thing people want from a data platform's API is boring: run this pipeline, tell me if it worked. Usually right after a deploy, a dbt seed change, or a nightly job that has to finish before something else starts.

Here's how to do that with Datanika — and, more usefully, the one place this is easy to get wrong in a way your CI won't notice. We got it wrong ourselves; the fix is at the end of that section.

The short version

curl -sS --fail-with-body -X POST \
  "https://app.datanika.io/api/v1/pipelines/1/run?wait=true&timeout=300" \
  -H "Authorization: Bearer $DATANIKA_API_KEY"

One call. Blocks until the pipeline finishes, returns the run as JSON, and exits non-zero if the pipeline failed. In a set -e CI step, that is the whole integration.

The reason that last clause is worth a sentence is the rest of this section.

The status code carries the run's outcome

With ?wait=true, the endpoint polls the run until it reaches a terminal state, then answers with a status code that describes the run, not just the request:

What happened	Status
The run finished successfully	200
Still pending or running when your timeout expired	408
Terminal, but not successful (`failed`, `cancelled`)	422

The body is the serialized run in all three cases, so status and error_message are still there when you want detail:

{
  "id": 43,
  "target_type": "pipeline",
  "target_id": 1,
  "status": "failed",
  "started_at": "2026-08-30T14:00:02Z",
  "finished_at": "2026-08-30T14:00:44Z",
  "rows_loaded": 0,
  "error_message": "relation \"public.orders\" does not exist",
  "created_at": "2026-08-30T14:00:00Z"
}

That response is a 422. curl --fail trips on it, raise_for_status() raises on it, and your CI job goes red — which is what you wanted when you asked the API to wait.

The trap this replaced, because you will meet it elsewhere

Until August 2026, that same failed run came back as HTTP 200.

The reasoning was defensible and wrong. Your request succeeded — the server did its job, found the run, waited, serialized it, and handed it to you. The run is what failed, and the body says so plainly in status. Textbook HTTP.

The problem is that nothing in CI reads the body:

# The reflex every CI script has
curl --fail -X POST ".../pipelines/1/run?wait=true" -H "Authorization: Bearer $KEY"

--fail trips on 4xx and 5xx. A failed pipeline was a 200. Exit code 0. The job goes green, the dashboard downstream is stale, and nobody finds out until someone notices the numbers are yesterday's. A step that reports success on a failed load is worse than no step at all, because it converts a loud failure into a silent one.

What settled it was noticing the endpoint had already decided the question. It returned 408 when the run was still going at the timeout — and that isn't a transport failure either; the request was served perfectly. So 200 == failed wasn't a competing philosophy, it was an inconsistency with the endpoint's own behaviour. ?wait=true is the caller explicitly opting into "block until you know the outcome." If the outcome doesn't reach the status line, the option is half-built.

So if you are integrating some other pipeline API and it offers a "wait" mode: check what a failed run returns before you trust --fail. Trigger something you know is broken and look at the exit code. It is a two-minute experiment that this post exists because we ran late.

Why 422 and not 500

A 5xx means "our API broke, retry the request." Retrying this request would start a second pipeline run — a second extract, a second load, a second set of rows. A failed pipeline is not a transport failure and must not be retried like one.

422 says the opposite: the request was fine, and the thing you asked about did not succeed. The failure is in your pipeline — your credentials, your SQL, your source — not in our server. Retrying blindly is exactly the wrong move, and the status code should say so.

cancelled is a 422 as well. The check is "not success" rather than a list of failure names, so a terminal status added later cannot quietly rejoin the success branch.

Keep the body when you fail

Plain curl --fail discards the response body on an HTTP error, which throws away error_message — the one thing you want in the log.

Use --fail-with-body (curl 7.76+, so every current GitHub runner) to get the non-zero exit and the body. If you're on something older, capture the status code explicitly, as the full workflow below does.

The three endpoints

Runs are triggered on the resource, not on a runs collection. There is no POST /api/v1/runs.

What you're running	Endpoint	Key scope needed
A pipeline	`POST /api/v1/pipelines/{id}/run`	`pipelines:write`
A file upload	`POST /api/v1/uploads/{id}/run`	`uploads:write`
A dbt transformation	`POST /api/v1/transformations/{id}/run`	`transformations:write`

All three behave identically with respect to everything below.

Two modes: fire-and-forget, or wait

Without wait you get an immediate 202 Accepted:

{"run_id": 43, "status": "pending"}

Use this when CI's job is to kick off work — a nightly ingest that takes 40 minutes and nothing downstream is blocking on it. A 202 is about dispatch only; it says nothing about the outcome, and there is nothing to check with --fail.

With ?wait=true the request blocks until the run is terminal, then answers 200 / 408 / 422 as above.

timeout is in seconds, defaults to 120, and is clamped to 1–300. Passing timeout=3600 doesn't get you an hour — you get 300 seconds, then a 408. Status is polled every 2 seconds, and waiting doesn't occupy a worker, so a waiting request costs you nothing but the open connection.

A 408 is not a failure. The run is still going; you just stopped waiting. The body carries "timed_out": true, and the run's own status is still pending or running. That distinction matters because the remedy is different — a 422 means fix your pipeline, a 408 means wait longer or stop blocking on it. Treating them the same is how a slow Tuesday becomes a red build.

Runs longer than five minutes

Because of that 300-second ceiling, anything longer needs the async shape: trigger, then poll.

run_id=$(curl -sS --fail-with-body -X POST \
  "https://app.datanika.io/api/v1/pipelines/1/run" \
  -H "Authorization: Bearer $DATANIKA_API_KEY" | jq -r '.run_id')

deadline=$(( $(date +%s) + 3600 ))
while [ "$(date +%s)" -lt "$deadline" ]; do
  run=$(curl -sS "https://app.datanika.io/api/v1/runs/$run_id" \
    -H "Authorization: Bearer $DATANIKA_API_KEY")
  status=$(jq -r '.status' <<<"$run")
  case "$status" in
    success)             echo "done"; exit 0 ;;
    failed|cancelled)    jq -r '.error_message // .status' <<<"$run"; exit 1 ;;
  esac
  sleep 15
done

echo "Still running after 1h — check run $run_id in the app"
exit 1

GET /api/v1/runs/{id} is a plain read: it returns 200 with the run whatever its status, because here you are asking about the row rather than asking "did it work?". The .status field is the answer in this shape.

Note the case covers every terminal status, not just success. A loop that only watches for success runs until your CI timeout and then reports the wrong cause.

If you want the run's output while debugging, GET /api/v1/runs/{id}/logs returns {"run_id": …, "logs": "…"}.

Idempotent retries: the header that stops double-runs

CI reruns. Someone clicks "Re-run failed jobs", a runner gets evicted mid-step, a network blip makes curl retry. Any of those can fire the same trigger twice — and by default, twice means two runs, two loads, and two sets of rows.

Every POST endpoint accepts an optional Idempotency-Key header. Replay the same key and you get the original response back instead of a second run. Keys are cached for 24 hours, and it's opt-in — no header, no deduplication.

The natural key in GitHub Actions is the run identity itself:

curl -sS --fail-with-body -X POST \
  "https://app.datanika.io/api/v1/pipelines/1/run?wait=true&timeout=300" \
  -H "Authorization: Bearer $DATANIKA_API_KEY" \
  -H "Idempotency-Key: gha-${GITHUB_RUN_ID}-${GITHUB_RUN_ATTEMPT}"

Include GITHUB_RUN_ATTEMPT if a manual re-run should start a fresh pipeline, and leave it out if it shouldn't. That's a real decision, not a formality — decide it deliberately.

A complete GitHub Actions job

name: Refresh analytics

on:
  push:
    branches: [main]
    paths: ["dbt/**"]
  workflow_dispatch:

jobs:
  refresh:
    runs-on: ubuntu-latest
    steps:
      - name: Trigger the Datanika pipeline
        env:
          DATANIKA_API_KEY: ${{ secrets.DATANIKA_API_KEY }}
        run: |
          set -euo pipefail

          response=$(curl -sS -X POST \
            "https://app.datanika.io/api/v1/pipelines/1/run?wait=true&timeout=300" \
            -H "Authorization: Bearer $DATANIKA_API_KEY" \
            -H "Idempotency-Key: gha-${GITHUB_RUN_ID}-${GITHUB_RUN_ATTEMPT}" \
            -w "\n%{http_code}")

          code=$(tail -n1 <<<"$response")
          body=$(sed '$d' <<<"$response")

          case "$code" in
            200) echo "Loaded $(jq -r '.rows_loaded' <<<"$body") rows" ;;
            408) echo "::warning::Still running after 300s — run $(jq -r '.id' <<<"$body")"
                 exit 1 ;;
            422) echo "::error::Pipeline run $(jq -r '.status' <<<"$body") — $(jq -r '.error_message // "no message"' <<<"$body")"
                 exit 1 ;;
            *)   echo "::error::API returned $code"; echo "$body"; exit 1 ;;
          esac

Four branches, because there are four genuinely different situations: it worked, it's still going, your pipeline broke, or our API did. --fail-with-body collapses the middle two into one non-zero exit, which is fine when you only need pass/fail — but the messages above are what you'll want at 2am, and the 408 branch is the one people most often want to handle differently.

Cancelling a run

Update, 15 September 2026. One clause in this section is no longer true. A run cancelled while it is running now keeps the status cancelled when its task finishes; it is no longer overwritten back to success. Everything else here still holds: cancelling does not stop the worker, the extract and the load run to the end, and the run is billed for everything it processed. The original text follows as it was published.

If your workflow is cancelled, the pipeline it started is not. A 202 handed the work to a background worker, and CI walking away doesn't reach it.

And the API cannot currently clean it up for you — plan around that rather than around the endpoint. POST /api/v1/runs/{id}/cancel exists and will return 200 with "status": "cancelled" — but that call only writes the status onto the run row. It does not stop the worker. The extract keeps reading, the load keeps writing, and when the task finishes it overwrites the status back to success. Tracked as core#657, along with the reason there is no cancel button in the app either: shipping a control onto that behaviour would spread the wrong impression to every user instead of only to API callers.

So don't wire a cleanup step and believe it. Until cancellation actually stops work:

Bound the blast radius instead of the run. Smaller, more frequent pipelines beat one long job you might want to kill.
Assume anything you triggered will finish. Check the app rather than your CI log for what a cancelled workflow left behind.
On usage-based plans, a run you "cancelled" keeps metering until it completes on its own. That's the practical reason this is a limitation worth stating rather than a detail.

The one accurate part of the old advice: the endpoint returns 409 with "not_cancellable" when the run has already finished, which is a perfectly normal outcome and not something to treat as an error.

Give CI its own key, scoped down

API keys are created in Settings → API Keys in the app, carry the etf_ prefix, and are shown once. They're hashed with SHA-256 before storage, so a lost key can't be recovered — you create a new one and revoke the old. Full details on the API keys page.

For a CI key, set the scopes explicitly rather than leaving them empty (empty means full access):

pipelines:write — to trigger
runs:read — to poll status and read logs

That's a key that can start and observe one kind of work and cannot delete a connection, read your credentials, or create a schedule. If it leaks into a build log, the blast radius is a pipeline someone can already trigger from the UI.

Set an expiry on it too, and rotate it on a calendar rather than after an incident.

Rate limits

Each key is rate-limited independently, per plan, and exceeding it returns 429 with a Retry-After header. Current per-plan limits are in the API reference — a fan-out matrix build that triggers one pipeline per shard is the realistic way to hit them, so back off on 429 rather than retrying immediately.

The polling loop above is well inside every tier: at sleep 15 it costs 4 requests a minute, and ?wait=true polls server-side, so a 300-second wait is one request against your budget, not 150.

Note that 429 is a genuine transport-level "try again", unlike the 422 above. It's the one 4xx here you should retry.

Why this is easier here than in a three-tool stack

The reason this post is short is architectural. In a Fivetran + dbt Cloud + Airflow stack, "run the pipeline and tell me if it worked" is three APIs, three auth schemes, three status vocabularies, and a decision about which failure counts. Here, extract, load, and transform are the same run object with one status field, so CI asks one question once.

That argument, with numbers attached, is in Datanika vs the Modern Data Stack.

What this post does not cover

Scheduling. If you want a pipeline to run nightly, use a schedule instead of a cron-triggered CI job. Cron in CI gives you the worst of both — an extra dependency and no visibility in the app.
Webhooks back into CI. There is no outbound "run finished" callback into a workflow today; poll, or use a notification channel to tell a human.
Every endpoint. The API reference has the full surface, including connections, schedules, and bulk import.

Every status code, header, scope name and default here was read out of the live OpenAPI document and the shipped route handlers, not out of our own docs. That matters more than usual for this post: the 200-on-failure behaviour in "the trap this replaced" was **found while writing the first draft of it, filed as core#663, and fixed before this went out. Writing the tutorial is what surfaced the bug — so if you find a discrepancy, open an issue and we'll fix the post, or the API.

DEV Community: Evgenii Timofeev

Google Sheets to Your Warehouse: the Share Step Is the One That Fails

The setup, briefly

The step that fails

The button that says "not tested"

Configure the upload

The first run is the actual test

Put it on a cron

The one-line version

A Customer 360 from HubSpot and Stripe: the Join Is the Work

What the two systems actually give you

Step 1 — Normalize the identifiers before you join anything

Step 2 — Three joins, in descending order of trust

Tier A — an explicit key in Stripe metadata (exact)

Tier B — normalized email (good, and incomplete)

Tier C — email domain → company domain (recovers real customers, over-matches on public domains)

Step 3 — Measure the match rate; don't assume it

Step 4 — Now the 360 is boring, which is the goal

Step 5 — Ship the unmatched list to the humans

What this does not do

Wire it up

Next steps

We're Adding a Volume Dimension to Our Pricing. Here's the Math, and Here's Why.

What changed

Why "volume" was missing

The math that broke it

Why GB and not MAR

How we meter honestly

What a month costs

What stays

What we'll do if we got the numbers wrong

What this doesn't mean

The numbers, measured

How We Built a 5.8x Faster Data Pipeline with DLT + Arrow

The three generations

v1: Custom Airflow operators (99 min for 54M rows)

v2: DLT + JSON normalization (174 min — slower)

v3: DLT + Arrow mode (17 min — 5.8x faster)

The numbers

MySQL: user_recipe_rel (54M rows, production)

MongoDB: hostedImages (8.6M documents, dev)

The file_max_bytes war story

Why this matters for Datanika

Key takeaways

How to Build a Stripe Revenue Dashboard with dbt

What we're building

Step 1 — Land Stripe in your warehouse

Step 2 — Declare the source

Step 3 — Staging models: clean and cast

Step 4 — The revenue marts

Collected revenue by month

Active subscriptions and churn

Customer lifetime value

True MRR from subscription items (the honest caveat)

Step 5 — Test what you built

Step 6 — Schedule the whole pipeline

Step 7 — Point a dashboard at the marts

What you get

What it costs

Next steps

Your Data Pipelines, Now an MCP Server

What your agent can actually do

Read-only by default, and why that matters

What the agent leaves behind

Setting it up

Where this fits

Self-Hosting Datanika with Docker Compose

Why self-host at all

Prerequisites

The five-minute quick start

What's actually running

The configuration that actually matters

Migrations and first boot

Taking it to production (the honest part)

Bring your own database

Upgrading

Kubernetes, if that's your world

Self-hosted vs. managed — the honest split

Where to go next

Four Identical Red Runs: One Was Our Watchdog Working, Three Were Its Corpse

Tier A — an explicit key in Stripe `metadata` (exact)

MySQL: `user_recipe_rel` (54M rows, production)

MongoDB: `hostedImages` (8.6M documents, dev)

The `file_max_bytes` war story

3. `curl -sf` succeeds on an HTML error page

GitHub Actions gives you `pipefail` only if you ask for it, and asking looks like a no-op