DEV Community: Evgenii Timofeev

Introducing the Datanika REST API v1

Evgenii Timofeev — Mon, 31 Aug 2026 14:46:52 +0000

Your Pipelines, Programmable

Today we're shipping the Datanika REST API v1 — 36 endpoints that give you full programmatic control over your data pipelines.

Everything you can do in the UI, you can now do via HTTP: create connections, trigger pipeline runs, query run history, manage schedules, and configure notification channels.

What's Included

Full CRUD for Every Resource

Resource	Endpoints
Connections	List, Get, Create, Update, Delete, Test
Uploads	List, Get, Create, Update, Delete, Run
Pipelines	List, Get, Create, Update, Delete, Run
Transformations	List, Get, Create, Update, Delete, Run
Schedules	List, Get, Create, Update, Delete
Runs	List, Get, Logs
Notification Channels	List, Get, Create, Update, Delete

Authentication

API keys use Bearer token authentication. Create a key in Settings > API Keys, then include it in your requests:

curl -H "Authorization: Bearer etf_your_key_here" \
     https://app.datanika.io/api/v1/connections

Keys are scoped — you can limit a key to read-only access, run execution only, or specific resource types. See the full scope reference in our docs.

Rate Limiting

Requests are counted per API key in a fixed 60-second window:

Plan	Requests/minute
Free	30
Pro	120
Enterprise	300

Per key, not per organization — two keys get two independent budgets. A per-second burst ceiling runs alongside the per-minute limit; it is the same on every plan, so it isn't a tier dimension.

Rate limit headers (X-RateLimit-Remaining, Retry-After) are included in every response so your integrations can handle throttling gracefully.

Self-hosted users can configure limits via API_RATE_LIMIT_RPM and API_RATE_LIMIT_BURST environment variables.

Corrected 2026-08-31. This section originally published a per-plan burst column (5 / 15 / 30 requests per second) and described the window as sliding. Neither was true of the shipped code: the burst ceiling has always been a single setting applied identically to every plan, and the window is fixed rather than sliding. The per-minute numbers are unchanged. See the API reference for the current statement.

Interactive Docs

The API ships with an OpenAPI 3.0 spec and interactive Swagger UI at /api/v1/docs. Try endpoints directly from your browser — no Postman needed.

Use Cases

CI/CD integration — Trigger a pipeline run after your data model tests pass:

curl -X POST https://app.datanika.io/api/v1/pipelines/1/run \
     -H "Authorization: Bearer etf_ci_key"

Custom monitoring — Poll run status from your own alerting system:

curl https://app.datanika.io/api/v1/runs?status=failed \
     -H "Authorization: Bearer etf_monitor_key"

Bulk setup — Provision pipelines from a shell script or your infra automation. Every write endpoint is exposed and supports the Idempotency-Key header, so re-running the script is safe. POST each resource in turn — connections first, then the upload or pipeline that references them, then a schedule:

curl -X POST https://app.datanika.io/api/v1/connections \
     -H "Authorization: Bearer etf_admin_key" \
     -H "Content-Type: application/json" \
     -H "Idempotency-Key: bootstrap-2026-04-14-src" \
     -d @source-postgres.json

Available write endpoints: /connections, /uploads, /pipelines, /transformations, /schedules, /notifications/channels. Keep the JSON bodies in version control next to your infra code.

Scheduled reporting — Fetch run history for weekly reports:

curl "https://app.datanika.io/api/v1/runs?target_type=pipeline&limit=100" \
     -H "Authorization: Bearer etf_report_key"

What's Next

We're working on webhook triggers (run your pipeline when an external event fires) and a CLI tool built on top of the API. If you have feedback or feature requests, open an issue on GitHub.

Try It

The API is available now on all plans, including Free. Create an API key and start automating.

Full documentation: datanika.io/api/reference

API Keys — create and manage API keys
Self-Hosting Guide — the API works on self-hosted deployments too
36 Connectors — everything you can manage via the API
Pipeline Templates — prefilled starting points your API-driven workflows can clone: Stripe → Postgres, Postgres → BigQuery, CSV → DuckDB

I Added 32 Connectors to My Data Platform. Most Took Less Than a Day Each.

Evgenii Timofeev — Mon, 31 Aug 2026 14:41:51 +0000

When I started, Datanika had 4 database connectors. Now it has 32: 27 sources, 11 destinations, and 11 dbt adapters. The secret is that I'm not writing connectors from scratch.

The Architecture

For databases (Postgres, MySQL, MSSQL, ClickHouse, DuckDB, MongoDB), there's a dlt source adapter per type. Each one maps connection credentials to dlt's config format and handles type-specific quirks. The actual extraction is dlt's job — I'm just wiring up the UI and storing config in the database.

For SaaS sources (Stripe, GitHub, HubSpot, Salesforce, Shopify, Jira, Slack, Google Analytics, Facebook Ads, and more), I use dlt's REST API source or its verified sources. Each connector is really just a config: base URL, auth method, and a list of default endpoints. Stripe, for example, ships with 6 endpoints pre-configured (customers, charges, invoices, subscriptions, products, prices).

For destinations (BigQuery, Snowflake, Redshift, Databricks, Azure Synapse, ClickHouse, DuckDB), it's dlt destination objects plus matching dbt adapters.

The Hard Ones

The hardest connectors weren't the obvious ones. Two stand out:

ClickHouse has three different table engine modes — standalone MergeTree, clustered ReplicatedMergeTree, and ClickHouse Cloud. Each requires different DDL and a different mental model for replication.
MongoDB needed a custom dlt source because the standard one didn't handle nested documents the way I wanted. Flattening nested arrays into relational tables for analytics is more opinionated than it sounds.

Everything else was mostly: read the API docs, fill in the dlt config schema, write 5–10 tests, ship it.

Phased Rollout

I rolled the SaaS connectors out in three phases over two days, not all at once:

Phase A: 4 connectors (the highest-demand ones)
Phase B: 7 more
Phase C: 7 more

Each phase was a single commit with tests. This kept reviewable diffs small and let me catch architecture issues early. If phase A broke something subtle in the connection model, phase B would have surfaced it before I shipped 14 more connectors on top.

The Lesson

Pick tools that are themselves extensible. dlt's plugin architecture meant "add a connector" was mostly config, not code. If I'd built extraction from scratch, I'd still be on connector #4.

This is also why Datanika uses dbt-core for transformations rather than rolling our own SQL execution engine. The ecosystem has solved problems I don't need to re-solve — packages, tests, snapshots, materializations, lineage. Building on it gives users 100% dbt compatibility for free.

What's Next

I'm thinking about adding Webhooks (push-based ingestion) and a generic "S3-compatible" source that works with R2, MinIO, Backblaze, etc. — anything that speaks the S3 API.

What connectors would you want to see next? Open an issue on GitHub or try the existing 32 at datanika.io/connectors.

Try It

I Built an ETL Platform Solo. 18 Phases, 1,400+ Tests, One Month.

Evgenii Timofeev — Mon, 31 Aug 2026 14:36:49 +0000

Most ETL tools are built by teams of 20+. I built one by myself in about a month.

I'm building Datanika — a platform for managing data pipelines. Extract with dlt, transform with dbt-core, orchestrate with Celery and APScheduler, all wrapped in a Python UI using Reflex.

Scope, Not Speed

The key decision was to not build everything at once. I broke the entire project into 18 phases, each small enough to finish in 1–3 days:

Phase 1: Models and auth
Phase 2: Connections and pipeline config
Phase 3: dbt integration
Phase 4: Scheduling and dependency DAGs
Phase 5: Polish, dashboard, RBAC
... and so on

Each phase had a clear deliverable and a short feedback loop. If I couldn't finish a phase in three days, the scope was wrong, not the timeline.

TDD Without Compromises

Each phase followed strict TDD: failing test first, then implementation, then commit. No exceptions. Even when the feature was "obviously going to work."

That discipline is the only reason I didn't drown in regressions. By the end, I had 1,424 tests across 73 test files — covering models, services, tasks, security, end-to-end flows, and even multi-tenant isolation. (Plus 73 dedicated security tests for SQL injection, XSS, path traversal, auth attacks, and tenant boundary checks.)

The tests run in under a minute on SQLite. CI catches regressions before they reach a PR review.

Layered Architecture Saves You

The hardest part wasn't any single feature. It was keeping the architecture clean enough that phase 18 didn't break phase 1.

Layered architecture helped — every change had to fit one of these layers:

Layer	Responsibility	Rule
`models/`	ORM definitions	No business logic
`services/`	Business logic	All state changes happen here
`tasks/`	Celery wrappers	Thin shells around services
`ui/state/`	Reactive state classes	Calls services and tasks
`ui/pages/`	Reflex page functions	Returns components, no logic

Strict boundaries meant I could add features in phase 18 without rewriting foundations from phase 1. Need to add a new pipeline type? Add a model, a service method, a task wrapper, a state action. Done.

I write more about why I picked a public schema with org_id filtering over the textbook schema-per-tenant approach in another post.

The Open-Core Plugin System

One non-obvious win: the hooks system. Core emits events like connection.before_create and run.upload_completed. The cloud plugin (billing, quotas, metering) subscribes to those events. Core never imports the plugin — the plugin only loads when DATANIKA_EDITION=cloud is set.

This means the open-source core has zero billing code. It's not crippled — it's just complete without billing. Self-hosted users get every feature except the Paddle integration, which they don't need anyway.

What I Learned

Scope control matters more than speed. A 45-step plan with clear boundaries beats "let me just build the whole thing." Most solo projects fail at scope — they either expand until burnout or never ship anything that works end-to-end.

Pick boring tools. Postgres, Celery, Redis, dlt, dbt-core — none of these are exciting in 2026. All of them work. I'd rather spend my creativity on the product than on figuring out which obscure framework to bet on.

Tests as a forcing function. TDD isn't about test coverage. It's about being forced to articulate what "done" means before you write the code. That alone catches 80% of design mistakes.

See For Yourself

Building this in public. Feedback welcome — open an issue on GitHub.

How I Split Open-Source Core From Paid Cloud Without Forking the Codebase

Evgenii Timofeev — Mon, 31 Aug 2026 14:31:47 +0000

Datanika's core is open source under AGPL-3.0. The billing layer — subscription management, quota enforcement, Paddle integration, usage metering — lives in a separate repo called datanika-cloud and is proprietary. Core never imports cloud. Cloud imports core, but only to subscribe to events.

There's no fork. There are no feature flags. There's no if ENTERPRISE_EDITION: branching inside core functions. Just a generic hooks system and a plugin that registers handlers at startup.

The Problem With Conditional Branching

Most open-core projects get the boundary wrong. They have one codebase with enterprise features gated behind runtime checks:

def create_connection(org_id: int, ...):
    if is_enterprise():
        check_quota(org_id)   # ← this line is the problem
    # ... actual business logic

That check creates a permanent coupling between core and the paid tier. Once you have one of those lines, you need them everywhere. Core now has to know about plans, quotas, and billing logic. The "open-source" version is cluttered with enterprise hooks that are dead code for most users. Contributors see billing logic in files they're trying to fix bugs in.

I didn't want that. I wanted a core that doesn't know billing exists, and a billing layer that watches from the outside.

The Solution: A Generic Hook System

The whole boundary is ~30 lines of Python. Core ships a generic event emitter:

# datanika/hooks.py
_handlers: dict[str, list[Callable]] = {}

def on(event: str, handler: Callable) -> None:
    _handlers.setdefault(event, []).append(handler)

def emit(event: str, **kwargs) -> None:
    for handler in _handlers.get(event, []):
        handler(**kwargs)

Services in core call emit() at specific checkpoints:

# datanika/services/connection.py
from datanika.hooks import emit

async def create_connection(org_id: int, name: str, ...) -> Connection:
    emit("connection.before_create", org_id=org_id)   # ← anyone listening?
    # ... actual business logic
    return connection

The core doesn't know or care what connection.before_create means to downstream code. If no handler is registered, emit() is a no-op. The service moves on and creates the connection.

The Plugin Side

The cloud plugin has one entry point — init_cloud(app) — called once at startup when the environment variable DATANIKA_EDITION=cloud is set:

# datanika-cloud/datanika_cloud/plugin.py
from datanika.hooks import on
from datanika_cloud.billing.quota import (
    check_connection_quota, check_schedule_quota,
    check_seat_quota, check_run_quota, check_sso_quota,
)
from datanika_cloud.billing.meter import (
    handle_model_runs, handle_upload_runs, handle_transformation_run,
)

def init_cloud(app: rx.App) -> None:
    # Quota enforcement — reactive
    on("connection.before_create", check_connection_quota)
    on("schedule.before_create", check_schedule_quota)
    on("membership.before_create", check_seat_quota)
    on("run.before_execute", check_run_quota)
    on("sso_config.before_create", check_sso_quota)

    # Usage metering — reactive
    on("run.models_completed", handle_model_runs)
    on("run.upload_completed", handle_upload_runs)
    on("run.transformation_completed", handle_transformation_run)

    # ...

Eight lines register eight hook handlers. When a user tries to create a connection, the check_connection_quota handler gets called with the org ID. It looks up the org's plan, checks the connection count, and raises QuotaExceededError if they're at the limit. Core sees the exception propagate, rolls back, and returns an error to the user.

Core has no idea a plugin exists. It just called emit("connection.before_create", org_id=42) and got an exception back. It handles the exception the same way it would handle any other.

The Additive Side

I need to be honest about one thing: the plugin doesn't only react. It also adds things the core never had:

def init_cloud(app: rx.App) -> None:
    # ... hook handlers above ...

    # Merge billing translations into core i18n cache
    register_translations(CLOUD_TRANSLATIONS)

    # Add a billing link to the sidebar
    extra_sidebar_links.append(("nav.billing", "/settings/billing", "credit-card"))

    # Register a Reflex page at /settings/billing
    app.add_page(billing_page, route="/settings/billing", ...)

    # Register a Paddle webhook route
    app._api.routes.append(webhook_route)

The i18n translations, the sidebar link, the billing page, and the webhook route are all additive — they're new surfaces, not modifications to existing core behavior. The core exposes extension points (extra_sidebar_links, app._api.routes, app.add_page) that any plugin could use. Cloud happens to be the only one using them today.

The mental model is: core defines what's possible; plugins choose what to activate. Quotas are a kind of activation — "hey, watch this event and block it if the org is over limit". UI additions are another kind — "hey, put this link in the sidebar". Neither requires modifying core code.

Self-Hosted Gets Everything

Because the plugin only loads when DATANIKA_EDITION=cloud is set, self-hosted users running docker compose up get:

All 36 connectors — PostgreSQL, BigQuery, Stripe, Salesforce, everything
dbt transformations, tests, snapshots, packages
The full visual pipeline builder
Scheduling with cron + dependency DAGs
Multi-org with RBAC and audit logging
SSO (SAML + OIDC) — the code is in core, just not gated to a plan
9 languages
The REST API

What they don't get:

Paddle billing overlays
Plan-based quota enforcement (they have no plans, so no limits)
Usage metering to a central ledger

If you self-host, nothing is "locked". The quota handlers simply aren't registered, so there's nothing to check against. You get a platform without billing — which is exactly what self-hosted users want.

There's no "community edition" and "enterprise edition" distinction at the code level. There's only one codebase, and the plugin is either loaded or not.

Why Not Feature Flags?

The obvious alternative is feature flags: one codebase, runtime checks. if settings.billing_enabled: ... everywhere you'd otherwise hook.

I considered it. Rejected it because:

It doesn't separate the license. The code is still in the same repo, still compiled into the same binary. If I want the billing logic to be proprietary, feature flags don't help.
It couples unrelated concerns. Every function that needs a quota check now imports from a billing module. The "open-source" part isn't really open-source — it depends on billing internals to decide whether to run them.
It makes testing harder. Every test has to mock or disable the billing flag. With a plugin, tests in core never see any billing code at all.
It leaks into docs. The OSS README has to explain feature flags a self-hosted user should never enable. Noise for contributors.

The hook system avoids all four. Core's tests run against a core without any handlers registered. They're testing the real, shipped behavior for OSS users. The plugin has its own test suite that registers handlers and verifies them in isolation.

The One Rule

Getting the boundary right took me longer than the implementation. The rule I landed on, which I wrote on a sticky note and kept re-reading:

Core defines the events. Cloud chooses which ones matter.

Core must never import from cloud. Cloud may only import from core's public interface — no reaching into private modules. If I find myself wanting to add a new hook because cloud needs one, that's fine — it means the core probably has an extension point it should have exposed anyway. But I never add hooks that only cloud uses without also documenting them as general extensibility.

This means core is genuinely usable by someone who forks it and writes a completely different plugin — for their own billing system, for custom observability, for an internal analytics layer. The hooks are generic, the events are stable, and the plugin pattern is reproducible.

What's in Each Edition

For anyone curious about where the line falls:

Feature	Open-core (AGPL)	Cloud plugin (private)
Pipeline builder, scheduling, dbt	✅	—
36 connectors	✅	—
SSO (SAML + OIDC) code	✅	—
Notification channels	✅	—
REST API v1	✅	—
Audit logging	✅	—
Paddle checkout overlay	—	✅
Plan model + Subscription model	—	✅
Usage ledger + hourly overage sync	—	✅
Quota enforcement	—	✅
Billing settings page	—	✅
Free / Pro / Enterprise plan records	—	✅

Notice what's not in the cloud plugin: SSO, notifications, RBAC, RESTful API. Those are in core. They're sometimes associated with "enterprise features" in other products, but Datanika treats them as basic platform functionality — if you want SSO and you're self-hosting, you get SSO. The plugin is purely about commercial concerns (subscriptions, quotas, metering), not feature gating.

Try It Both Ways

If you want to see the architecture in action:

Self-hosted (plugin off): docker compose up — all features, no billing
Cloud (plugin on): app.datanika.io — same features, plus subscription management
Pricing page for cloud: /pricing — Free / Pro / Enterprise tiers

And if you're building your own open-core product, the one thing I'd recommend is: start with the plugin boundary, not the features. Decide what "core" is going to mean, write down the rule, and don't break it even when it's tempting. Six months in, you'll either have a clean separation or a tangled mess — and the difference is 100% about whether you enforced the rule on day one.

Would love to hear how others handle open-core monetization. Open an issue or discussion on GitHub — I'm collecting notes for a follow-up post on the business side of open-core, not just the technical one.

I Wrote a Full SaaS UI Without a Single Line of JavaScript — the architecture choices that made this kind of boundary possible
I Designed Multi-Tenancy Wrong — the same layering principle applied to the Postgres side
Self-Hosting Guide — run the open-core edition on your own infrastructure
Pricing — what the cloud plugin adds on top

MongoDB `Authentication failed`: You're Authenticating Against the Wrong Database

Evgenii Timofeev — Mon, 31 Aug 2026 12:58:38 +0000

You have a MongoDB user. You created it yourself. You can paste the password into mongosh and get a shell. Then you put the same credentials into an application, a BI tool, or an ELT connector, and get:

pymongo.errors.OperationFailure: Authentication failed.

The credentials are fine. The problem is that you and MongoDB disagree about which database the user lives in.

The database in the URI is also the auth database

Here is the connection string almost everyone writes:

mongodb://analytics_ro:s3cret@mongo.internal:27017/production

Read that as a human and it says: connect to the production database as analytics_ro. Read it as MongoDB does and it says something extra — and look for analytics_ro **inside* production*.

The database component of a MongoDB URI serves double duty. It names the default database for operations and the authentication database, unless you override the second with authSource. The rule in the connection-string spec is:

authSource given → use it.
Not given, but a database is in the path → use the path database.
Not given and no database in the path → use admin.

Almost every real connection string has a database in the path. So almost every real connection string silently opts into the middle rule.

Why this hits nearly everyone

Because the place your user actually lives is admin, and you probably didn't choose that.

The official MongoDB Docker image creates its user in admin. That is what MONGO_INITDB_ROOT_USERNAME does.
MongoDB Atlas stores database users in admin.
Every managed provider and essentially every "create a read-only user" tutorial starts with use admin.

So the default is a mismatch: your user is in admin, your data is in production, and your URI told the driver to look for the user in production. There is no user there. Authentication fails, correctly, and the error message tells you nothing about which database it searched.

Confirming it in thirty seconds

mongosh takes the same flag, so you can isolate the variable without touching your application:

# Fails — looks for the user inside `production`
mongosh "mongodb://analytics_ro:s3cret@mongo.internal:27017/production"

# Works — looks for the user in `admin`
mongosh "mongodb://analytics_ro:s3cret@mongo.internal:27017/production?authSource=admin"

If the second one connects, you have your answer, and it was never a password problem.

You can also just ask where the user is:

use admin
db.system.users.find({}, { user: 1, db: 1 })

The db field on each record is the authentication database. That is the value authSource needs.

The fix

Append it to the URI:

mongodb://analytics_ro:s3cret@mongo.internal:27017/production?authSource=admin

Or pass it as a parameter, if your driver prefers that shape:

MongoClient(
    host="mongo.internal",
    port=27017,
    username="analytics_ro",
    password="s3cret",
    authSource="admin",     # where the user lives
)["production"]             # what you want to read

Note what the two lines mean. authSource is where the user is defined; the database you select afterwards is what you want to read. Keeping them straight in your head is most of the battle.

If you use Atlas, the string it hands you already contains authSource=admin — which is why Atlas users often never learn this rule until the first time they hand-assemble a URI.

Why this bug is so good at hiding

Here is the part that makes it survive code review, CI, and a local test run: an unauthenticated mongod is completely unaffected.

Start MongoDB with no --auth, connect with no credentials, and there is no authentication step to get wrong. Every query works. That is exactly what a developer laptop looks like, and exactly what a throwaway container in a test suite looks like.

The mismatch only appears against a server that actually enforces authentication — which is to say, only in staging and production, and only after everything has been signed off.

We know this shape well, because we shipped it. Our own MongoDB connector built mongodb://user:pass@host:port/<target-db> with no authSource and therefore could not authenticate against any standard deployment. It went unnoticed for the reason above: the instance it was developed against had no auth enabled. It was one of several connector defects we found and wrote up in 2,300 Passing Tests and a CSV That Loaded One Row, and the rule that came out of that audit — a connector is not done until a row has been observed in the destination — is the one that catches this class.

How Datanika handles it now

Datanika's MongoDB connector authenticates against admin by default, and builds authSource into the URI accordingly.

That default is deliberate, and it is the opposite of the previous behaviour rather than a compatible extension of it. Defaulting to the target database would have been the backwards-compatible choice, and it would have meant the connector stayed broken for everyone who didn't already know about a setting nobody had told them about. Defaulting to admin fixes the configuration almost everyone actually has.

Both code paths agree on it now. The URI is assembled by one function that Test Connection and the run path both call, so the button's verdict and the run's outcome cannot disagree about the same connection. That was worth fixing on its own: for a while they did disagree, and in the worst direction — Test Connection reported failure for a configuration whose runs succeeded, telling users their working credentials were broken.

Full walkthrough in the MongoDB setup guide, and the connector's capabilities and limits are on the MongoDB connector page.

Two caveats we owe you, because both are live right now.

1. If your user is not in admin, the connection form cannot say so. The setting exists in the config — it is auth_source — but the mongodb form renders only Host, Port, User, Password and Database, so there is no input for it. You can still set it, by ticking Use raw JSON on the connection form and adding the key by hand; but a connection saved that way loses it the next time it is saved from the structured form, and authentication silently reverts to admin. Tracked as core#638. If your user lives inside the database you are reading, keep the connection in raw-JSON mode until that lands. A setting with no surface is not a setting — it is a guess that happens to be right most of the time, and we shipped exactly that.

2. If your MongoDB is Atlas, none of this reaches the auth step at all. We build every URI as a plain mongodb:// string with no transport options, so the driver negotiates no TLS — not "TLS if the server offers it", none. Atlas requires TLS, so the handshake fails first and authSource never gets a chance to be wrong. There is no mongodb+srv:// support either, so the seedlist hostname Atlas gives you cannot be entered. The general rule is any server that requires TLS: Azure Cosmos DB's Mongo API, a self-hosted net.tls.mode: requireTLS, and Amazon DocumentDB unless its tls cluster parameter has been explicitly disabled. Tracked as core#626.

The rule above is still the rule — it is a MongoDB rule, not a Datanika one, and it will save you the same afternoon in mongosh, in your application code, and in whatever else you point at that cluster. It is our connector that cannot reach Atlas yet, and we would rather say so on the page than let you find out at the first run.

The general version

The lesson generalises past MongoDB: when a connection string has one slot doing two jobs, someone is going to configure one of them by accident.

The database in a Mongo URI is a default database and an auth realm. A Postgres search_path is a resolution order and a write target. A role is often both an identity and a permission set. In every case the failure looks like a credentials problem, because the error is raised by the auth layer — and in every case checking the credentials is the one thing that will not help.

When authentication fails against credentials you are sure of, stop re-checking the password and start asking what scope the server resolved them in.

Datanika is an open-source ELT platform — extraction, loading, transformation and scheduling in one UI. Browse the connectors or self-host it.

dlt and dbt Silently Filled My Disk. Here's the Cleanup System I Had to Build.

Evgenii Timofeev — Mon, 31 Aug 2026 12:53:35 +0000

After running pipelines for a few days in production, disk usage was climbing steadily. Turns out both dlt and dbt leave temporary files everywhere, and neither cleans up after itself.

The Problem

dlt creates a pipeline directory per run with state files, staging data, and load packages. If you use the same pipeline name, old state accumulates. If you use unique names — which Datanika does for concurrency safety — orphaned directories pile up.

dbt leaves compiled SQL, manifest files, and run results in target/. Every dbt run adds to it. With per-tenant dbt projects at dbt_projects/tenant_{org_id}/, multiply that by the number of active tenants.

On a single-tenant setup this is manageable. On a multi-tenant SaaS, it's a disk bomb.

The Fix: Three Layers

Layer 1: Cleanup in the task itself

Every pipeline and transformation Celery task has a finally block that calls the appropriate cleanup function:

cleanup_pipeline() — removes dlt's working directory after the run completes
clean_target() — wipes dbt's target/ directory before each run starts

This catches the normal case: a run starts, finishes (or fails), and its temporary files are cleaned up in the same execution context.

Layer 2: Hourly maintenance sweep

A dedicated Celery Beat task runs every hour and catches what the task-level cleanup misses — orphaned files from crashed runs, zombie directories from aborted tasks, accumulated state from edge cases:

Sweep	Default threshold	What it cleans
`cleanup_orphaned_dlt_dirs`	24 hours	dlt pipeline working directories older than threshold
`cleanup_dbt_targets`	48 hours	dbt `target/` directories under `tenant_*/` older than threshold
`purge_old_runs`	90 days	Soft-deletes `Run` records older than threshold
`cleanup_orphaned_archives`	—	Upload archive files with no matching DB record

All thresholds are configurable via settings: maintenance_dlt_max_age_hours, maintenance_dbt_max_age_hours, maintenance_run_retention_days.

Layer 3: Protecting active runs

The hourly sweep had a race condition in the first version — it could delete files from a pipeline that was still running. If a dlt run takes 2 hours and the sweep fires at hour 1.5, it would see a "stale" directory and nuke it.

Fix: the sweep checks for active runs before cleaning any tenant's directories. If Run.status == 'running' for any run associated with that tenant, its directories are skipped entirely. Simple, conservative, correct.

The Config

Four settings, all tunable per deployment:

# In .env or settings
MAINTENANCE_DLT_MAX_AGE_HOURS=24      # dlt orphaned dirs
MAINTENANCE_DBT_MAX_AGE_HOURS=48      # dbt target/ dirs
MAINTENANCE_RUN_RETENTION_DAYS=90     # old run records

Self-hosted users can adjust these based on disk size and run frequency. The defaults are conservative — 24 hours for dlt, 48 hours for dbt, 90 days for run history. On a busy system with hundreds of daily runs, you might want to tighten the dlt threshold to 12 hours.

Why This Matters

This isn't exciting work. Nobody writes blog posts about temp file cleanup.

But if you're running dlt or dbt in production with any kind of frequency — especially in a multi-tenant context — you need something like this or you'll wake up to a full disk at 3 AM. The disk-full failure mode is particularly nasty because:

PostgreSQL stops accepting writes
dlt can't stage extracted data
dbt can't write compiled SQL
Celery tasks fail silently (no disk to write logs to)
Even docker logs might not capture the error (log rotation relies on disk space)

The three-layer cleanup — in-task, hourly sweep, race-condition protection — keeps disk usage predictable regardless of tenant count or run frequency. On the €12/mo Hetzner box, disk usage has been flat at ~20 GB for weeks, even with daily runs.

dbt Per-Tenant Architecture — the tenant_{org_id}/ pattern that makes this cleanup necessary
My SaaS Runs on €12 a Month — the infrastructure that this cleanup protects
Architecture Overview — where the maintenance Celery Beat task fits in the system
Self-Hosting Guide — tuning these thresholds for your deployment

dbt Was Not Designed for Multi-Tenant SaaS. Here's How I Made It Work.

Evgenii Timofeev — Mon, 31 Aug 2026 12:48:32 +0000

dbt wants a project directory on disk. One dbt_project.yml. One profiles.yml. Files in models/. This works great for a single team. It doesn't work at all when you have hundreds of tenants.

In Datanika, every organization gets its own dbt project at dbt_projects/tenant_{org_id}/. When a user creates a transformation in the UI, the backend writes a .sql file to their tenant's directory. When they run it, we invoke dbt against that directory with the tenant's own profiles.yml — generated on the fly from their destination credentials.

If you want the broader multi-tenancy context (why Datanika uses a shared public schema in Postgres but per-tenant directories for dbt), I wrote a separate post on how I designed multi-tenancy wrong and then fixed it. This post is the dbt-specific follow-up.

The Directory Layout

Every tenant gets a directory that looks like a stock dbt project:

dbt_projects/
├── tenant_1/
│   ├── dbt_project.yml
│   ├── profiles.yml
│   ├── models/
│   │   ├── stg_orders.sql
│   │   └── stg_orders.yml
│   ├── macros/
│   ├── tests/
│   ├── snapshots/
│   └── target/        ← ephemeral, cleaned on every run
├── tenant_2/
│   └── ...
└── tenant_N/
    └── ...

ensure_project(org_id) creates the skeleton on demand. The profile name is tenant_{org_id} — dbt's profiles.yml lookup key. That's how we keep profiles isolated even though multiple tenants can share the same destination type (e.g. both tenant 1 and tenant 2 load into BigQuery, but with different service account keys).

Problem 1: target/ Fills Up the Disk

dbt leaves target/ directories full of compiled SQL and run artifacts. With many tenants, disk usage grows fast. A single tenant's target/ is small, but 100 tenants × repeated runs = gigabytes of stale compiled SQL you'll never look at.

Fix, part one: clean_target(org_id) before every pipeline and transformation run. No historical build artifacts survive past the next execution.

Fix, part two: an hourly Celery Beat task (cleanup_dbt_targets) that sweeps orphaned target/ directories older than 48 hours. If a tenant runs nothing for 48 hours, their target/ is fair game for deletion — the next run will rebuild it.

Together, these two loops keep disk usage predictable even as tenant count grows.

Problem 2: schema.yml Gets Duplicate Entries

Originally, I generated one schema.yml per dbt project listing every model. It worked until users started moving models between schemas — the file kept accumulating duplicate entries as models were deleted and re-added, and dbt would throw cryptic parse errors.

Fix: per-model YML files instead of one big shared file. Every stg_orders.sql gets a sibling stg_orders.yml with its own schema + tests declaration. Move a model? The YML moves with it. Delete a model? The YML goes too.

models/
├── stg_orders.sql
├── stg_orders.yml        ← only describes stg_orders
├── mart_revenue.sql
└── mart_revenue.yml

Less clever, far more robust.

Problem 3: The SQL Editor Needs Macro Resolution

Datanika's SQL editor has autocomplete for ref() and source() macros — when you type {{ ref(, it should suggest the tenant's existing models. That means parsing the tenant's dbt project to know which models and sources exist right now.

dbt compile handles this correctly, but it's slow — 500ms to a few seconds depending on project size. Unacceptable for an autocomplete dropdown.

Fix: cache the model/source catalog per tenant, refresh after every run. The cache lives in Redis (keyed by org_id) and is invalidated on pipeline success. Autocomplete reads from cache, which is microseconds. The only lag is right after a run — and by then, the user isn't typing, they're looking at results.

Problem 4: Everything Else Has to Be Per-Tenant Too

Once the directory pattern is in place, every dbt feature needs to be wired through the same tenant path:

Snapshots (SCD Type 2) — stored in tenant_{org_id}/snapshots/, run with dbt snapshot against the tenant's profile
Packages (dbt deps) — each tenant has its own packages.yml and dbt_packages/ folder. Installing dbt_utils is a per-tenant operation, not a global one
Source freshness — tenant_{org_id}/models/sources.yml lists their sources with freshness thresholds; dbt source freshness runs per-tenant
Tests — generic tests (unique, not_null, accepted_values, relationships) live in the same per-model YML; singular tests go in tenant_{org_id}/tests/

Each feature required wiring through the same tenant directory pattern. None were conceptually hard. All were fiddly.

What This Costs

The dbt integration is easily the part of the Datanika codebase with the most "I wish dbt had a proper API for this" comments. dbt's CLI-first design assumes one project per process. Running N projects in the same Python process means managing:

Working-directory changes (dbt inspects $PWD aggressively)
Logging output capture per invocation
target/ cleanup between runs so stale artifacts don't leak
Profile isolation so tenant A's credentials never resolve to tenant B's warehouse

You can see the result in the UI — SQL editor, tests, snapshots, packages, freshness, all per-tenant. It works. It's not elegant under the hood, but it's contained in a single DbtProjectService and the rest of the codebase doesn't have to care.

The Lesson

dbt is a brilliant tool for single-tenant data teams. For multi-tenant SaaS, you have to build the tenancy layer yourself — and the right architecture is filesystem-per-tenant plus aggressive target/ cleanup, not something clever at the Python level.

If I were starting today, I'd still use dbt-core. The alternative is rebuilding macro resolution, ref graph tracking, incremental materializations, and SCD Type 2 snapshots — all of which dbt has solved. The wrapper code is annoying, but it's once-only work.

Anyone else running dbt in a multi-tenant setup? Open an issue on GitHub — I'd love to compare notes.

Architecture overview — how the pieces fit together
Transformation Guide — writing dbt models in the UI
Multi-tenancy mistake — the Postgres side of the same problem
32 connectors, most took less than a day — the other place where dlt and dbt's ecosystems paid off
Star on GitHub

I Designed Multi-Tenancy Wrong. Then I Fixed It Before It Mattered.

Evgenii Timofeev — Mon, 31 Aug 2026 09:13:02 +0000

My original plan for Datanika was schema-per-tenant in PostgreSQL. Each organization gets its own schema. Clean isolation. Textbook approach.

Then I started implementing services and realized: none of my SQLAlchemy queries set search_path. Every service method would need schema-switching logic. Alembic migrations would need to run against every tenant schema. Testing would become painful.

So I scrapped it.

What I Did Instead

I moved everything to the public schema with an org_id column on every table. Simple WHERE clause filtering. One migration path. One set of queries.

Every model inherits from TenantMixin:

class TenantMixin:
    org_id: Mapped[int] = mapped_column(BigInteger, ForeignKey("organizations.id"), nullable=False)

Every query is scoped through a session-level filter:

async def get_connections(session, org_id: int) -> list[Connection]:
    result = await session.execute(
        select(Connection).where(Connection.org_id == org_id)
    )
    return list(result.scalars())

That's it. No schema switching, no SET search_path, no per-tenant migrations. Tenant isolation is enforced by application code, not by the database catalog.

"But Schema-per-Tenant Is More Enterprise"

This felt like a downgrade at first. Schema-per-tenant sounds more "enterprise." Hosting providers love to talk about it. AWS RDS docs reference it. It's the "right" answer in academic database textbooks.

But for a SaaS where tenants share the same table structures, org_id filtering wins on every dimension that actually matters:

Aspect	Schema-per-tenant	Shared schema + `org_id`
Migrations	Run N times (one per tenant)	Run once
Connection pooling	Expensive (per-schema warmup)	Cheap (shared pool)
Query plans	N copies (one per schema)	One copy, cached
Adding a new column	DDL on N schemas	DDL on 1 table
Testing	Need per-tenant test fixtures	One fixture, many `org_id` values
Cross-tenant analytics	Hard (UNION across schemas)	Trivial (`GROUP BY org_id`)

The only thing schema-per-tenant gives you is physical-level isolation that protects against application bugs. But if your app has the kind of bug that lets one tenant query another tenant's data, the bug is in your security layer — and physical isolation is just papering over it. Better to test the security layer thoroughly. (I have 5 dedicated tenant isolation security tests plus 73 total security tests.)

Where Schema Isolation Still Makes Sense

I kept one thing from the original design: per-tenant dbt project directories on disk at dbt_projects/tenant_{org_id}/. dbt needs real files on disk, and a single dbt project can't cleanly handle multiple tenants' models, sources, and configs in one directory tree. So that isolation still makes sense — different problem, different solution.

The pattern I learned: pick the isolation model per concern, not per database.

App data with identical schemas: shared table + tenant column
Per-tenant SQL files dbt needs to read: filesystem-level isolation
Per-tenant secrets or keys: encrypted column with rotation, not separate vaults

Test Suite Bonus

The biggest unexpected win: my entire test suite runs on in-memory SQLite because there's no schema-switching to worry about. 1,400+ tests run in under a minute. CI is fast. Local development is fast. New engineers (well, future me) can run tests without spinning up a PostgreSQL container.

If I'd gone with schema-per-tenant, every test would need a real Postgres connection to set search_path. SQLite doesn't even support schemas the way Postgres does. The test suite would have been 5–10× slower.

The Lesson

Pick the isolation model that matches your actual query patterns, not the one that looks best in an architecture diagram. For config tables with identical schemas across tenants, a shared table with a tenant column wins.

Pivots early in a project are cheap. Pivots after you've shipped to customers are expensive. If you're starting a multi-tenant project right now, write a few service methods first and see how the queries look before committing to a schema design.

Have you hit a similar architecture pivot early in a project? Reply on the GitHub discussions — I'm collecting war stories.

2,300 Passing Tests and a CSV That Loaded One Row

Evgenii Timofeev — Mon, 31 Aug 2026 09:06:18 +0000

Someone on the team went to take a screenshot of a first pipeline run. Twelve rows of customers.csv, into DuckDB — the zero-credentials template our own CSV guide calls "the first pipeline you ever run on Datanika."

The run came back green. One row landed. Its columns were:

file_name, relative_path, file_url, mime_type, modification_date, size_in_bytes

Not one customer column. Not a name, not an email, not an ID. The connector had faithfully loaded a description of the file instead of the file.

At that moment the test suite was at roughly 2,300 tests, all passing, including a dedicated block of them for exactly this code path.

Why the suite could not have failed

Here is the shape of every test we had for the file-source builder:

@patch("datanika.services.dlt_runner.filesystem")
def test_build_csv_source(self, mock_fs):
    mock_fs.return_value = "csv_src"
    result = runner.build_source(conn, {"bucket_url": "s3://bucket/data/"})
    mock_fs.assert_called_once_with(bucket_url="s3://bucket/data/", file_glob="*")
    assert result == "csv_src"

Read the last two lines slowly. We assert that we called the library with the arguments we just passed in, and then we assert that the return value equals the string we ourselves told the mock to return.

Both assertions are about our own typing. Neither one can observe what the library actually does. filesystem() in dlt is a lister — it yields file metadata, and you are expected to pipe it through a transformer like read_csv() to get contents. We never added the transformer. There were zero occurrences of read_csv, read_jsonl or read_parquet anywhere in the package.

The mock happily played the part of a working connector, because a mock will play any part you write for it. The suite was not weak here, and it was not under-maintained. It was structurally incapable of failing, and it had been green for months on that basis.

That is the part worth sitting with. A failing test tells you something. A passing test tells you something only if it could have failed. We had 2,300 signals and no way to tell which of them carried information.

So we went looking for the others

If one connector could be this broken behind green tests, the honest assumption is that others were too. So we stopped asking "do the tests pass" and started asking a different question for every connector: has this one ever been proven to move a single row?

Not "is it implemented." Not "is it covered." Has a row gone in one end and come out the other, against a real database, observed in the destination.

The answer for a lot of them was no. What that turned up:

A glob matching zero files reported success. Point a source at the wrong path, or an S3 prefix someone emptied, and you got a green run with zero rows in about four seconds. Wrong path, moved file, and correct-but-empty were indistinguishable. Test Connection couldn't help either — it returned "Test not applicable for this type" unconditionally for every non-database source, so a broken path tested exactly like a working one.

DuckDB loads never reached the Data Catalog. The duckdb_engine driver was missing from the image, and the failure was swallowed as "Catalog sync failed (non-fatal)." Both of our getting-started guides tell you to verify your first run by browsing the Catalog. For DuckDB, that had never once worked.

Three warehouse destinations couldn't authenticate. BigQuery, Databricks and Synapse stored credentials under the names our form used and handed them to dlt, which wanted different names — BigQuery stores project and keyfile_json where dlt expects project_id, private_key and client_email, and nothing parsed the keyfile at all. The translation layer covered SQL databases only. Nobody had noticed, because nobody had completed a run.

Kafka subscribed to a topic that didn't exist. The connection form stores topics as a comma-separated string. The builder did topics if isinstance(topics, list) else [topics] — so orders,events became a single topic named, literally, orders,events. It subscribed successfully. It consumed nothing, forever.

MongoDB looked for your user in the wrong database. The database in a Mongo URI doubles as the auth database, so omitting authSource means "the user lives inside the database you're reading" — which is not where anyone puts it. Atlas, every managed provider, and every Docker setup using MONGO_INITDB_ROOT_USERNAME create users in admin. First connection attempt failed for basically everyone.

And failure notifications had never fired. This one is the sharpest, because we had written a tutorial about it. The Slack, Telegram, email and webhook channels were all real, the Run Failed event was real, and the handler that formats the message was real. Nothing ever emitted the failed-run event. The branch was unreachable code, and had been since it shipped. It works now — but for a while the honest description of that feature was "a correctly implemented notification for an event that is never announced."

The rule we apply now

Every one of those is fixed and live. But the fixes are less interesting than the rule that came out of it, which is now how we review connector work:

A connector is not done when its tests pass. It is done when a row has been observed in the destination.

Concretely, three things changed.

Probes run against the real thing. Postgres, MySQL, MongoDB, Kafka and DuckDB now have tests that spin up a real container, load real rows, and query the destination to count them. Not the run status — the destination. A green run means the load finished, which is a different claim from "your data is there."

Every fix ships with a probe we watched fail. Before a fix lands, its test is run against the broken code and has to go red. This sounds obvious. It is also exactly the step that, had anyone taken it on the file connector, would have caught all of this months earlier — because the moment you feed that test a realistic listing-shaped payload, the assertion result == "csv_src" is revealed as meaningless.

Known-broken connectors are pinned red, not skipped. Where something can't be proven yet, it carries a strict expected-failure marker rather than a skip. A skip is silent forever. A strict marker fails the build the day the behaviour starts working, which forces someone to notice and remove it. A test that starts passing should be an event, not a non-event.

What we still can't claim

The audit is not finished, and it would be a bad look to write this post and then round up.

Snowflake and Synapse are verified at the configuration layer only. We check that the credentials we hand to dlt are the ones dlt declares it wants. Nothing has connected. No ODBC driver exists in our test environment, and a Snowflake trial signup was refused by their risk engine.
Google Analytics and Facebook Ads are implemented, not proven. Both were built against the documented API shapes with no credentials anywhere, including in CI. Their first real run will be the actual test, and we would rather say that than let you find out.
Google Ads is row-proven against a local HTTP server, not against Google. The parsing, flattening and auth headers are verified; the round trip to Google's own API is not.
The tracking issue for the whole audit is still open. We are working through the remaining builders in the same way.

We would rather publish that list than a rounder number. If you are evaluating a data platform, the useful question is not how many connectors it claims — it is which of them anyone has watched move a row, and whether the vendor will tell you the difference.

Why this is a post at all

There is an argument for not writing this one. It is a catalogue of our own bugs, some of them embarrassing, on the exact path a new user takes first.

We are writing it because the alternative is worse. If you self-host Datanika you can read every line of this in the commit history anyway — the issues are public, the fixes are public, and the test that couldn't fail is right there in the diff. An open-source project that only publishes its wins is asking you to trust a filtered feed of a repository you can just go read.

And the general lesson isn't ours alone. If you have a connector, an SDK wrapper, or any integration tested entirely by patching the library at the boundary, the odds are decent that your suite is asserting your own arguments back at you. The cheap check is to break the code on purpose and confirm something turns red. If nothing does, you don't have a test — you have a very long, very green description of what you intended.

Datanika is an open-source platform that runs dlt extract-and-load and dbt-core transformations behind one UI, with 36 connectors, scheduling and run history. Self-host it or start free.

I Wrote a Full SaaS UI Without a Single Line of JavaScript

Evgenii Timofeev — Mon, 31 Aug 2026 09:05:32 +0000

No React. No TypeScript. No separate frontend repo. Just Python.

I'm building Datanika — a data pipeline platform — and chose Reflex for the entire UI. Reflex compiles Python to React under the hood. You write state classes and component functions in Python, and it handles the rest.

Why Reflex

I'm a solo developer. Maintaining two codebases (Python backend + JavaScript frontend) would have killed the project before it shipped. Every feature would need API contracts, type duplication, a separate dependency tree, and a separate CI pipeline. For a single person shipping 18 phases in a month, that's not scope control — that's guaranteed failure.

With Reflex, my pipeline service, my UI state, and my page components all share the same language, the same types, the same imports. Need to show a list of connections on the frontend? The state class calls the service class directly. No fetch, no JSON parsing, no zod schema:

class ConnectionState(BaseState):
    connections: list[dict] = []

    async def load(self):
        async with AsyncSessionLocal() as session:
            self.connections = await ConnectionService.list(session, org_id=self.current_org_id)

The service returns Python objects, the state holds them, the page renders them. One mental model, one type system, one debugger.

It's Not Perfect

I'm not going to pretend Reflex is flawless. It isn't. A few things bit me:

Reflex 0.8.x uses Starlette internally, not FastAPI. Custom API routes can't use FastAPI decorators — you have to build starlette.routing.Route objects and append them to app._api.routes. This bit me when wiring up OAuth callbacks for Google and GitHub sign-in, and again when adding the Paddle webhook endpoint. It's not hard, but it's not documented anywhere prominently either. I wrote it down in my architecture doc so I'd never have to re-discover it.

The WebSocket event system has quirks. Reflex uses WebSockets to sync state between the Python backend and the React frontend. Most of the time it just works. Occasionally you hit an edge case where a state update doesn't propagate, and the fix is adding a yield in an async handler that doesn't obviously need one. I learned to treat "the button didn't update" as "missing yield" before any other hypothesis.

Wrapping React components is possible but the docs are thin. When you need something Reflex doesn't ship natively — a cron expression picker, a DAG/graph visualizer, a Monaco SQL editor — you wrap a React component in Python. It works. The wrapper pattern is clean. But the docs are thin, so I learned it by reading Reflex's own source code for the components they do ship.

Hot reload is sometimes confused. Reflex's dev server watches Python files and recompiles. When you touch content.config.ts or similar, the recompile sometimes picks up stale state. Ctrl+C, restart, done. Rare but annoying.

What I Got in Return

The tradeoff was worth it. I built the following, all in Python, in one repo, by myself:

Login, signup, social OAuth (Google + GitHub)
SAML + OIDC SSO for Enterprise
Pipeline builder with connection picker and load modes
Full-screen SQL editor with autocomplete and dbt macro resolution
DAG visualizer for pipeline dependencies
Drag-and-drop CSV / JSON / Parquet file uploads
Multi-org with RBAC (owner/admin/editor/viewer)
Billing settings page with Paddle overlay checkout
Notification channels config (Slack, Telegram, email, webhook)
9-language i18n with runtime switching
Data catalog browsing all tables and models
API keys management page

Every one of those would have needed a separate React component, a state-management integration, a TypeScript type, and a test setup if I'd used a split-stack architecture. Instead they're Python functions calling Python services calling Python models.

The Layered Architecture Payoff

Because everything is Python, the layered architecture I set up in phase 1 extends all the way to the UI:

models/      → ORM, no logic
services/    → business logic, all state changes
tasks/       → Celery wrappers around services
ui/state/    → Reflex state classes, call services and tasks
ui/pages/    → functions returning rx.Component, no logic

Each layer imports the one below it. Reflex pages import state classes, which import services, which import models. No layer skips down. No layer imports up. When I add a new feature, I add one thing to each layer, in order.

This is the same layering I described in the multi-tenancy mistake post — and it's the same one that made the dbt-per-tenant complexity containable. Put the gnarly stuff in a service, expose it through a clean interface, and the layer above it doesn't have to care.

When Reflex Fits — and When It Doesn't

Reflex isn't the right answer for every app. I wouldn't use it for:

Public marketing sites — you want Astro, Next.js, or similar. This blog is built with Astro, not Reflex, precisely because it's static content.
Consumer apps with heavy client-side state — games, editors, real-time collab tools. Reflex can do it, but you're fighting the framework.
Apps where the frontend team is separate from the backend team — the whole point of a unified stack is that the same person owns both. If you have a dedicated React team, give them React.

But for internal tools, B2B SaaS, admin panels, data platforms, Reflex is seriously underrated. These apps have:

Forms and tables (Reflex strength)
CRUD workflows (Reflex strength)
Server-driven state (Reflex strength)
Small frontend teams or no frontend team at all (Reflex strength)

If you're a backend developer building a SaaS and dreading the frontend, Reflex is worth a serious look.

Try It

Everything I described above is open source under AGPL-3.0. You can clone the Datanika repo, run docker compose up -d, and see Reflex in production:

Self-host with Docker
Architecture overview — including the "Reflex 0.8.x uses Starlette" note buried in there
Star on GitHub
Try the cloud version free — same codebase

Curious what others think about Python-only full-stack. Has anyone else tried this for production SaaS? Open an issue or discussion on GitHub — I'd love to compare notes.

I Built an ETL Platform Solo. 18 Phases, 1,400+ Tests, One Month. — how the scope control worked
I Designed Multi-Tenancy Wrong — where layered architecture started paying off
dbt Was Not Designed for Multi-Tenant SaaS — same layering applied to dbt integration

DEV Community: Evgenii Timofeev

Introducing the Datanika REST API v1

Your Pipelines, Programmable

What's Included

Full CRUD for Every Resource

Authentication

Rate Limiting

Interactive Docs

Use Cases

What's Next

Try It

I Added 32 Connectors to My Data Platform. Most Took Less Than a Day Each.

The Architecture

The Hard Ones

Phased Rollout

The Lesson

What's Next

Try It

I Built an ETL Platform Solo. 18 Phases, 1,400+ Tests, One Month.

Scope, Not Speed

TDD Without Compromises

Layered Architecture Saves You

The Open-Core Plugin System

What I Learned

See For Yourself

How I Split Open-Source Core From Paid Cloud Without Forking the Codebase

The Problem With Conditional Branching

The Solution: A Generic Hook System

The Plugin Side

The Additive Side

Self-Hosted Gets Everything

Why Not Feature Flags?

The One Rule

What's in Each Edition

Try It Both Ways

Related

MongoDB `Authentication failed`: You're Authenticating Against the Wrong Database

The database in the URI is also the auth database

Why this hits nearly everyone

Confirming it in thirty seconds

The fix

Why this bug is so good at hiding

How Datanika handles it now

The general version

dlt and dbt Silently Filled My Disk. Here's the Cleanup System I Had to Build.

The Problem

The Fix: Three Layers

Layer 1: Cleanup in the task itself

Layer 2: Hourly maintenance sweep

Layer 3: Protecting active runs

The Config

Why This Matters

Related

dbt Was Not Designed for Multi-Tenant SaaS. Here's How I Made It Work.

The Directory Layout

Problem 1: target/ Fills Up the Disk

Problem 2: schema.yml Gets Duplicate Entries

Problem 3: The SQL Editor Needs Macro Resolution

Problem 4: Everything Else Has to Be Per-Tenant Too

What This Costs

The Lesson

Related

I Designed Multi-Tenancy Wrong. Then I Fixed It Before It Mattered.

What I Did Instead

"But Schema-per-Tenant Is More Enterprise"

Where Schema Isolation Still Makes Sense

Test Suite Bonus

The Lesson

Related

2,300 Passing Tests and a CSV That Loaded One Row

Why the suite could not have failed

So we went looking for the others

The rule we apply now

What we still can't claim

Why this is a post at all

I Wrote a Full SaaS UI Without a Single Line of JavaScript

Why Reflex

It's Not Perfect

What I Got in Return

The Layered Architecture Payoff