DEV Community: Zephico Technologies

Contract engineer vs. full-time hire: the real cost math

Zephico Technologies — Wed, 12 Aug 2026 20:15:19 +0000

When teams compare a contractor's monthly rate against a salary, the salary usually looks cheaper. That comparison is wrong on both sides, and since we place engineers on contract for a living, it's worth showing the honest version of the math — including the cases where hiring full-time is the right call.

What a full-time senior engineer actually costs

Take a senior engineer in the US or Western Europe with a headline salary somewhere around $150–200k. The real number is bigger:

Employment overhead. Payroll taxes, health insurance, pension contributions, equipment, software seats, office or stipend. Finance teams typically model fully-loaded cost at 1.25–1.4× salary. Your $180k engineer costs the company roughly $230–250k a year before they write a line of code.

Recruiting. An agency fee runs 20–25% of first-year salary — $35–45k for one hire. Do it with an internal recruiter and you're paying their salary plus your engineers' interviewing hours, which are real hours that stopped producing software.

The vacancy itself. Time-to-hire for senior engineers is commonly two to three months, plus notice period. That's a quarter of roadmap that either slips or lands on the rest of the team. Nobody books this to a budget line, which is why it's the most underestimated cost on the list.

The exit. If the project ends, priorities shift, or the hire doesn't work out, you're into severance, notice periods and — in much of Europe — a legal process. The option to stop paying has a price, and full-time employment doesn't include it.

What a contract engineer actually costs

The monthly rate — and that's mostly it. No recruiting fee, no payroll overhead, no severance exposure. The engineer starts within days rather than months, and when the project ships you scale down without a process.

There are real costs on this side too, and pretending otherwise would be salesmanship: onboarding time to your codebase, a management relationship to maintain, and knowledge that walks out when the contract ends unless you deliberately capture it. (We wrote about running contract engineers well — most of these costs are controllable.)

The break-even question

The comparison isn't "rate vs. salary." It's:

Horizon. If you're confident the role exists in three years, employment amortizes its fixed costs and wins. If the need is a project, a migration, a spike in roadmap — the fixed costs never amortize.
Certainty. Hiring is a two-to-three-month commitment to a guess about your future roadmap. Contracting converts that guess into a monthly decision.
Knowledge type. Core product architecture compounds in a long-tenured employee's head. Skills like "build the Databricks pipeline" or "stand up the Retool console" are transferable — you're buying the capability, not the tenure.

Where we tell people to hire instead

If the work is your core product and the horizon is years, hire. A staff engineer who has lived in your codebase for four years is worth more than any rotation of outsiders, and no honest staffing company should claim otherwise. Contract engineers win at the edges: projects with an end date, skills you need now but not forever, and teams that need senior capacity this month, not next quarter.

Run the fully-loaded numbers for your own case before deciding. If the contract column wins, this is what our version looks like.

Originally published on the Zephico blog.

Medallion architecture on Databricks: what actually matters

Zephico Technologies — Wed, 12 Aug 2026 20:14:58 +0000

Every Databricks pitch deck has the same three-layer diagram: bronze for raw data, silver for cleaned data, gold for business-ready tables. The diagram is fine. The problems start when teams treat it as an architecture instead of what it really is — a naming convention for a set of decisions you still have to make.

Here's where those decisions actually bite, based on the lakehouse builds we've done.

Bronze: append-only or you'll regret it

The single most common mistake we see is teams "cleaning up" data on the way into bronze — deduplicating, fixing types, dropping malformed rows. It feels tidy. It's also how you lose the ability to reprocess history when your parsing logic turns out to be wrong.

Bronze should be an append-only, schema-on-read record of exactly what the source system sent you, with ingestion metadata (_ingested_at, _source_file) and nothing else. Storage is cheap. The ability to replay six months of raw events after finding a bug in your silver logic is priceless.

Silver: this is where your real data model lives

Silver is not "bronze but cleaner." It's where you commit to a data model: one row per entity, resolved keys, enforced schemas, quarantined bad records. Two rules that have served us well:

Every silver table gets expectations. Delta Live Tables' expect_or_drop (or plain constraint checks in a batch job) with quarantine tables for the failures. A silver table without declared expectations is a bronze table with a misleading name.
Silver models the domain, not the report. If a table exists because one dashboard needs it, it belongs in gold. Silver tables should survive a BI-tool migration untouched.

Gold: let it be boring and duplicated

Gold tables can be denormalized, redundant, and aggressively shaped for one consumer. That's the point. Resist the urge to build one "canonical" gold layer that serves every team — you'll end up with a committee-designed table that serves nobody and takes four teams to change.

The parts the diagram doesn't show

Unity Catalog from day one. Retrofitting governance onto a running lakehouse is weeks of migration work; starting with it is an afternoon of setup.
Cost visibility before cost optimization. Tag jobs by pipeline and team first. Most "Databricks is expensive" complaints turn out to be one forgotten streaming cluster or an oversized all-purpose cluster someone uses for ad-hoc SQL.
A reprocessing story. Decide before launch how you'll rebuild silver from bronze — full replays, partition-scoped replays, or versioned logic. If the answer is "we'd figure it out," you don't have a lakehouse, you have a pile of Parquet.

None of this is exotic. That's the real lesson: medallion architectures fail on discipline, not on technology. Get the boring parts right and the diagram takes care of itself.

Originally published on the Zephico blog.

Agentic AI and big data: what your platform needs before agents are useful

Zephico Technologies — Sun, 09 Aug 2026 22:25:16 +0000

The pitch for agentic AI on enterprise data is genuinely appealing: instead of a chatbot that answers one question at a time, you get something that plans, queries, checks its own work, calls a second tool, and comes back with a finished piece of analysis — or actually does the thing. Reconcile these two systems. Find out why yesterday's pipeline was late and open a ticket. Pull the accounts at churn risk and draft the outreach list.

The demos work. What determines whether the same thing survives contact with your actual data estate has very little to do with which model you picked, and almost everything to do with what's underneath it. Here's the honest list.

What "agentic" actually changes

A single-shot LLM feature — summarize this, answer that — fails visibly. It returns something wrong, a human reads it, and the human moves on. An agent fails differently: it takes a wrong step early, then builds four more steps on top of it, and the output is a confident, internally consistent artifact that's wrong in a way nobody can see without redoing the work.

Two properties drive most of the difficulty:

Multi-step means compounding error. If each step in a chain is 90% reliable, a five-step task is roughly 60% reliable end to end. That math is why agents that look brilliant on a rehearsed demo feel flaky on real work — the demo was three steps on clean data, the real task is eight steps on messy data.

Tool use means real consequences. The moment an agent can write to a table, open a ticket, send a message, or trigger a job, "the model hallucinated" stops being an annoyance and becomes an incident. Read-only agents and acting agents are different risk products and should be scoped, reviewed and rolled out differently.

The data platform is the bottleneck, not the model

Every agent that touches your data is doing one of two things: retrieving from documents, or querying tables. On the table side — which is where most enterprise value sits — the agent's ceiling is set by your semantic layer, not by model quality.

If your warehouse has three definitions of "active customer," inconsistent grain between your orders and sessions tables, and columns named flag_2 with no description, an agent will do exactly what a new analyst with no context would do: pick something plausible and be wrong. The difference is that the new analyst asks someone by Wednesday, and the agent never does.

Concretely, before agents are worth piloting on your data:

Curated tables, not raw ones. Point agents at governed, well-modeled gold tables with documented columns — not at the bronze layer, not at production replicas. Narrow, clean scope is the single highest-leverage thing you control.
Metadata that carries meaning. Table and column descriptions in a catalog (Unity Catalog, if you're on Databricks) aren't documentation hygiene here — they're the mechanism by which the agent knows what it's looking at.
One definition per metric. Agreed definitions for revenue, churn, active user, encoded once as views or metric definitions. If humans argue about it, the agent will simply pick a side and present it as fact.
Known freshness. An agent that doesn't know a table lags by six hours will happily report yesterday's number as today's.

None of this is agent-specific work. It's the same modeling and governance discipline that makes BI trustworthy — the difference is that agents make skipping it expensive faster, because they scale confident wrong answers to more people.

Tools beat raw access

The instinct with a capable model is to hand it broad SQL access and let it figure things out. In practice, the systems that hold up do the opposite: they expose a small number of narrow, well-described, deterministic tools, and let the model decide which to call rather than how to compute.

get_revenue_by_region(start_date, end_date, region?) is a tool an agent uses correctly almost every time. "Here's a SQL endpoint against 400 tables, good luck" is a tool that works in the demo and produces silent join errors in month two. Every piece of business logic you move out of the model's improvisation and into a tested function is a step you no longer have to evaluate probabilistically.

The design rules that matter: keep the tool count small enough that selection is unambiguous, make each description precise about what it returns and at what grain, return structured data with explicit units and date ranges, and make errors informative — an agent recovers well from "no rows for that region; valid regions are X, Y, Z" and badly from a stack trace.

Identity, permissions and audit

This is where agentic projects most often stall in security review, and the concern is legitimate. If an agent queries data using one service principal with broad access, then every user of that agent effectively inherits that access — your row-level and column-level controls are gone, laundered through a chatbot.

What a defensible setup looks like: the agent runs queries under the identity of the requesting user, so existing catalog permissions apply unchanged; every tool call is logged with inputs, outputs and requester; write actions are scoped to purpose-built service accounts with the narrowest possible grants; and anything consequential — money moving, records changing, messages leaving the building — either runs through human approval or is confined to a reversible sandbox. Decide which actions require a human before the pilot, not after the first bad one.

Cost and latency at big-data scale

Agentic patterns multiply everything. One user question can become a dozen model calls and half a dozen queries as the agent explores, checks and retries. On a lakehouse, a careless exploratory query against a multi-terabyte table isn't a rounding error — it's real compute, and it can be triggered in a loop.

The controls that matter are unglamorous: point agents at pre-aggregated gold tables rather than letting them scan raw history; enforce query limits, row caps and timeouts at the warehouse layer, not in the prompt; cap the number of steps per task; cache aggressively, since agents re-ask the same sub-questions constantly; and route cheap steps to a small model, reserving the expensive one for planning and synthesis. Put a per-task cost budget on the pilot and alert when it's exceeded. You want to learn the shape of that number in week two, not from a quarterly bill.

You are evaluating trajectories, not answers

Evaluation for a single-shot feature is a graded set of questions with known-good answers. That doesn't transfer cleanly to agents, because two runs can reach the same correct number by different routes, and one of those routes may be luck.

So evaluate both: final-answer correctness against a domain expert's spot check, and the trajectory — did it call sensible tools in a sensible order, and did it stop when it should have? Track step count and cost per task alongside accuracy, because the failure mode you're most likely to hit isn't a wrong answer, it's an agent taking fourteen steps to produce a right one at ten times the expected cost. Keep the eval set versioned and rerun it on every prompt or tool change, or you're shipping on vibes.

Where this pays off first

The best first agentic use cases are internal, reversible, and sit on data you already trust. Data engineering toil is a strong candidate: triaging a failed pipeline by reading logs, checking upstream freshness, and drafting a summary with a suggested cause — a task where the agent gathers context and a human decides. Analyst support is another: multi-step ad hoc questions that today queue behind a person, on a curated domain like support tickets or subscription billing.

What we'd avoid for a first project: anything customer-facing, anything that writes to a system of record without review, and anything spanning domains where the data model is still contested. Those aren't permanent exclusions — they're just a bad place to learn what your platform can't yet support.

A sane sequence

Pick one domain with clean, well-understood tables. Fix the metadata and metric definitions in that domain first — that's usually the real project, and it's worth doing whether or not the agent ships. Build three to five narrow tools instead of granting broad access. Run under user identity with full logging. Set step, row and cost ceilings. Write twenty real tasks with known-good answers and grade both output and trajectory. Then, and only then, widen the scope.

If your catalog and semantic layer are already in good shape, that's a matter of weeks. If they aren't, the agent pilot quietly becomes a data platform cleanup — which is worth knowing before you commit to a launch date, not after.

That platform work is most of what makes agentic AI real, and it's what our data engineering and AI team does for clients on Databricks and AWS — governed tables, tool layers, evaluation harnesses, and the cost controls that keep the whole thing affordable. If you're scoping an agentic project and want a straight read on whether your data is ready for it, talk to us.

Originally published on the Zephico blog.

Snowflake to Databricks: what the migration actually costs you

Zephico Technologies — Tue, 04 Aug 2026 23:55:05 +0000

Most Snowflake-to-Databricks migrations get sold on cost and delivered on something else. The credit line item is what gets the project funded, but the teams that finish happy are usually the ones that moved for a different reason: they wanted ML, streaming and GenAI workloads living next to the analytics data instead of shuttling between two platforms. If your only justification is the bill, read the breakeven section below before you commit — the honest number is longer than the deck says.

We're a Databricks shop, and we've written elsewhere about how to choose between the two platforms if you haven't committed yet. This post assumes you have.

What actually changes underneath

The two platforms look similar from a SQL console and are structurally different behind it. The mapping worth internalising before planning anything:

Layer	Snowflake	Databricks
Storage	Proprietary micro-partitions inside Snowflake	Delta Lake files in your own S3/ADLS/GCS bucket
Compute	Virtual warehouses, T-shirt sized	Job clusters, all-purpose clusters, SQL Warehouses, Photon
Governance	Role hierarchy, row access policies, masking policies	Unity Catalog across tables, models, notebooks, dashboards
Sharing	Secure Data Sharing	Delta Sharing (open protocol)
Billing unit	Credits	DBUs, priced differently per compute type

The storage row is the one with the most downstream consequences. On Snowflake, storage and compute are separate line items on the same bill; on Databricks, storage is your cloud provider's problem and your cloud provider's invoice. That's a genuine benefit — the data stays readable by other engines — but it also means your "Databricks cost" and your "data platform cost" stop being the same number, and finance needs to know that before the first invoice arrives.

Pick a strategy before you pick a tool

Three patterns, and the choice determines everything after it:

Lift-and-shift. Replicate schemas one-to-one, translate the SQL, cut over. Fastest, and it faithfully preserves every design compromise you made in Snowflake — including the ones you made because Snowflake billed you that way. Defensible when a contract is expiring and the calendar is the constraint.

Re-platform. Translate the schema into a medallion structure, keep the business logic, redesign the layout. This is where most estates should land: you get Delta's file layout, liquid clustering and Photon working for you rather than inheriting a shape that fought them.

Re-architect. Rebuild the pipelines around Databricks-native patterns — streaming ingestion, Delta Live Tables, jobs instead of tasks. Longest timeline, best economics on the far side, and only realistic if the team is being funded for a platform program rather than a migration ticket.

The mistake is choosing lift-and-shift for speed and then expecting re-architect savings. Snowflake-shaped tables on Databricks compute cost what Snowflake-shaped tables cost.

The effort lives in the SQL, not the data

Copying the data is a solved problem. Bulk export to Parquet, land it in object storage, CREATE TABLE ... USING DELTA, validate. For a large estate it's a scheduling exercise, not an engineering one — and Lakehouse Federation lets you query Snowflake in place while you sequence it, which is worth using as a bridge rather than as a destination.

The code is where the schedule goes. Automated converters — Databricks' own tooling, BladeBridge, the assistant — will get you most of the way through straightforward SQL. What they don't handle is the long tail, and the long tail is disproportionate:

JavaScript stored procedures have no Databricks equivalent. They get rewritten as PySpark or SQL scripting, by hand, by someone who understands what they were doing. If you have dozens of these, that's your critical path.
Semi-structured handling. FLATTEN, LATERAL FLATTEN, Snowflake's VARIANT semantics and its particular null-handling in JSON paths all need deliberate translation. The queries usually run after conversion; they just quietly return different rows.
Timezone and date arithmetic. Snowflake and Spark disagree on enough edge cases that anything doing fiscal-calendar or session-window logic needs test coverage, not eyeballing.
Clustering. Snowflake's automatic clustering has no direct equivalent. You choose Z-ordering or liquid clustering deliberately, per table, based on actual query predicates.

dbt projects port more easily than people expect — adapter swap plus incremental strategy tuning — but BI semantic layers do not. Re-pointing a connection is not a migration; the metrics need re-validation against the old platform, number by number.

Design Unity Catalog before you move anything

Snowflake's role model does not map cleanly onto Unity Catalog's catalog.schema.table hierarchy, and trying to translate it mechanically produces a permission structure nobody can reason about. Do the design first — catalogs, schemas, group ownership, grant boundaries — and migrate into it. We've written up what the migration itself involves and how to run governance afterwards.

Treat this as the one moment you can delete roles. Mature Snowflake estates accumulate roles the way filesystems accumulate temp files, and a migration is the only time you'll have political cover to not recreate them.

Validate in three layers, then run in parallel

Structural checks — row counts, column types, null distributions — catch the copy errors. Semantic checks — aggregates at several grains, not just the total — catch the logic errors. Financial checks mean regulatory and revenue figures matched exactly, because "close enough" on a restated number is a career event for someone in finance.

Then run both platforms in parallel for your critical workloads, comparing outputs daily, for weeks rather than days. Parallel running is expensive and boring and it is the single control that keeps a migration from becoming an incident. Budget for the double bill explicitly so nobody has to justify it mid-project.

The breakeven number, honestly

Migration costs real money: engineering time, parallel running, retraining, and the conversion long tail that always overruns. Against that, savings depend entirely on workload mix. ML and exploratory workloads usually see meaningful reductions. Steady, predictable SQL reporting often lands at rough parity — Databricks compute isn't free, and a SQL Warehouse left running behaves exactly like a Snowflake warehouse left running.

So model it per workload, not as a platform-level percentage. And model the cleanup: a large fraction of any mature estate is tables and jobs nobody consumes. Finding those in discovery and not migrating them is frequently a bigger saving than the platform switch itself.

If you're scoping this, the cheapest useful step is a discovery pass — inventory, dependency map, credit consumption by workload, and an honest count of the stored procedures that will need hand conversion. Zephico is a Databricks Consulting Partner, and our certified engineers run that assessment as a fixed-scope engagement before anyone commits to a timeline. Talk to us if you'd rather know the number before the project starts than halfway through it.

Originally published on the Zephico blog.

When Retool is the right answer (and when it isn't)

Zephico Technologies — Wed, 29 Jul 2026 00:02:20 +0000

We're Retool-certified and we build internal tools with it for clients — so you'd expect us to say Retool is always the answer. It isn't, and knowing where the line sits is most of the value of working with people who use it daily.

Where Retool wins, clearly

Ops consoles over existing data. Refund workflows, order lookups, inventory corrections, support tooling — anything where the data already lives in PostgreSQL, an API, or a warehouse, and the "app" is really forms, tables and buttons with permissions. A tool your ops team needs would take a product squad six weeks in React; in Retool it's days, and the six weeks were never going to be approved anyway. That's the honest comparison: Retool doesn't usually replace a custom build, it replaces nothing getting built.

Approval workflows with audit requirements. Role-based access, audit logs, and SSO come with the platform. For teams facing GDPR requests or SOC 2 audits, replacing "shared spreadsheet plus direct database edits" with an audited tool is often the entire business case.

Tools with fewer than a couple hundred internal users. Retool's pricing and performance are built for internal scale, not consumer scale.

Where we talk clients out of it

Customer-facing anything. The moment external users log in, you're fighting the platform on branding, latency, licensing and auth. Build customer-facing surfaces with a real framework.

Heavy client-side logic. If your "internal tool" is actually an interactive editor, a scheduling engine, or anything with complex client-side state, the visual builder becomes the bottleneck. You'll write thousands of lines of JavaScript in string fields and wish you had a repo. (Some of it can live in modules and version control now, but the gravity is still wrong.)

Core business workflows you'll iterate on for years. Retool is fast to build and fine to maintain, but a workflow that is the business — underwriting, dispatch, pricing — eventually deserves the testability, review culture and hiring pool of a conventional codebase. We've migrated tools out of Retool for exactly this reason, and that's fine: it earned its keep for two years first.

The pattern that works

Start the workflow in Retool while it's still changing weekly. When it stabilizes and hardens into the business, graduate it to a custom build with the requirements now fully known. The Retool version becomes your working spec — the cheapest, most accurate spec you'll ever write.

The wrong question is "Retool or real code?" The right one is "how much certainty do we have about this workflow?" Low certainty, move fast in Retool. High certainty and high stakes, invest in code. We're happy building either — which is exactly why you can trust the recommendation.

Originally published on the Zephico blog.

How to make contract engineers actually work: lessons from the other side

Zephico Technologies — Wed, 29 Jul 2026 00:01:59 +0000

We place contract engineers into teams across North America and Europe, which means we've watched the same engagement succeed at one company and stall at another — with the same engineer. The difference is almost never the engineer. It's a handful of structural choices the client makes in the first two weeks.

Treat week one as an investment, not a cost

The engagements that compound all look the same at the start: the contract engineer gets a working dev environment on day one, a real (small) ticket in the first week, and a named person to ask questions of. The ones that stall spend three weeks "getting access sorted" while the engineer bills hours reading a wiki.

If your security process takes two weeks to provision access, start it before the start date. This sounds obvious. It is skipped constantly.

Give outcomes, not tickets

A contract engineer who receives pre-chewed tickets performs like a junior no matter how senior they are, because all the judgment was spent by whoever wrote the ticket. The teams that get senior output hand over problems: "our nightly pipeline overruns into business hours — own it." Then the engineer's experience actually gets used, and the interesting decisions surface in review where your team can see the reasoning.

Timezone offset is a feature if you design for it

With a team in India and a client in New York, there are roughly four hours of overlap and twenty hours of relay. Teams that fight this — insisting on full-day synchronous presence — burn out the engineer and get the worst of both worlds. Teams that design for it get a genuine advantage: work specced in the client's afternoon is running by their next morning.

The design is simple: overlap hours are for decisions (standups, reviews, pairing on anything ambiguous), non-overlap hours are for execution, and everything decided in a call gets written down because someone will act on it eight hours later.

Measure integration, not utilization

The metric that predicts a successful engagement isn't hours billed — it's how quickly the contractor becomes indistinguishable from the team in your tools. Are they reviewing your engineers' PRs by week three? Are they in the incident channel? Did someone ask them a question this week? If after a month the contractor is still a ticket-taker on the outside of every discussion, the engagement is failing regardless of what the timesheet says.

Say the quiet part in the contract

Notice periods, IP assignment, who owns the laptop, what happens to access on the last day — none of this is awkward if it's written down at the start, and all of it is awkward later if it isn't. A staffing partner who resists clear contract terms is telling you something.

Most of this list is just "treat contract engineers like engineers." The companies that do get senior output within days of the start date — which is the entire reason they went with contractors instead of a six-month hiring pipeline.

Originally published on the Zephico blog.

Databricks Workflows vs Airflow vs Dagster: Picking an Orchestrator

Zephico Technologies — Tue, 28 Jul 2026 21:13:07 +0000

Every data team eventually asks the same question: what runs our pipelines, on what schedule, with what retry logic, and who gets paged when it fails. The answer used to default to Airflow because there wasn't a real alternative. Now there are three reasonable defaults, and they optimize for different things. Picking wrong doesn't break anything on day one — it shows up eighteen months later as either an operations team drowning in scheduler maintenance or an engineering team fighting a platform that won't do what they need it to.

Here's the actual tradeoff, not the vendor pitch version.

Databricks Workflows: the path of least resistance, if you're all-in on Databricks

Databricks Workflows is the orchestrator built into the platform. Jobs, clusters, Unity Catalog permissions, and Workflows all share the same control plane, which means you're not maintaining a separate scheduler, not managing a second set of credentials, and not debugging why an external system can't see a table that Unity Catalog says it can. Task dependencies, retries, cluster reuse across tasks, and job-level alerting all come for free.

The cost is exactly what you'd expect from a platform-native tool: it orchestrates Databricks well and everything else poorly. There's no first-class way to trigger a task in your orchestration DAG that waits on a Salesforce export, calls an internal API, or coordinates a dbt run against a warehouse that isn't Databricks SQL. You can bolt these in with webhooks and external scripts, but you're fighting the tool rather than using it. Workflows also doesn't give you the asset-lineage or testing story that Dagster does — it schedules tasks, not data assets.

If your data platform genuinely is Databricks end to end — ingestion, transformation, ML, serving — Workflows removes an entire category of operational overhead you'd otherwise be paying for nothing. Teams in this position who reach for Airflow anyway usually do it out of habit, not need, and end up running two schedulers where one would do.

Airflow: the incumbent, and the bill that comes with it

Airflow's argument is maturity and reach. It has the largest operator ecosystem of any orchestrator, a decade of production hardening, and a hiring pool that already knows it. If your pipelines touch a dozen systems — a couple of warehouses, three SaaS APIs, an on-prem database, a message queue, and Databricks somewhere in the mix — Airflow (or something with equivalent breadth) is close to mandatory, because it's the only one built from the ground up to sit above heterogeneous systems rather than inside one of them.

What Airflow charges you for that breadth is operational weight. Self-hosted, you're running a scheduler, a metadata database, a web server, and workers, and you own upgrades, DAG-parsing performance as the DAG count grows, and the sharp edges around task isolation. Managed options — MWAA on AWS, Astronomer, Google's Cloud Composer — remove the infrastructure burden but add real cost and still leave you managing DAG deployment and dependency versioning yourself. Airflow's execution model is also fundamentally task-centric: it knows a task ran, not what data it produced or whether that data is fresh, valid, or the same asset another DAG also writes to. Teams that want a data-asset-first mental model layer tools like dbt or add asset-check conventions on top to compensate.

Airflow 3 narrowed some of these gaps — better task isolation, some asset-aware scheduling — but the operational model is still: you run this, or you pay someone to run it for you.

Dagster: asset-based thinking, at the cost of ecosystem size

Dagster starts from a different premise: instead of orchestrating tasks, you declare the data assets your pipeline produces and the dependencies between them (software-defined assets). The scheduler then figures out the DAG from the asset graph. In practice this means significantly better local development — you can materialize a single asset, mock inputs, and test it in isolation without spinning up the whole pipeline — and lineage that's a natural byproduct of how you define pipelines rather than something you bolt on afterward. For teams that have been burned by Airflow DAGs that pass silently but produce garbage data, Dagster's typed I/O and asset-check model is a meaningful upgrade in confidence.

The tradeoff is ecosystem size. Dagster's integration library is smaller than Airflow's, the community and hiring pool are both younger, and if you need an obscure operator for a legacy system, you're more likely to be writing it yourself than pulling it off the shelf. Dagster also orchestrates across heterogeneous systems fine — it's not Databricks-specific — but you're trading Airflow's breadth for a better development model, not getting both for free.

The real decision framework

Ignore the marketing angle from each vendor and the decision comes down to two questions: how much of your stack lives outside Databricks, and how much do you value asset-centric thinking and local testability over ecosystem breadth.

Single-platform-on-Databricks shop, most pipelines are notebooks and Databricks jobs. Use Workflows. You'll be maintaining fewer systems and every feature you'd otherwise build (retries, alerting, cluster lifecycle) is already there.
Orchestrating across many heterogeneous systems — multiple warehouses, SaaS APIs, on-prem systems, Databricks as one node among several — Airflow's breadth usually wins, especially if your team already knows it or you're prepared to pay for a managed instance.
Same heterogeneous situation, but you're building new and value strong typing, asset lineage, and a real local dev/test loop over ecosystem size — Dagster deserves a serious evaluation, particularly for greenfield platforms where you're not migrating years of existing DAGs.

The honest answer really is "it depends on your existing stack," and any framework that skips that and hands you one universal answer is selling something. The one mistake we see repeatedly is teams running Airflow purely to orchestrate Databricks jobs, paying the full operational cost of a general-purpose scheduler to do a job Workflows does natively — that's worth a hard look before you commit to a migration or a managed Airflow contract.

If you're mid-evaluation and want a second opinion on which of these fits your actual pipeline mix, our Databricks data engineering team has built and migrated orchestration on all three. Get in touch and we'll walk through your stack with you.

Originally published on the Zephico blog.