DEV Community: Andrew Tan

Iceberg Is Easy to Demo. The Maintenance Bill Arrives Later.

Andrew Tan — Thu, 24 Sep 2026 08:08:26 +0000

Open table formats solve portability. They do not solve operations. The real work starts after the launch post: compaction, retention, multi-catalog sprawl, and keeping batch and streaming paths consistent.

The demo that never ages

Here is how the Iceberg pitch usually goes:

An engineer stands in front of a screen and runs a query against a Parquet dataset. Then they run the same query against an Iceberg table. The results are identical. The audience nods. Then the engineer shows time travel—rolling back to a previous snapshot—and the room actually murmurs. Somebody asks about schema evolution. The engineer adds a column, rewrites nothing, and the old queries still work. The committee is sold.

Six months later, the same team is in a different room. This one has no screen. Just a spreadsheet, a growing list of alerts, and a question nobody asked during the demo:

Who owns the compaction?

The demo showed what Iceberg makes possible. It did not show what Iceberg makes your problem.

What the spec promises vs. what the team inherits

Open table formats are a genuine advance. Portability across engines, snapshot isolation, partition evolution, and hidden partitioning are real capabilities that solve real pain. If you are trying to escape a vendor-locked warehouse, Iceberg, Delta Lake, and Hudi are the best paths out.

But the spec is not the system. The spec says how the metadata should be laid out. It does not say how your team should keep that metadata from growing without bound, how to coordinate compaction across multiple writers, or what happens when two different query engines disagree on which snapshot to read.

Here is what the demo skips:

Compaction is not automatic

Every insert, update, and delete creates new files and new metadata entries. Left alone, a high-velocity table accumulates thousands of small files. Query performance degrades. Metadata files bloat. The table that looked fast in the demo starts timing out in production. Somebody has to schedule compaction, monitor it, tune it, and handle the failures when two jobs try to rewrite the same partition.

Catalog sprawl is real

Iceberg tables need a catalog: Hive, Glue, Nessie, Polaris, a custom REST service. Each catalog has its own consistency model, its own authentication, its own upgrade cycle. A team that adopts Iceberg to avoid vendor lock-in often ends up managing two or three catalog systems instead of one warehouse. The lock-in shifts from the storage format to the catalog layer.

Retention policy is a distributed problem

When a snapshot expires, Iceberg marks the data as unreachable. But the underlying files still exist in object storage until something deletes them. That "something" is your problem. If you set aggressive retention to save on storage, you might lose the ability to roll back when a downstream job produces bad results. If you keep everything, your storage bill compounds while your S3 bucket turns into an archaeological dig.

Batch and streaming see different tables

A batch job writing to Iceberg produces large, well-structured files. A streaming job produces small, frequent files. If both paths write to the same table, the query planner has to handle two radically different file layouts. The streaming path needs frequent compaction to stay readable. The batch path needs stable files to avoid recomputation. Coordinating these two rhythms in one table is harder than the architecture diagrams suggest.

The demo showed a single writer and a single reader. Production rarely works like that.

Why "open" creates new fragmentation

The promise of open table formats is interoperability. Query the same data from Spark, Trino, Flink, DuckDB, Snowflake, BigQuery. In practice, each engine supports a different subset of the spec at a different maturity level.

One engine supports position deletes but not equality deletes. Another supports time travel but only for tables written by its own catalog. A third supports partition evolution but requires a specific metadata version that breaks older readers. The table is "open" in theory. In practice, it is coupled to the specific combination of engines and catalog versions your team happens to run.

This is not a criticism of the projects themselves. Iceberg, Delta, and Hudi are moving fast and improving rapidly. The issue is that teams adopt them expecting liberation and discover a new kind of operational surface area. Instead of one vendor to blame, they have a matrix of version compatibilities to manage.

The hidden cost is cognitive load. Your data engineers now need to understand not just their pipelines, but the compaction scheduling, catalog consistency model, metadata format version, and engine-specific behaviors of every tool that touches the table. That expertise does not come from a demo.

The checklist nobody runs before adopting

If your team is evaluating an open table format, here are the questions that matter more than query performance in a benchmark:

Who owns compaction, and what happens when it fails?

Compaction is not a one-time setup. It is a continuous background process that competes for the same compute resources as your production queries. If compaction falls behind, queries slow down. If compaction corrupts a partition, recovery is manual and stressful. You need an owner, a runbook, and a way to detect when compaction is not keeping up.

What is your catalog exit strategy?

Catalogs are the real lock-in point. If you commit to Glue today, can you migrate to Nessie or Polaris later without rewriting table paths and reconfiguring every downstream job? Most teams do not test this until they are forced to.

How do you handle late data and backfills?

Batch backfills and streaming late arrivals both rewrite historical partitions. Open table formats handle this better than raw Parquet, but they do not eliminate the coordination problem. If a backfill runs while a streaming job is appending to the same partition, you need to understand isolation semantics, retry behavior, and exactly what each engine does when it sees a conflict.

What is your metadata growth plan?

Metadata files are small but they multiply. A table with daily snapshots and hourly compaction can generate thousands of metadata files per month. Object storage is cheap, but LIST operations are not free. Some query engines load the full metadata tree into memory. At a certain scale, metadata itself becomes a performance bottleneck.

Who gets paged when a query returns wrong results?

Snapshot isolation is great until someone reads from the wrong snapshot because the catalog was briefly inconsistent. Or because a streaming job committed an incomplete batch. Or because two engines interpreted the same metadata file differently. Debugging these issues requires expertise in the format, the catalog, and the specific engine. The on-call rotation just got deeper.

If you cannot answer these questions with something more specific than "we'll figure it out," you are not adopting a technology. You are taking on a new operational domain.

Where the tradeoff is worth it

I want to be fair. There are situations where open table formats are the clear right choice:

You are actively escaping vendor lock-in.

If your warehouse provider is raising prices, deprecating features, or limiting egress, the portability of an open format is worth the operational overhead. The alternative is staying trapped.

You genuinely need time travel and rollback.

Some workloads—especially in regulated industries or financial services—require the ability to reconstruct historical state exactly. The snapshot model is not a nice-to-have here. It is a compliance requirement.

You run multiple compute engines on the same data.

If your analytics team uses Spark, your BI team uses Trino, and your ML pipeline uses DuckDB, a shared open table format eliminates the extract-transform-load dance between systems. The coordination cost is real, but it is lower than maintaining three separate copies of the same dataset.

You have the team for it.

If you have engineers who understand metadata formats, compaction strategies, and catalog consistency models, the operational burden is manageable. If you do not, you are outsourcing the expertise to consultants and hoping they stay available.

Where layline.io fits

At layline.io, we do not sell a table format. We sell a processing runtime that handles both batch and streaming workloads on the infrastructure you already have. That includes open table formats when they make sense, and traditional storage when they do not.

The reason this matters: a lot of teams adopt Iceberg because they need batch and streaming to coexist, and they have been told that open table formats are the only way to unify them. That is not true. The unification happens at the processing layer, not the storage layer. If your runtime can write well-structured files in batch mode and handle micro-batches in streaming mode—while managing compaction, backfills, and late data in the same workflow—the storage format becomes a configuration choice, not an architectural commitment.

We see teams who adopted Iceberg for the right reasons and then discovered that the hard part was never the format. It was the operational coordination around it: keeping batch and streaming paths consistent, handling schema changes without breaking downstream consumers, and making sure the same business logic produces the same results regardless of when the data arrives.

That is the problem we focus on. The table format is a detail. The operational model is what determines whether the system works at 2 AM on a Tuesday.

The question to ask before the demo

Next time a vendor shows you a slick Iceberg demo—time travel, partition evolution, engine switching that looks effortless—ask them this:

"Show me the compaction schedule. Show me the catalog failover. Show me what happens when a streaming job and a batch backfill hit the same partition. Show me the storage bill after six months of metadata growth. And show me who gets paged when a query engine reads a snapshot that was partially written."

If the answer is a reference to the documentation, you are looking at the easy part. The hard part is what you will own for the next three years.

Open table formats are not a scam. They are a real, valuable technology with a genuine operational cost that the marketing rarely mentions. The teams that succeed are the ones who price that cost upfront, assign ownership before the first table is created, and treat the format as one component of a larger operational system—not as a magic layer that makes infrastructure problems disappear.

The demo is the beginning. The maintenance bill is where the story actually starts.

If your team is weighing open table formats and trying to understand the full cost of ownership, get in touch. We work with teams on exactly this problem—and the operational reality is usually more manageable than the fear, once you know what to plan for.

Compliance Fails in Transit: Why Data Governance Has to Move Into the Pipeline

Andrew Tan — Tue, 15 Sep 2026 11:04:53 +0000

The audit that never finds the real problem

Here's a scene that plays out in regulated industries every quarter:

The compliance team runs their governance scan. The data catalog shows classification tags on every warehouse table. The lakehouse has column-level security policies. The BI tool enforces row-level access. Everyone signs off. The auditors leave happy.

Meanwhile, a CDC feed is replicating customer PII from a transactional database to an analytics cluster. Nobody classified the stream itself — they classified the destination table, which is fine, except the stream passes through a staging topic where three different services subscribe. One of them is an AI enrichment job that sends fragments of the data to an external LLM API. The API call doesn't show up in the data catalog because the catalog only catalogs what lands, not what moves.

The governance program is impeccable at rest. It's nearly blind in motion.

This isn't a tooling failure. It's a category error. We've built governance around the assumption that data lives in tables and files, and that if we control the endpoints, we control the story. But modern pipelines don't work like that. Data is in constant motion — across brokers, through transformations, into model training sets, out to partner APIs — and most governance frameworks haven't caught up.

Why at-rest governance misses the hard cases

Batch-era governance made sense for batch-era architecture. Data moved in scheduled loads. You knew when it arrived. You could scan it, classify it, and apply policy before anyone queried it. The pipeline was basically a delivery truck, and governance happened at the loading dock.

Streaming and CDC broke that model. Data moves continuously. It gets joined in flight. It gets enriched by external services. It gets filtered, split, and routed to multiple consumers before it ever touches a governed destination. By the time it lands, the sensitive parts may have already leaked, been copied, or sent somewhere the catalog can't see.

The gap shows up in specific, painful ways:

Classification after landing is often too late. If a CDC stream carries unmasked credit card numbers through three intermediate topics before they reach the warehouse, the warehouse column-level policy is a Band-Aid on a bleed that happened upstream.

Batch governance tools don't understand streaming semantics. A data catalog can tell you what's in a table. It can't tell you that a streaming join is pulling PII from one topic and correlating it with behavioral data from another, creating a new composite dataset that nobody reviewed.

Lineage breaks at the seams. Most lineage tools trace table-to-table relationships. They don't trace the API call that enriches a record mid-pipeline, or the model that trains on a snapshot of the stream, or the reverse ETL job that pushes derived data back to a CRM. The map looks complete until you follow a record through its actual lifecycle.

Retention policies ignore movement-time copies. You set a 90-day retention on the warehouse. Great. But the stream consumer that materialized a view in Elasticsearch? The S3 bucket that the pipeline writes to for replay? The partner API that received a copy? Retention is only as strong as its weakest replica, and most governance programs don't know where the replicas are.

What governance-in-motion actually means

Moving governance into the pipeline doesn't mean replacing your data catalog or your lakehouse security model. It means extending policy enforcement to the places where data is actually moving.

Here are the four capabilities that matter:

Policy at the boundary

Instead of classifying data after it arrives, classify it when it enters the pipeline. A CDC feed from a customer database should carry its classification tags with it — PII, financial data, health records — and those tags should persist through every reshape, join, and routing decision. If a downstream consumer tries to send tagged data to an unapproved destination, the pipeline should block it, not log it and move on.

This sounds obvious, but most pipelines don't do it. The metadata that matters for governance — classification, consent flags, retention requirements — is usually stripped out during normalization or stored in a separate catalog that the runtime doesn't consult.

Approval gates for high-risk flows

Some data movements shouldn't happen without a second look. A pipeline that starts replicating a new table to an external analytics tool. A stream that begins including a newly added column with geolocation data. An AI workflow that wants to send customer transcripts to a third-party model.

These aren't failures. They're normal operations. But they're also the moments where governance risk concentrates. The right model isn't to block everything and wait for a ticket. It's to let low-risk flows proceed automatically and flag high-risk ones for approval — with the approval gate embedded in the pipeline itself, not in a separate workflow tool that the engineer may or may not remember to use.

Retention and deletion that follow the data

When a customer exercises their right to be forgotten, or when a retention period expires, the request needs to reach every copy of the data — not just the warehouse table. That includes stream replays, materialized views, model training snapshots, and partner API caches.

In practice, this means the pipeline runtime needs to track where data was sent and maintain the mapping between a record's identity and its replicas. Most streaming platforms don't do this. They treat each message as independent, with no memory of where it came from or where it went. Governance in motion requires a different model: messages carry identity, and the runtime maintains provenance.

Lineage that includes transformations, not just tables

Table-to-table lineage is useful for batch pipelines. For streaming and CDC, it's incomplete. You need to know that a particular enrichment step appended third-party data, that a join correlated two datasets that were supposed to stay separate, or that a filter silently dropped records that should have been retained for compliance.

This means lineage can't be a post-hoc scan of query logs. It has to be built into the runtime, capturing what each step did to the data as it happens.

The rollout path: start with interfaces

Nobody rebuilds their governance model in one project. The teams that succeed start small and expand.

The right place to begin is the pipeline boundary — the points where data enters or leaves your system. CDC feeds from production databases. API calls to external services. Reverse ETL jobs that push data back to operational tools. These are the highest-risk, highest-visibility interfaces, and they're where governance failures show up first.

Pick one interface. Add classification tagging at the source. Add a policy check before the data exits. Add lineage tracking for that one flow. Prove it works. Then expand to the next interface.

The teams I see doing this well share one trait: they treat governance as a runtime concern, not a documentation concern. The policy isn't written in a wiki and hoped for. It's enforced by the pipeline, tested in CI, and versioned with the code. When the policy changes, the pipeline changes. When the pipeline changes, the policy is revalidated.

Where this gets harder than it sounds

I want to be honest about the obstacles, because "just move governance into the pipeline" is easy to say and hard to do.

Existing tools aren't built for it. Most data catalogs, security scanners, and lineage tools were designed for batch warehouses. They expect scheduled scans, not continuous streams. They expect tables, not topics. Extending them to cover motion usually means custom integration work that the vendor doesn't support.

Performance matters. Adding classification checks, policy lookups, and lineage logging to a streaming pipeline adds latency. For high-throughput flows, the overhead has to be negligible — which means policy decisions need to be cached, evaluated asynchronously, or pushed to the edge of the pipeline where they don't block the critical path.

Organizational ownership is fuzzy. Data governance is often owned by a compliance team that doesn't write pipeline code. Platform engineering owns the runtime but doesn't set policy. Getting these teams to agree on what "governance in motion" means, who maintains it, and who gets paged when it breaks is often harder than the technical implementation.

The standards are still forming. There's no universal protocol for attaching governance metadata to streaming records. No standard API for pipeline policy enforcement. The vendors are building their own models, and they don't interoperate well. If you're betting on a specific approach, you're partly betting on which vendor's model wins.

Where layline.io fits

I won't pretend we solve every part of this. At layline.io, we've focused on the runtime layer: a processing engine for both batch and streaming that can carry metadata through transformations, enforce policies at pipeline boundaries, and maintain lineage across the full flow.

The bet is that governance shouldn't be a separate system sitting next to your pipeline, parsing logs and hoping it catches everything. It should be part of the pipeline itself — embedded in the same runtime that moves the data, enforced by the same code paths that transform it, visible in the same observability layer that tells you whether the pipeline is healthy.

This matters because the alternative is fragmentation. One tool for cataloging, another for streaming, another for batch, another for lineage, another for policy management. Each integration point is a seam where governance can leak. The teams I talk to are tired of seams.

We're not the only ones building in this direction. Confluent is pushing stream governance. Airbyte is framing sovereignty as an architectural problem. The whole market is moving toward the same conclusion: data in motion needs the same controls as data at rest. We're building our piece of that stack.

The question to ask your governance team

Next time your compliance team signs off on a clean audit, ask them this:

"Can you show me where customer PII goes between the production database and the warehouse? Not the endpoints — the path. Every topic, every enrichment call, every replica."

If the answer is a diagram with two boxes and an arrow, you don't have governance. You have hope.

The good news is that fixing it doesn't require ripping everything out. It requires moving the control point from the destination to the journey. Start at the boundaries. Add policy to the pipeline. Make lineage part of the runtime. The tools are getting better, the patterns are getting clearer, and the teams that figure this out first will have a genuine operational advantage — not just cleaner audits, but fewer incidents, faster recovery, and data infrastructure they can actually trust.

Fresh Enough for an AI Agent? The Data Freshness Budget Nobody Defines

Andrew Tan — Tue, 08 Sep 2026 13:09:25 +0000

Most teams talk about AI context as a storage problem. The harder question is whether the data is fresh enough for the action the agent is about to take.

A lot of AI infrastructure conversations are still aimed at the wrong bottleneck.

Teams debate vector databases, prompt caching, long-term memory, MCP servers, and which model should sit behind the agent. All of that matters. None of it answers the question that actually determines whether an agent is safe to use in production.

How fresh is the data when the agent decides to do something?

That question sounds boring. It is not boring when the agent sends the wrong refund, approves the wrong order, escalates the wrong incident, or calls the wrong tool because it is looking at a customer record from twenty minutes ago.

Most data teams already understand quality, lineage, and schema drift. Freshness gets treated like a nice-to-have until agents enter the picture. Then freshness stops being a dashboard preference and turns into an operational boundary.

Context is not the same as permission

This is the part many teams skip.

An agent can have plenty of context and still have the wrong context for the action you want it to take. Ten million rows in a warehouse do not help if the one field that matters changed three minutes ago and your sync runs every hour.

That is the difference between informative data and actionable data.

Informative data helps an agent explain what happened last week. Actionable data lets it decide what should happen right now.

Those are not the same workload.

A support copilot summarizing the last five tickets can tolerate some lag. An agent that decides whether a refund already happened cannot.

A sales assistant drafting an account brief can work from nightly CRM syncs. An agent that routes a live lead based on current product usage probably needs data that is a few minutes old at most.

A finance bot preparing a monthly variance summary can stay batch. An agent that freezes a suspicious payout cannot.

The mistake is treating all AI context as one category called "AI-ready data."

It is not one category. It is a stack of decisions with different freshness requirements.

The freshness budget

The cleanest way to think about this is a freshness budget.

Every agent task has a maximum acceptable lag between what happened in the source system and what the agent sees when it acts.

That lag budget depends on the consequence of being wrong.

Here is a simple version:

Agent task	Typical freshness budget	What happens if you miss it
Weekly account summary	24 hours	Slightly stale narrative
Internal KPI Q&A	1 to 4 hours	Confusing answers, low trust
Sales lead routing	5 to 15 minutes	Bad prioritization, slower follow-up
Customer support refund decision	Under 5 minutes	Duplicate refunds, policy mistakes
Fraud or risk intervention	Seconds to 1 minute	Real money loss

This is not a universal table. It is a forcing function.

Most teams never define these numbers at all. They just say they want "real-time AI" or "AI-ready pipelines" and hope the stack sorts itself out later.

It will not.

If you do not define a freshness budget, the default budget becomes "whatever the pipeline already does." That is usually an accident, not a design choice.

Batch is still fine for a lot of AI work

I do not think the answer is to push every workload into streaming.

That is expensive. It also creates a new class of operational problems if the team does not need it.

Some AI use cases are perfectly happy on batch:

Writing internal summaries
Preparing research briefs
Tagging support conversations for trend analysis
Drafting renewal prep notes
Enriching planning docs
Creating weekly executive recaps

These tasks benefit from completeness more than immediacy.

You usually want the full record, not the newest event from fifteen seconds ago.

This matters because a lot of AI messaging right now frames the future as one giant real-time system. That is a good way to overspend.

The better question is narrower: where does low-latency data actually change the quality of the action?

If the answer is nowhere, keep it batch.

If the answer is specific interfaces, move those interfaces first.

The dangerous middle

The real trouble is not clearly batch and not truly real-time.

It is the mushy middle where teams have pipelines that update every hour, or every thirty minutes, or whenever a connector feels like catching up, and then they hand that data to an agent with permission to act.

That is where mistakes get expensive.

Hourly freshness sounds decent until you map it to the workflow.

If a customer upgrades at 10:02 and the agent sees the old plan until 11:00, you have almost an hour where it can deny an entitlement the customer already paid for.

If an order is canceled at 2:11 and your inventory assistant does not learn that until 3:00, it can reorder stock you no longer need.

If a chargeback flag lands in one system before another, your risk agent may see a healthy account and approve the next transaction with full confidence.

Nothing is obviously broken in those examples. The jobs ran. The tables updated. The dashboard probably looks fine.

The problem is that the action window is tighter than the data window.

That gap is where agent failures live.

Approval gates are a freshness strategy too

There is another point that gets lost in the rush to automate.

Sometimes the right answer is not faster data. Sometimes the right answer is an approval gate.

If an agent works on stale or ambiguous context, you do not always need to block the use case forever. You may just need to change the last step.

Let the agent gather data. Let it draft the response. Let it recommend the action. Then require a human approval if the freshness budget is not met or if the decision touches money, compliance, access, or customer-facing risk.

This is not a failure of automation. It is part of the design.

Good AI systems do not just think about model quality. They think about when not to act alone.

That is especially true when the underlying data arrives through a mix of batch syncs, CDC pipelines, event streams, and external APIs that all move at different speeds.

You need freshness by interface, not by platform slogan

A lot of vendor positioning blurs this on purpose.

The promise usually sounds something like this: connect all your data, feed your agents, unlock real-time decisions.

Fine. But which decisions?

You do not need one giant freshness target for the whole company. You need freshness targets by interface.

Start with the places where an agent crosses from advice into action:

approving or denying something
sending a customer communication
changing access or entitlements
moving money
opening or closing incidents
triggering downstream systems automatically

Those interfaces deserve explicit lag limits, monitoring, fallback behavior, and ownership.

If the pipeline misses the budget, what should happen?

Maybe the action pauses.

Maybe the agent can still draft but not send.

Maybe it can operate on a narrow safe subset of tools.

Maybe it needs a human in the loop.

The point is to define this before the first incident teaches it to you.

What this means for the data stack

Once you define freshness budgets, the stack conversation gets easier.

You can stop arguing about batch versus streaming as ideology.

Some interfaces need event-driven processing. Some need CDC with tight SLAs. Some are fine with scheduled syncs. Some need a hybrid path where the same workflow handles both historical backfills and low-latency updates.

That last category is the one that gets painful fast if your architecture splits batch and streaming into separate systems.

Now you are maintaining two versions of the same business logic. One answers historical questions. One powers live actions. They drift. Your agent gets inconsistent state depending on which path it touched.

That is why I think the better long-term design is not "real-time everywhere." It is one runtime that can handle both batch and streaming, plus the orchestration around them, without forcing the team to rebuild the workflow every time a freshness requirement changes.

That is also where layline.io fits. The point is not to turn every pipeline into a stream. The point is to let teams tighten freshness where the action requires it, keep batch where it is enough, and manage both in one operational model.

The practical test

If your team is rolling out AI agents right now, ask one question for every agent action:

What is the maximum age of data we can tolerate before this action becomes unsafe, wrong, or embarrassing?

Write the number down.

If nobody can answer it, you do not have an AI problem yet. You have a requirements problem.

And if the answer is "it depends," that is fine. Break it down by interface until it stops depending.

The teams that do this well will not be the ones with the loudest AI stack. They will be the ones that know which decisions need seconds, which need minutes, which can wait until tomorrow, and where a human still belongs in the loop.

That sounds less exciting than the usual agent demos.

It is also the difference between a helpful system and one that creates a new kind of on-call rotation.

Why 85% of Enterprises Aren't Ready for AI

Andrew Tan — Tue, 01 Sep 2026 13:50:43 +0000

Fivetran's 2026 benchmark found 85% of enterprises running agentic AI on infrastructure that isn't ready. The model isn't the problem. The data pipelines feeding it are.

Here's a pattern showing up in an increasing number of places:

A company spends six months evaluating LLMs: They run benchmarks, negotiate contracts, build a proof of concept. The model looks great. Then they deploy it to production and the agents start hallucinating in ways that have nothing to do with the model at all.

The AI is fine, but the data feeding it isn't.

This is what Fivetran's Enterprise Data Infrastructure Benchmark Report 2026 is actually measuring when it says 85% of enterprises aren't ready for agentic AI. Not that their models are wrong, but that their pipelines are wrong.

What "not ready" actually means

The phrase "AI-ready infrastructure" gets used to sell a lot of things. It usually means something vague about cloud scalability. That's not what the Fivetran report is measuring.

The three concrete problems they identified:

Stale data. The agent is reasoning over yesterday's state. For a customer service agent, that means it doesn't know a refund was already issued. For a fraud detection agent, it's working from patterns that are 18 hours old. Batch pipelines running hourly or nightly can't support agents that need to act on what's happening now.

No schema enforcement. Source systems change constantly. Columns get renamed, types get widened, new fields appear. If your pipeline doesn't enforce schema contracts downstream, an agent can receive malformed data that looks valid — and confidently act on it.

No observability. Most pipeline monitoring tells you whether a job ran. It doesn't tell you whether the data is correct. An agent can be consuming data from a pipeline that's technically running but silently dropping 20% of events. You won't know until someone notices the agent is acting strangely, and by then the damage is done.

These aren't new problems. They're the same data quality issues that have plagued analytics teams for years. What changed is the blast radius. A data quality problem in a BI dashboard is a bad chart. A data quality problem in an agentic system is an autonomous decision made on bad information.

The stack mismatch

The AI-ready data stack problem is a deep mismatch in architecture assumptions.

Most enterprise data stacks were built around batch processing. Nightly ETL jobs. Daily warehouse refreshes. Dashboards that update every morning. The entire system was optimized for throughput over freshness.

AI agents have different requirements. They need data that's current, not just accurate. They need to act on what's happening in the last few seconds or minutes, not the last twelve hours. And when they're wrong, they need the system to catch it — not just log it and move on.

The companies that are ready — the 15% — didn't necessarily replace their entire stack. Most of them shifted to real-time where it matters and kept batch where it makes sense. The customer order stream runs in real-time. The quarterly cost accounting still runs nightly. The difference is intentionality: they made explicit decisions about which data needs freshness and built pipelines accordingly.

The three properties that actually matter

There's a lot of noise about what "AI-ready" infrastructure requires. Usually it's a vendor's checklist that conveniently maps to their product. So here's the version that maps to the actual failure modes:

Freshness. Not just "real-time for everything" — that's expensive and often unnecessary. But for the data your agents actually act on, you need to know the lag and have guarantees about it. If your customer data is 4 minutes old, fine — as long as your agent knows that and acts on it. What breaks agents is when they assume data is current and it isn't.

Consistency. Agents often combine data from multiple sources to make a decision. If those sources are running on different schedules or at different freshness levels, you get subtle inconsistencies that are hard to debug. A customer lookup says the account is active; the transaction stream hasn't caught up and shows it as pending closure. The agent makes a decision that's correct for each source individually but wrong for the combined state.

Observability at the event level. Pipeline monitoring tells you about jobs. Agent reliability requires monitoring individual events. Are events being processed or dropped? Is the schema matching what the agent expects? Are there bursts that overwhelm downstream consumers? This is a different class of monitoring than most teams have built.

What this means for the build vs. buy decision

The AI wave is surfacing a bill that's been accumulating in a lot of engineering orgs: the cost of treating data infrastructure as a solved problem.

Teams that built custom batch pipelines two years ago and called it done are now facing a hard choice: retrofit real-time onto a system that wasn't designed for it, or rebuild. Neither option is cheap. Retrofitting tends to produce the two-pipeline problem — a batch system for history, a streaming system for real-time, two codebases doing roughly the same thing with slightly different logic and perpetually diverging results.

The teams handling this well are the ones who don't have to make that choice. When batch and streaming run as the same pipelines with the same tooling, switching a workflow from hourly to real-time is a configuration change, not a rewrite. The observability, the schema enforcement, the failure handling — it comes with the platform, not as a custom build on top.

That's what layline.io is built around. Not just real-time for its own sake, but the ability to make explicit, intentional decisions about data freshness across the stack — without maintaining two separate systems to get there.

The question to sit with

The 85% number is striking. But the more interesting question is: how many of those enterprises know they're in that 85%?

The teams that are in trouble aren't usually the ones with obviously broken pipelines. They're the ones with pipelines that appear to work — jobs running, dashboards loading, no active incidents — but with silent reliability issues that only show up when an AI agent starts making consequential decisions on the data.

If your agents are acting strangely and you've already ruled out the model, look at the data.

The Hidden Costs of Building Your Own Batch-Streaming Integration Layer

Andrew Tan — Tue, 25 Aug 2026 10:36:20 +0000

The Hidden Costs of Building Your Own Batch-Streaming Integration Layer

With AI-assisted coding, building your own data pipelines looks cheaper than ever. But the real costs aren't in the initial build—they're in the maintenance, the on-call rotations, and the accumulated complexity that compounds over time.

Here's a conversation that keeps happening:

Engineering Manager: "We need a new data pipeline for the customer analytics project."

Senior Engineer: "I can build that. With Cursor and Copilot, I can have the core logic done in a couple of days."

EM: "What about maintenance?"

SE: "It's just a Python script with some Airflow orchestration. How hard can it be?"

Three months later, the engineer who built it is on vacation, the pipeline is failing silently, and nobody can figure out why the customer segment counts don't match the source system. The "simple Python script" has grown to 2,400 lines, touches three different databases, and has exactly zero documentation about what the business logic is actually supposed to do.

The AI coding revolution has made the build decision feel almost free. What it hasn't changed is the own decision — and that's where most of the cost lives.

The Honest Accounting

When teams estimate the cost of building their own data integration layer, they usually model something like this:

Cost Item	Estimated
Initial development	2-3 weeks of engineer time
Infrastructure	Existing Kubernetes cluster
Maintenance	"Just keep it running"
Total first-year cost	~$30K loaded

Here's what the spreadsheet actually looks like after twelve months:

Cost Item	Actual
Initial development	4 weeks (scope crept)
Infrastructure	$8K/year in compute, storage, network
On-call burden	15-20 hours/month paging, debugging, fixing
Schema drift incidents	3 major, 8 minor (data quality failures)
Failed retry handling	Built ad-hoc, never quite right
Documentation debt	Still zero, now critical
Knowledge silo risk	One engineer understands it
Total first-year cost	~$85K loaded + opportunity cost

The gap isn't because engineers are bad at estimation. It's because the spreadsheet only captures the work you can see upfront. The real costs accumulate invisibly: the 2 AM pages, the "quick fixes" that become permanent, the subtle data corruption that takes days to detect.

The Two-Pipeline Problem

There's a specific failure mode that hits teams building their own batch-streaming infrastructure: the divergence problem.

You start with batch. It's straightforward. You write a job that runs every hour, extracts data, transforms it, loads it somewhere. Works fine.

Then the business asks for real-time. "Can we get this data in seconds instead of hours?"

So you build a streaming pipeline. Kafka, maybe Flink or Spark Streaming. It consumes the same source data and delivers to the same destination. But the transformation logic is different — streaming has different constraints, different state management, different failure modes. You can't just port the batch code over.

Now you have two pipelines doing roughly the same thing. They produce slightly different results because the batch join is outer and the streaming join is inner, or because the batch job handles late data differently than the streaming window. When someone asks why the numbers don't match, you have to debug both systems.

Six months in, you've got:

Two codebases to maintain
Two sets of infrastructure to monitor
Two failure modes to understand
Two on-call rotations (or one very unhappy person)
And one persistent question: why can't we just have one pipeline?

The honest answer: because batch and streaming are genuinely different paradigms, and most DIY stacks aren't built to unify them.

The Hidden Complexity Multipliers

Beyond the obvious costs, there are three complexity multipliers that don't show up in initial estimates:

Schema Evolution

Your source system changes. A column gets renamed. A type gets widened. A new nullable field appears. In a managed platform, this is handled. In your custom pipeline, it's a code change, a deployment, and a prayer that you didn't break downstream consumers.

The real cost isn't the change itself. It's the coordination: notifying every team that consumes this data, updating their schemas, testing the integration, rolling back if something goes wrong. A two-hour code change becomes a two-week project.

Failure Handling at Scale

A simple retry loop is easy. Exponential backoff, a dead letter queue, some alerting — you can build that in an afternoon.

But production failure handling is fractal. What happens when the destination is down for an hour? What happens when a message is too large? What happens when a schema mismatch causes a parse failure? What happens when the same event gets delivered twice? What happens when network partitions create split-brain situations?

Each edge case needs handling. Each handler needs testing. Each test needs maintenance. The "simple retry logic" grows into a distributed systems concern that nobody on the team has deep expertise in.

Observability Gaps

You need to know: Is the pipeline running? Is it keeping up with the source? Are events being processed or dropped? What's the latency? What's the error rate? What's the cost per million events?

Building this visibility isn't just adding a metrics endpoint. It's designing the right metrics, building the dashboards, setting the right alerts (not too noisy, not too quiet), and training the team to interpret them. It's another system to build, maintain, and debug.

When Building Actually Makes Sense

I want to be fair. There are situations where building your own integration layer is the right call:

You have extremely specific requirements that no vendor handles well — unusual data formats, custom security constraints, exotic deployment environments.

You have the team for it — distributed systems engineers who've operated Kafka at scale, who understand exactly-once semantics, who've debugged backpressure problems at 3 AM.

It's a genuine differentiator — the data processing layer is core to your product, not just infrastructure. You're not building a pipeline; you're building a competitive advantage.

You're at a scale where vendor costs exceed build costs — though be honest about what "build cost" includes. Most teams underestimate by 2-3x.

For everyone else, the calculation usually favors buying — if you account for the full cost of ownership.

The Vendor Evaluation That Actually Matters

If you're comparing vendors, the feature matrix is the wrong place to start. Most platforms have similar capabilities on paper. What matters is the operational model:

How do they handle the 2 AM problem? When something breaks in production, who gets paged? Is it your team debugging their infrastructure, or their team debugging your pipeline?

What's the migration path if you leave? Data pipelines are sticky. Understand what it costs to extract your logic and move it elsewhere.

Do they unify batch and streaming? Or will you end up with two pipelines anyway, just in someone else's infrastructure?

What's the real TCO? Include training, integration time, the cost of waiting for features you need, and the opportunity cost of engineering time spent managing the platform.

Where layline.io Fits

I won't pretend this is an unbiased take. At layline.io, we built a platform specifically for teams who've done the honest accounting and decided that building isn't the right call.

The core bet: batch and streaming shouldn't be separate pipelines. They should be the same workflows, the same tooling, the same team. When you need real-time, you don't rebuild. You adjust a configuration.

The operational burden sits with us. Schema evolution, failure handling, observability — that's the platform's job, not yours. Your team focuses on the business logic, not the distributed systems plumbing.

Is it cheaper than building your own? That depends on how honestly you account for the build cost. If you're counting two weeks of development and calling it done, probably not. If you're including the on-call rotation, the maintenance burden, the schema drift incidents, and the opportunity cost of engineers not building product features — then usually, yes.

The Question to Ask

Before your team commits to building, ask this:

"If we build this ourselves, who owns the 2 AM page when it breaks six months from now? And do they know what they're signing up for?"

If the answer is clear and everyone understands the commitment, build away. If there's hesitation, or if the answer is "we'll figure that out later," do the honest accounting. The numbers might surprise you.

The News Everyone Saw Coming

Andrew Tan — Tue, 25 Aug 2026 09:19:24 +0000

Dagster joining Prefect signals the end of standalone orchestrators. The winners will be unified platforms that combine orchestration and processing — and that's exactly where the market is heading.

In August 2026, Dagster Labs announced it would be joining forces with Prefect. Two of the most visible modern orchestration tools — both born as reactions to Airflow's limitations — are now under one roof. The press releases talk about "combining strengths" and "accelerating the future of data workflows."

The reality is simpler: the standalone orchestrator market is consolidating, and fast.

This isn't a surprise to anyone who's been watching. Venture funding for orchestration-only startups dried up two years ago. The category leaders have been searching for exits or additional funding rounds with increasingly defensive terms. Customers have been asking harder questions about roadmaps, pricing stability, and long-term viability.

What's different now is the clarity. Dagster and Prefect joining isn't just another acquisition. It's confirmation that standalone orchestration — scheduling tasks, managing dependencies, handling retries — isn't a sustainable standalone business. The tools that survive will be the ones that do more.

The Pattern: What Happens After the Press Release

Vendor consolidation in enterprise software follows a predictable script. The announcements are always optimistic. The outcomes for customers are more mixed.

Pricing changes, usually upward. The combined entity needs to show returns. "Synergies" often translate to reduced discount flexibility, new tier structures, or module-based pricing that used to be included. The Talend customers who saw renewal jumps after the Qlik acquisition aren't outliers. They're the norm.

Roadmap shifts, sometimes dramatically. Features that don't serve the combined product strategy get deprioritized. The Dagster asset model and the Prefect flow model may both survive, or one may become the "legacy" approach that receives maintenance-only updates. Teams betting on specific differentiators find themselves on the wrong side of architectural bets.

Integration debt accumulates. The tools don't merge instantly. For 12-24 months, customers run on "combined" platforms that are really two separate codebases with integration layers. Bug fixes take longer because they have to work across both systems. Documentation drifts out of sync. The migration path from "old" to "new" is promised but delayed.

None of this is malicious. It's just what happens when point solutions in a shrinking market try to survive. The standalone orchestrator category is consolidating because the economics stopped working.

Why the Winners Will Be Unified Platforms

Here's the part the consolidation story misses: the tools that survive won't be orchestrators at all. They'll be platforms that happen to include orchestration.

The standalone orchestrators tried to differentiate on scheduling models, developer experience, or observability. They treated orchestration as the product. But orchestration was never the end goal — it was always a means to an end. Teams don't wake up wanting better task schedulers. They wake up wanting reliable data pipelines.

Modern data infrastructure is moving toward unified platforms for a simple reason: the split between "orchestration" and "processing" is artificial. When your orchestrator (Airflow, Dagster, Prefect) is separate from your processing engine (Spark, dbt, custom Python), you pay a coordination tax. Multiple mental models. Multiple monitoring systems. Multiple failure modes at the integration seams.

The platforms that are winning — Databricks, Snowflake, and a new generation of unified data infrastructure — don't treat orchestration as a separate concern. It's built in. Your workflows schedule themselves, retry on failure, enforce dependencies, and trigger downstream work without a separate coordination layer.

This is where the market is heading. Not more standalone orchestrators. Fewer seams between orchestration and execution.

What This Means for Teams Making Choices Now

If you're running production workflows on Dagster, Prefect, or any other orchestration tool facing consolidation pressure, you have an opportunity. The market transition creates a window to move to something better — not just different.

Here's what to look for in a platform that will survive the consolidation:

Unified batch and streaming in one runtime. The split between "batch orchestrator" and "streaming processor" is another artificial seam that's collapsing. Teams need both. Maintaining separate tools for scheduled jobs and real-time flows doesn't make sense anymore.

Orchestration integrated with processing, not bolted on. The scheduler should understand your data, not just your task dependencies. When a step fails, you want the system to know what data was affected, not just that a task returned a non-zero exit code.

Sustainable business model, not venture-scale growth targets. The consolidation is happening because the standalone orchestrator market couldn't support venture-scale returns. Look for platforms with clear paths to profitability, reasonable pricing models, and business structures that don't require acquisition or IPO to survive.

Clear migration paths from the tools being consolidated. The best platforms right now are the ones actively helping teams migrate from Dagster, Prefect, and Airflow — not because they're orchestrators, but because they're proving they can replace the entire coordination layer with something simpler.

Where We Fit in This Transition

At layline.io, we've been building what the market is moving toward: a unified platform for both batch and streaming data processing where orchestration is intrinsic, not external.

We didn't set out to build a better orchestrator. We set out to eliminate the need for separate orchestration entirely. When your processing engine can schedule itself, retry intelligently, and maintain lineage without a separate coordination layer, the "orchestrator" category becomes a legacy concept.

The consolidation of standalone orchestrators validates this direction. The market is telling us what we already knew: teams are tired of maintaining separate scheduling layers on top of their actual data work. They want infrastructure that handles the full lifecycle — from event ingestion through transformation to delivery — without handoffs between systems.

For teams currently on Dagster or Prefect, this is actually good news. The consolidation creates urgency to evaluate alternatives, and the alternatives have gotten significantly better. A platform that handles both your scheduled batch jobs and your real-time event processing, with unified observability and no coordination seams, isn't a risky bet on a new category. It's the stable, proven direction the whole market is moving.

The Consolidation Creates Opportunity

Dagster and Prefect joining forces won't be the last move in this market. Kestra will face the same pressure. Airflow's position is stable but not growing. The standalone orchestrator category is shrinking toward a few acquired survivors and gradual absorption into platforms.

This isn't a crisis for data teams. It's a clearing of the landscape. The fragmentation of the last five years — five different orchestrators, three different streaming systems, separate monitoring for each — is giving way to consolidation around unified platforms.

The teams that come out ahead will be the ones that treat this transition as an upgrade opportunity, not a migration burden. The platforms you're moving to are better than the tools you're leaving. They're simpler to operate, cheaper to maintain, and designed for the workloads you're actually running.

The consolidation should excite you if you've been waiting for the data infrastructure market to mature. The standalone tool era is ending. The unified platform era is beginning. And that's exactly what most data teams actually need.

If you're evaluating your orchestration strategy or considering alternatives to consolidated vendors, get in touch. We're helping teams migrate from standalone orchestrators to unified platforms — and the results are consistently better than expected.

CDC Is the Plumbing Everyone Forgets Until It Breaks

Andrew Tan — Mon, 03 Aug 2026 14:20:52 +0000

Change Data Capture is the invisible layer enabling real-time analytics and event-driven systems — but most teams only think about it after their first production incident

The Invisible Layer That Everything Depends On

Real-time dashboards. Event-driven microservices. Data lakes that stay current. Behind every one of these modern data architectures sits a component that most teams don't think much about: Change Data Capture.

CDC's job is simple enough — watch database transaction logs and emit events whenever data changes. New order? Event. Status update? Event. Customer deletion? Event. The concept is elegant, and when it works, it just works.

But there's a problem. CDC is the plumbing of modern data infrastructure: invisible when it functions, catastrophic when it fails, and somehow always an afterthought in architecture reviews. Teams spend weeks debating Kafka topologies and Spark configurations, then slap in a CDC connector with default settings and move on.

Six months later, the call comes. The dashboard is six hours behind. The inventory sync is showing yesterday's data. The CEO is asking why customers can buy products that don't exist. And nobody can figure out why — because the CDC connector is "healthy" according to the monitoring dashboard.

This pattern plays out across the industry with remarkable consistency. The issue isn't that CDC is fundamentally unreliable. It's that the gap between what teams assume it does and what it actually does is wide enough to hide production incidents until they become business problems.

What CDC Actually Does (And What Teams Assume It Does)

At its core, Change Data Capture watches your database transaction log and emits events whenever data changes. Insert a row? Event. Update a field? Event. Delete a record? Event. The concept is beautifully simple.

But the simplicity is deceptive. Here's what CDC actually captures versus what teams assume it captures:

What teams assume	What actually happens
"Every change is captured immediately"	There's latency. Sometimes milliseconds, sometimes seconds, sometimes longer if the connector is backlogged.
"The events are in the same order as the transactions"	Not necessarily. Parallel replication, commit ordering, and eventual consistency can scramble sequences.
"Schema changes are handled gracefully"	Adding a column? Fine. Renaming one? Dropping one? Changing a type? Your CDC pipeline may need manual intervention.
"It's just a log tail, what could go wrong?"	Connector crashes, replication slot exhaustion, disk space issues on the source DB, network partitions...

The gap between assumption and reality is where incidents breed.

The Three Failure Modes Nobody Talks About

After watching a dozen CDC implementations go sideways, I've noticed three failure patterns that don't get enough attention in the tutorials and vendor demos.

1. The Schema Drift Trap

Your application team adds a new column to the orders table. It's a harmless change — a nullable delivery_notes field. They deploy on Tuesday. By Thursday, your data warehouse has incomplete records because the CDC connector is still using the old schema and silently dropping the new field.

The worst part? The connector doesn't fail. It just produces events that are technically valid but practically wrong. Your data quality monitors don't catch it because the schema validator thinks everything is fine. You only discover the gap when someone asks why the delivery notes report is blank for half the week.

2. The Replication Slot Bomb

PostgreSQL users, this one's for you. CDC connectors use "replication slots" to track which WAL (Write-Ahead Log) entries they've processed. If your connector goes down — or even just slows down significantly — those slots hold onto log entries. The database can't reclaim that disk space.

I've seen teams wake up to production databases at 95% disk capacity because a flaky CDC connector was holding replication slots hostage. The fix is a manual cleanup job that feels terrifying to run at 2 AM. The prevention? Monitoring and alerting that most teams don't set up until after the first incident.

3. The Consumer Coupling Problem

CDC emits a firehose of events. Every microservice, analytics job, and data warehouse sync that cares about database changes taps into that stream. It's elegant and decoupled — until it isn't.

What happens when one slow consumer can't keep up? Backpressure propagates. The CDC connector buffers, then drops, then crashes. Or worse: it keeps running but falls behind, and your "real-time" pipeline has a 20-minute lag that nobody notices because the metrics dashboard shows "connector healthy."

The fix is usually some form of buffering (Kafka, Kinesis, a message queue) between the CDC source and the consumers. But now you've added latency and another piece of infrastructure to manage. The simple plumbing has become a complex subsystem.

Sizing for Reality, Not for Hope

Here's a fictional conversation:

Me: "How many transactions per second does your CDC need to handle?"

Them: "Oh, maybe a few hundred during peak."

Me: "And what's your biggest table?"

Them: "About fifty million rows."

Me: "What happens when you run a bulk update on that table?"

Them: "...We do those sometimes."

CDC connectors aren't sized for your average transaction volume. They're sized for your worst-case transaction volume. That quarterly data cleanup job that touches ten million rows? That generates ten million CDC events in a burst. If your connector can't handle the spike, you get lag, backpressure, or dropped events.

The teams that do this well plan for bursts from day one. They set up monitoring on replication lag, not just connector health. They test their failure modes: what happens if the connector restarts mid-bulk-update? What happens if the destination is down for an hour?

Design Decisions That Make CDC Manageable

CDC doesn't have to be a ticking time bomb. Here are the patterns I've seen work in production:

Separate CDC Infrastructure from Analytics Infrastructure

Don't run your CDC connector on the same cluster as your Spark jobs or your BI queries. When the analytics team runs a heavy join that saturates the network, your CDC events shouldn't suffer. Give CDC its own lane.

Idempotent Consumers Are Non-Negotiable

CDC events can be duplicated. Connectors restart, network partitions happen, at-least-once delivery is the default. If your downstream consumer can't handle "process this order update twice," you're going to have data corruption. Build idempotency in from the start.

Schema Registries Save Sanity

Use a schema registry (Confluent Schema Registry, AWS Glue, or similar) to track changes to your event schemas. When the application team changes a table, the schema change flows through the registry and your consumers can adapt programmatically instead of breaking silently.

Monitor What Matters

"Connector is running" is the wrong metric. Monitor:

Replication lag (how far behind is the CDC from the database?)
Event processing rate (are we keeping up with production?)
Schema change events (did something change in the source we need to know about?)
Dead letter queue depth (what couldn't be processed and why?)

Where layline.io Fits: CDC Without the Footguns

At layline.io, we've watched teams struggle with CDC enough that we built a dedicated Debezium Source Asset directly into the platform. The goal isn't to reinvent CDC — Debezium is excellent — but to wrap it in the reliability and observability that production systems need.

Instead of running a standalone connector that you have to babysit, layline.io gives you:

Visual pipeline design that includes CDC sources as first-class citizens. You see the data flow from database to destination on a single canvas. When something breaks, you know exactly where.

Built-in backpressure handling through Apache Pekko's actor-model streaming. When downstream systems slow down, layline.io throttles gracefully instead of dropping events or crashing connectors.

Unified retry and error handling across the entire pipeline. CDC events that fail to process don't vanish into a log file — they go through the same retry mechanisms as every other data source.

Schema-aware transformation that can adapt to changes in the source database without manual intervention. Add a column, rename a field, change a type — the pipeline adjusts instead of breaking.

The broader point: CDC is too important to be an afterthought. It deserves the same engineering rigor as the rest of your data infrastructure. Whether you use layline.io or build your own stack, treat CDC like the critical component it is — not like plumbing you can ignore until the basement floods.

Your Data Warehouse Is Not Your Data Pipeline

Andrew Tan — Fri, 31 Jul 2026 11:14:14 +0000

Teams keep forcing their warehouse to do integration work it was never designed for. The result is ballooning costs, opaque failures, and architectures that become harder to maintain the more they ‘succeed.’ Here’s the case for separating data movement from analytics storage.

The expensive truth about modern data stacks

Spend enough time around data platform teams and you hear the same story. A company builds out its "modern data stack" — warehouse, processing layer, orchestrator — and everything looks clean on the architecture diagram. Then the warehouse bill starts to climb. Ingestion jobs fail more often than anyone expected. And every time something breaks, it takes half a day to figure out whether the problem is in the load, the reshape, the orchestrator, or the warehouse itself.

At some point, someone on the team says the quiet part out loud: "I think we built a really expensive integration tool by accident."

They are usually right.

The category error

A data warehouse is a query and storage engine. It is optimized for one thing: answering analytical questions fast over large datasets.

A data pipeline is a movement and processing runtime. It is optimized for something different: getting data from where it is to where it needs to be, in the right shape, at the right time, reliably.

Those are different jobs. But in the last decade, we've quietly asked the warehouse to do both.

It started innocently. Warehouses got better at loading data. Then they got stored procedures. Then dbt turned SQL into a processing layer. Then orchestrators started triggering warehouse queries to move data between tables. And before anyone named it, the warehouse had become the default integration layer.

The result is predictable. The warehouse is excellent at analytics. It is mediocre at integration. And when you force it to do integration at scale, you pay for it in three currencies: cost, reliability, and architectural fragility.

What goes wrong when the warehouse becomes the pipeline

The compute bill becomes a surprise

Warehouse compute is priced for analytical queries. Analysts run a few big queries, wait for results, and go make decisions. The compute is bursty and human-paced.

Integration workloads don't look like that. They run continuously or on tight schedules. They move millions of rows. They run the same conversions over and over. They don't pause to let humans read dashboards.

When you run this kind of workload inside a warehouse, the meter spins differently. It is common for a "simple" hourly sync to consume more credits than the entire analytics workload. Not because the warehouse is bad, but because it's the wrong engine for the job.

Failures become opaque

A pipeline has a clear job: take data from A, transform it, deliver it to B. When it fails, you want to know which step failed and why.

When the warehouse is the pipeline, failure is distributed across layers. Was the load slow because the warehouse was overloaded? Did the orchestrator lose its connection? Did the reshape query hit a timeout? Is the data wrong because of the source, the conversion, or a change to the warehouse execution plan?

Debugging becomes archaeology. You dig through query history, orchestrator logs, and warehouse metrics, trying to reconstruct what actually happened. The tools are all there. The clarity isn't.

Latency is whatever the warehouse decides

If your pipeline is a series of warehouse queries, your latency is bounded by warehouse scheduling. A query waits in a queue. It compiles. It runs. Maybe it gets preempted. Maybe it scales up. Maybe it doesn't.

For batch analytics, this is fine. No one cares if a nightly report finishes at 3 AM or 3:15 AM.

For operational use cases, it's not fine. Fraud detection, inventory updates, customer-facing dashboards — these need minutes or seconds, not warehouse-queue time. When the warehouse is your pipeline, you inherit its pace. And its pace is designed for analysts, not operations.

Lock-in deepens

The more integration logic lives inside the warehouse, the harder it becomes to leave. Your rewrites are in warehouse-specific SQL dialects. Your orchestration is tied to warehouse sessions. Your data quality rules run as warehouse queries. Even your cost visibility is warehouse-shaped.

This isn't a conspiracy. It's just what happens when one tool becomes responsible for too many jobs. The migration cost grows until it feels easier to stay unhappy than to leave.

What clean separation looks like

The fix isn't to throw out the warehouse. The warehouse is good at what it does. The fix is to let it do what it does and stop asking it to do everything else.

In practice, that usually means two platforms, not one:

Integration and orchestration runtime

This is where data moves, gets reshaped, gets validated, and gets routed to the right consumers. It also schedules pipelines, retries failures, enforces dependencies, and triggers downstream work — both inside the platform and in external systems. It runs on an engine designed for continuous data flow, not query latency.

Warehouse

This is where data is stored and queried. It receives clean, ready-to-query data from the integration layer. It doesn't worry about how the data got there, when the next load arrives, or what to do if a job fails. It just answers questions.

Logically, you can still think of integration and orchestration as separate concerns. Operationally, they often belong in the same runtime. A pipeline that can move data but can't schedule itself, retry itself, or trigger the next step is only half useful. The best platforms combine both.

When these concerns are separated from the warehouse, each tool gets simpler. The integration layer is optimized for throughput and reliability. The orchestrator is optimized for dependency management and failure recovery. The warehouse is optimized for query performance.

Most importantly, problems stay in their lane. When ingestion fails, you look at the integration runtime. When a report is wrong, you look at the warehouse. When a job doesn't run, you look at the orchestrator — which, in a clean setup, is part of the same runtime that moves the data.

When warehouse-as-pipeline is actually fine

I don't want to overstate this. For some teams, the warehouse-as-pipeline pattern works fine.

If you're small, your data volumes are low, your reshaping is simple, and your latency requirements are "tomorrow is fine," then keeping everything in one place is a reasonable tradeoff. The operational simplicity is worth more than the architectural purity.

The problems start when the pattern keeps scaling past its natural limit. A team that outgrows it usually knows. The bills get weird. The failures get mysterious. The idea of adding a real-time use case becomes a multi-month project instead of a configuration change.

The question isn't whether the pattern is bad. The question is whether it's still the right pattern for where you are now.

The migration path nobody takes

Most teams imagine this separation as a rip-and-replace project. It doesn't have to be.

The better approach is to extract the movement layer first. Pick one data source. Instead of loading it directly into the warehouse and then reshaping it there, move it through a dedicated integration runtime first. Clean it. Validate it. Then write the clean data to the warehouse.

The warehouse doesn't change much. The analysts keep querying the same tables. But now those tables are fed by a pipeline that is designed for feeding tables.

Once one source is moved, the pattern repeats. Source by source. Pipeline by pipeline. Over time, the warehouse stops being the integration hub and becomes what it was meant to be: the analytics hub.

Teams that do this successfully don't start with the hardest pipeline. They start with a boring one. The boring pipelines teach you the pattern without the risk. The hard pipelines get easier once the pattern is in place.

Where layline.io fits

I'll be direct: this is the architectural bet behind layline.io.

We built a data processing platform that handles the integration and orchestration layer — both batch and streaming — without making the warehouse do the heavy lifting. Pipelines move data, reshape it, validate it, and deliver it. They also schedule themselves, retry on failure, enforce dependencies, and trigger downstream workflows inside layline or in external systems.

The warehouse stores the data and queries it. Each tool does its own job.

Because layline handles both batch and streaming in the same runtime, you don't end up with one tool for your hourly loads and another tool for your real-time events. Same workflows. Same observability. Same team. And because orchestration is built in, you don't need a separate orchestrator sitting on top, coordinating between layline and everything else.

That's not a pitch for everyone. If your warehouse-as-pipeline setup is working and your bills are sane, you don't need us. But if you're staring at a tripled warehouse bill and wondering how a "simple" sync got so expensive, the separation we're describing is probably what you're actually looking for.

The question to ask your team

Pick your three most expensive warehouse workloads. Not the biggest analytical queries — the ones that run all day, moving and reshaping data.

Ask: are these workloads answering business questions, or are they just getting data into a shape where it can answer business questions?

If the answer is the second one, you've got integration work running in an analytics engine. That's not a moral failing. It's a very common architecture. But it's also a very fixable one.

The warehouse is a powerful tool. It just isn't the only tool.

The AI Productivity Gap: Why the Numbers Don't Add Up

Andrew Tan — Mon, 06 Jul 2026 13:00:31 +0000

Every enterprise dashboard claims AI is transforming the business. The actual productivity numbers tell a very different story — and understanding why matters for every team making AI investment decisions.

There's a gap between the story being told about AI in the enterprise and what companies are actually experiencing on the ground. You could watch this play out across industries for a while now, and the pattern is consistent enough that it's worth naming directly.

The pitch is familiar: AI tools will automate the repetitive work, amplify your team's output, and ultimately let you do more with less. The reality, for most organizations, looks quite different. The executives I speak with are largely describing the same experience — AI projects that showed early promise in demos and pilots, then ran into friction when exposed to the noise of real production environments.

This isn't an argument against AI adoption. It's an argument for being precise about where AI actually delivers value versus where it adds cost and complexity without a corresponding return.

The deployment failure pattern

The first thing that gets lost in AI coverage is how often production deployments fail quietly.

Announcements of AI initiatives tend to generate press. The quiet rollbacks that follow tend not to. But when you talk to operations teams candidly, the reversal pattern is common — systems that worked in controlled testing, connected to clean data and well-defined inputs, that degraded when exposed to the variability of real customers, real data, and real edge cases.

Customer-facing AI deployments have been particularly prone to this. The tolerance for errors in customer interactions is low, and the compounding effect of getting things wrong repeatedly erodes trust faster than any initial efficiency gain can offset. Teams that replaced human capacity with AI and then had to reverse course found themselves spending months rebuilding, often with more urgency than before.

The lesson isn't that AI customer interaction tools don't work — it's that the failure modes are underestimated during the planning phase, and the cost of a failed rollout exceeds the projected savings even when the initial deployment looked promising.

The accuracy ceiling

Why do production deployments fail at rates that don't match pre-deployment expectations? The answer is largely in how AI capability is measured versus how it needs to perform.

Benchmarks and vendor demos select for conditions where AI performs best. Production environments don't. The gap between benchmark accuracy and real-world accuracy is consistently larger than teams expect, particularly for anything involving ambiguous inputs, unusual edge cases, or tasks requiring contextual judgment.

In software development — which has been the proving ground for AI productivity claims — the productivity story is more nuanced than the marketing suggests. AI tools are genuinely useful for certain well-scoped tasks: generating boilerplate, explaining unfamiliar code, drafting documentation. But the secondary costs of AI-assisted development are underweighted: code review cycles get longer when you can't assume the same level of reliability you'd expect from an experienced engineer, security review becomes more necessary, and debugging AI-introduced errors can consume more time than writing equivalent code from scratch.

The net productivity effect, in practice, is much closer to neutral than the adoption narrative suggests. The teams I've seen extract real value from AI coding tools have been disciplined about scope — using AI in a narrow, well-supervised lane and keeping human judgment in the loop for anything that matters.

There's also a question of whether reliability improves sufficiently with more capable models. The structural challenge is that AI systems are fundamentally probabilistic — they approximate, they extrapolate, and their confidence doesn't reliably track their accuracy. Newer models are better, but the same category of failures persists. The question isn't whether AI will ever be reliable enough, it's whether the current generation is reliable enough for the specific task you're considering, and that requires honest evaluation rather than optimistic extrapolation.

The real cost equation

Even setting aside the reliability question, the economics of AI deployment have shifted in ways that deserve scrutiny.

When AI tools first entered the enterprise, pricing was structured to drive adoption — flat subscriptions that made ROI calculations appear straightforward. Many of those pricing models were, in retrospect, being offered well below the actual cost of providing the service. As the market has matured and providers have moved toward pricing that reflects real operational costs, the economics look quite different from the projections that justified many initial investments.

The teams that made commitments based on early pricing are now navigating a different cost environment. Usage-based pricing models mean that scaling up AI adoption increases costs non-linearly. The math that justified a pilot may not survive contact with production usage volumes.

There's also the indirect cost of integration overhead, maintenance, and the ongoing work of keeping AI systems calibrated as underlying models and APIs change. These costs are consistently underestimated in project planning and rarely appear in the productivity gain calculations that AI vendors highlight.

The honest ROI calculation for AI adoption needs to include the full cost picture: inference at realistic usage levels, integration and maintenance overhead, the cost of failures and rollbacks, and the opportunity cost of the engineering time spent managing AI systems rather than building product.

What this means for data infrastructure

The AI productivity story has a specific texture in this space worth unpacking.

The appeal of AI for data workflows is real: generating transformation logic, scaffolding pipeline boilerplate, navigating unfamiliar APIs. If AI could reliably handle these tasks, the productivity gains would be meaningful. The challenge is that data pipelines have near-zero tolerance for silent errors. A transformation that produces plausible-but-wrong output isn't just a bug — it's a corruption that propagates downstream before anyone notices.

The teams that handle this well use AI as a first-draft accelerator for well-defined, reviewable tasks, with automated validation and human review before anything touches production. That's a meaningfully different model from "AI replaces the engineer" — it's more like a junior colleague who needs supervision. That framing leads to better outcomes than treating AI as a reliable autonomous agent.

What doesn't work is using AI in the parts of data engineering where precision is non-negotiable and errors are hard to detect — schema transformations, data quality rules, anything that feeds downstream analytics that people make decisions with. The productivity gains in that zone tend to be negative once you account for the debugging and remediation work.

Calibrating the expectation

At layline.io, we've watched our customers navigate these trade-offs, and the pattern among teams that do it well is consistent: they're systematic about where AI helps and where it doesn't, they insist on validation at every stage, and they treat AI output the same way they treat any external input — with appropriate skepticism until it's been verified.

The AI productivity gap isn't closing on its own. The teams that navigate it well are the ones being precise about where AI genuinely adds value — and staying disciplined about everything else.

A few questions that have proven useful before any AI deployment in data workflows:

What does a failure look like, and how quickly would we detect it? Silent errors in pipelines are categorically more dangerous than visible failures. If the answer to "how would we detect it?" is "we'd notice when the numbers look off," that's not a detection mechanism.

What's the full cost at production scale? Usage-based pricing means the economics at pilot scale don't predict the economics at full deployment. Model it before you commit.

What's the rollback path? Given how often AI deployments require reversal, any adoption that doesn't include a tested rollback path is taking on more risk than the productivity potential justifies.

The upside of AI in data infrastructure is real. So is the downside of getting it wrong. The teams that capture the upside are the ones who go in with clear eyes about both.

Building data infrastructure where reliability isn't optional? Take a look at layline.io — the Community Edition is free to explore.

The AI Data Engineer: What Actually Changed (And What Didn't)

Andrew Tan — Wed, 01 Jul 2026 11:18:29 +0000

Every competitor blog is publishing 'AI is changing data engineering.' It's all breathless and vague. Here's the honest inventory — what LLM tooling genuinely helps with, what it still can't touch, and why the '80% automation' claims don't survive contact with production.

Why I'm writing this

Because AI is all the rage at the moment, some CTOs ask themselves: "Should AI replace half our data engineering team?"

That's the state of AI in data engineering right now. Everyone's publishing breathless content. Nobody's being specific. So here's my take on the topic:

What AI genuinely helps with

SQL generation is the clearest win. Copilot-style tools cut the time to write a first-draft analytical query by 50-70% for engineers with solid SQL fundamentals. You still need to review it. You still need to know what the answer should look like. But the blank-page problem is gone.

Schema documentation is dramatically faster. Getting from "we have 400 tables" to "we have documented 400 tables" used to take months of analyst time. With good LLM tooling, teams can get through this in weeks. The documentation isn't perfect, but it's good enough to be useful, which it often wasn't before.

Ad-hoc analysis has changed meaningfully for non-engineers. Business analysts who used to file tickets for "can you write me a query that…" can now get working answers to simple questions themselves. This is real productivity. It's also a meaningful reduction in interrupt-driven work for data engineering teams.

Code review drafts. Not a replacement for review, but catching the obvious stuff — unindexed joins, missing null checks, type mismatches — before a human looks at it saves time in aggregate.

These are real and they matter. I don't want to dismiss them.

What AI can't reliably handle

Here's where the gap between vendor claims and production reality opens up.

Schema evolution at scale

The hardest part of maintaining production pipelines isn't writing the code — it's knowing what to do when an upstream system changes a field type, deprecates a column, or starts sending data in a different format. This requires understanding the business logic behind the data, the downstream consumers, the historical context of why the field exists. An LLM that wasn't in the room when those decisions got made can't reliably reason about the right response. It'll give you something that looks right. It often isn't.

Stateful stream processing

A team can spend three months trying to get an LLM to correctly implement a windowed aggregation with late-arrival handling for their real-time fraud detection pipeline. The LLM could write the code. The code also runs. It produces wrong answers in edge cases that only show up in production, under specific ordering conditions, on days with unusual event volumes. Those bugs are the hard kind — they don't throw errors, they just silently corrupt your metrics. The model has no way to test its own output against the actual edge cases it will face.

Production failure recovery

When a Kafka consumer falls behind by 48 hours and you need to decide whether to replay, drop, or deduplicate — that's not a code generation problem. That's a judgment call that requires knowing your business, your SLAs, and the cost of each option. I've yet to see an LLM make that call correctly without significant human scaffolding.

A lead engineer at a cyber security company told me: "We got to about 70% automation on our standard ETL patterns. The last 30% is the stuff that actually breaks in production." He wasn't complaining. He understood why. But the 30% is what keeps data engineers employed.

The "80% automation" problem

Gartner published a prediction last year that 80% of data engineering work would be affected by 2027. I understand why they wrote it.

Here's the thing about 80%: the 80% they're talking about is scaffolding. Boilerplate. First drafts. The part that's genuinely 80% automatable for example is the part that was already relatively fast.

What remains is the 20% that takes 80% of the time — debugging why the data looks wrong, negotiating schema changes with upstream teams, reasoning about pipeline reliability under conditions nobody anticipated. That 20% is also the 20% where a wrong answer is expensive.

I'm not saying this to be pessimistic. The 80% matters. Freeing engineering teams from scaffolding is genuinely valuable. But the teams that plan for a world where this automation means fewer engineers are making a specific bet that the expensive problems will also get easier. They might. I'm not seeing evidence of it yet.

What I tell teams considering headcount reductions

Don't do it yet. Not because the technology isn't real, but because you're betting on the wrong variable.

The teams getting the most from AI tooling aren't the ones cutting headcount — they're the ones taking the same headcount and pointing it at harder problems. The engineers who used to spend their days on routine ETL work are now working on data quality frameworks, schema governance, real-time pipeline reliability. The output per engineer is higher. The quality of the output is higher. The team is harder to replace, not easier.

That's the story. AI is a productivity multiplier for data engineers. It's not THE data engineer.

A simple overview

I know I said I'd avoid the comparison table format. But this one is genuinely the clearest way to show it:

Task	AI helps	AI struggles
SQL generation	First drafts, 50-70% faster	Complex logic with subtle business rules
Schema docs	First pass, weeks not months	Accurate semantics without business context
Ad-hoc analysis	Simple questions for non-engineers	Questions requiring cross-system context
Pipeline code	Boilerplate, standard patterns	Stateful logic, edge-case handling
Schema evolution	—	Almost entirely human judgment
Failure recovery	—	Requires business + operational knowledge
Production debugging	—	LLMs don't know your specific history

The left column is real. The right column is why data engineering teams still exist.

Where layline.io fits

I'll be direct: the AI productivity gains I described above are easier to capture when your pipelines have explicit structure that LLMs can understand and extend.

At layline.io, we build pipelines with declarative configuration — the logic is in structured operators, not embedded in custom code (except for the casual Javascript or Python here and there and only where really necessary). That turns out to pair well with AI-assisted development. When an engineer asks an LLM to add a processing step, the LLM can reason about it clearly. When something breaks, the failure is in a known place rather than buried in bespoke code.

That's not why we built it that way. We built it that way because declarative pipelines are easier for humans to debug and maintain. The AI affinity turned out to be a side effect.

But it does mean that teams building on a structured foundation get more out of AI tooling than teams working in custom code. Something worth considering when you're making architectural choices that will matter in two years.

The question worth asking your team

Try this: pick your last five data incidents. For each one, ask whether an AI could have prevented it or diagnosed it faster.

For most teams the answer is "maybe 1 out of 5." The other four are problems an LLM can't reliably reason about — wrong business logic that is technically correct code, a schema change from an upstream team that nobody announced, an edge case in stream processing that only manifests at specific event volumes.

If you're evaluating AI tooling, that's your baseline. Not "will AI change data engineering" — of course it will. But "will AI eliminate the problems that actually hurt us?" That answer is no, not yet, and probably not without something changing that hasn't changed.

Data Lineage Is a Vanity Metric Without Business Context

Andrew Tan — Tue, 09 Jun 2026 15:32:58 +0000

Most lineage tools produce beautiful diagrams that don't answer the one question that matters: 'What breaks if this data is wrong?' Here's how to move from observability theater to business-critical lineage.

Dashboards that lie

Many companies spend north of six figures on data lineage tools. Their demos are impressive: sprawling visualizations showing every table, pipeline, and dependency across a data warehouse. Colors indicate freshness. Arrows show data flow. It looks like the control room of a nuclear power plant.

All of this is great and fancy, but one of the unanswered questions is what happens when table X has bad data.

You can click around the diagrams, zoom and pan, locate the table, inspect the downstream consumers and transformations it fed into. And then you can tell that twelve dashboards use 'customer address'."

The real question, though, is which business processes break. Does shipping stop? Do invoices go to the wrong place? Do compliance reports fail? You get the idea.

The dashboard instead knows that data flowed from A to B, but it had no idea what B was actually for.

Lineage theater

This is what I call lineage theater: the practice of building impressive-looking data flow diagrams that satisfy compliance checklists and vendor demos but don't actually help when things break.

The tooling vendors have optimized for the wrong thing. They're selling visualizations. What data teams need is context: the ability to trace a data quality issue to its business impact in under 60 seconds.

You can see this pattern across many companies. They implement lineage tools with great fanfare. The diagrams go up on office TVs (cool), and the data governance team writes documentation about the documentation. Then, six months later, an upstream system changes a column name and the lineage diagram lights up like a Christmas tree while the actual business impact remains a mystery.

The team ends up doing what they'd have done without the tool: paging through Slack, checking with stakeholders, manually tracing which reports matter for which decisions.

The business context gap

Here's the fundamental problem: technical lineage and business lineage are different things, and most tools only do the first one.

Technical lineage answers: Where did this data come from and where does it go?

Business lineage answers: What decisions depend on this data, and what happens if it's wrong?

The gap between them is where data disasters happen. A pipeline can be 100% correct from a technical standpoint: all jobs green, all tests passing: while producing output that's catastrophically wrong for the business.

Let's say you are a fintech company, and your loan approval model is technically perfect. The lineage shows clean data from application through feature engineering to model scoring. What the lineage doesn't capture is that a recent schema change had swapped two similarly named fields, "annual_income" and "monthly_income", in a way that the pipeline's validation rules didn't catch.

The model now treats monthly income as annual income. Approval thresholds that should have required $60,000/year are triggering on $5,000/month. The lineage diagram shows green arrows. The business outcome is a month of bad loans that take six months to unwind.

What useful lineage actually looks like

The teams that do lineage well have one thing in common: they treat it as a business mapping exercise, not a technical documentation task.

You need to takes a different approach: Every data asset in your warehouse has three tags:

Criticality: Is this used for regulatory reporting, operational decisions, or analytics only?
Downstream processes: Which business functions depend on this? (Not which tables, but which functions: billing, clinical decisions, compliance)
Error impact: What happens if this data is wrong? (Delay, financial loss, regulatory issue, patient safety)

The resulting lineage tool is technically simple: just a basic dependency tracker. But combined with those three tags, it tells exactly what you need to know when something breaks.

When your claims processing table has a data quality issue, you don't need to trace through fifteen downstream tables. You look at the tags, see "Criticality: Regulatory, Downstream: Monthly CMS filing, Error impact: $2M penalty if late," and knew immediately to escalate to the CFO and initiate the manual filing backup process.

The entire incident response takes minutes. No diagram navigation required.

Why we build the wrong thing

So why do teams keep buying visualization-heavy lineage tools that don't solve the real problem?

Part of it is procurement theater. The person buying the tool often isn't the person debugging the 2 AM incident. They're buying something that looks thorough for the compliance audit or the board presentation. Beautiful diagrams check boxes. Business context mapping requires organizational work that doesn't photograph well.

Part of it is the nature of how these tools are sold. Vendors demo with clean, synthetic data environments where the lineage is obvious. Real enterprise data environments are super messy: decades of legacy systems, undocumented transformations, tribal knowledge that's never been written down. Mapping business context requires talking to people, not just scanning code. It doesn't scale as cleanly as automated technical discovery.

And part of it is that technical lineage is easier to build. You can scan query logs, parse SQL, inspect DAGs. Business context requires interviews, documentation, ongoing maintenance as processes change. It's organizational work disguised as technical work.

How to fix your lineage

If you're already invested in a lineage tool (and most companies are at this point), you don't need to rip it out. You need to add business context to it.

Start with your incident history. Look at the last five data quality incidents that caused real business impact. For each one, identify:

What data was wrong
What business process broke
Who needed to know
How long it took to figure that out

Now go look at your lineage tool. Does it help with any of those questions? If not, you have your improvement roadmap.

Tag critical assets manually. Don't try to tag everything. Start with your top 20 data assets by business impact. For each one, document: what decisions it feeds, who owns those decisions, and what happens if the data is bad.

This takes time: maybe 30 minutes per asset; maybe more. But it turns your lineage from a pretty diagram into an operational tool.

Build business-aware alerting. Most data quality alerts are technical. "This job failed" or "this column has nulls." Add business-aware alerts: "The daily revenue summary has suspicious values, which feeds the CEO dashboard at 8 AM."

The alert should include not just what's wrong, but what depends on it and who needs to know.

Practice incident response. Run a tabletop exercise. Simulate a data quality issue in a critical upstream system. Time how long it takes to answer: which business decisions are affected, who needs to be notified, and what the mitigation options are.

If it takes more than five minutes, your lineage needs more business context.

The product I wish existed

I've looked at some of the lineage tools on the market. They're all variations on the same theme: scan your infrastructure, build a graph, show you pretty visualizations.

What I want is different. I want a tool that starts with business processes and works backwards. Map the decisions first, then trace to the data that feeds them. When something breaks, tell me which decisions are at risk, not just which tables are affected.

But you don't need a new platform to get better lineage. You need to stop treating lineage as a technical problem and start treating it as an organizational one. The diagram isn't the product. The business context is.

The test for your lineage tool

Here's a simple test. Pick a critical data asset in your system: something that would be painful if it were wrong. Now answer these questions without looking at code:

What business decisions depend on this data?
Who makes those decisions, and when?
What's the cost of being wrong?
Who needs to know if there's a quality issue?

If you can't answer those questions in 60 seconds, your lineage tool isn't doing its job: no matter how beautiful the diagram looks.

The goal isn't perfect observability. It's usable context. And that's harder to build, but infinitely more valuable.

Why I Stopped Believing 'Best Practices' and Started Trusting 'Works For Us'

Andrew Tan — Thu, 28 May 2026 12:00:54 +0000

I spent 18 months building the 'perfect' architecture. Then I watched a customer delete it in 20 minutes and replace it with a cron job. Here's what I learned about the 'best practice' trap — and why boring technology often wins.

The demo that didn't land

We were eighteen months into building layline.io when we got our first serious enterprise prospect. A Fortune 500 logistics company. Their data team had reviewed our architecture, liked the batch-plus-streaming approach, and scheduled a full-day workshop to dive deep.

We prepared for weeks. We built a demo that showed off everything: complex event processing, automatic backpressure handling, schema evolution. It was, by every textbook definition, a best practice architecture. Distributed. Fault-tolerant. Built to scale horizontally. The kind of system you'd draw on a whiteboard during a conference talk.

The workshop went well. The engineers asked good questions. Then, in the last thirty minutes, the senior architect leaned back and said something I'll never forget: "This is impressive. But we run everything on a single server with cron jobs, and it works. What would we actually gain from all this complexity?"

I had a hundred answers ready. Scalability. Resilience. Future-proofing. But I could see in his face that he wasn't asking for a technology comparison. He was asking me to justify why his current reality — boring, simple, working — was insufficient.

I couldn't. Not honestly.

The architecture I deleted

Three months later, I was in a different room with a different customer. This one was a mid-sized fintech. They'd been running a Kafka-based streaming pipeline for two years. It was falling over constantly. They'd hired consultants, upgraded hardware, rewritten their consumer logic twice. The system was "correct" by every distributed systems textbook. It was also a nightmare to operate.

In the meeting, their lead engineer showed me the architecture diagram. It was beautiful. Twelve microservices, three different persistence layers, a custom operational data store for state management. They'd followed every pattern from the Confluent blog and the Martin Kleppmann book.

"What if," I asked, "you just wrote the events to a file and processed them in batches?"

He stared at me. "That's... not streaming."

"No," I agreed. "But you're processing events hourly anyway because your downstream system can't handle real-time updates. You're paying the operational cost of a streaming architecture to achieve batch semantics."

They didn't buy layline.io that day. But six weeks later, I got an email. They'd deleted the entire architecture. Replaced it with a single process that read files and wrote to a database. A cron job, basically. Their p99 latency went from 200ms to five minutes — which didn't matter because their business process was daily. Their operational incidents went from three per week to zero. Their engineering team went from firefighting to shipping features.

The "wrong" architecture was better because it matched their actual constraints, not their aspirational ones.

The best practice trap

Here's what I've learned from 25 years of building and selling data infrastructure: best practices are context-dependent by definition, but they're marketed as universal truths.

The streaming-first architecture that Netflix needs is not the architecture a 50-person SaaS company needs. The microservices approach that lets Amazon deploy 10,000 times per day is not what your team of four engineers needs. The AI agent framework that raised $50 million in VC funding is not what your cron-based ETL needs.

But you wouldn't know that from reading industry content. Every vendor blog post, every conference talk, every architecture blueprint shows the same progression: start simple, then "graduate" to complexity as you grow. The implication is clear: simple is for beginners. Complexity is for serious practitioners.

This is backwards. Complexity is a liability that should be added reluctantly, not a badge of honor that should be pursued eagerly.

What "works for us" actually looks like

I've started asking customers a different question in early conversations: "What's the simplest thing that could work for your actual workload?" Not your projected workload in three years. Not your aspirational real-time use case that the CEO mentioned once. Your actual workload, today.

The answers are consistently surprising:

A healthcare company processing a million patient records per day does it with a single-threaded Python script that runs for four hours every night. It's been running for six years without modification. Why? Because the records arrive via FTP at 2 AM, and the doctors don't look at the dashboards until 8 AM.
A retail company processing point-of-sale data from 2,000 stores uses a three-node Kafka cluster. Not because they need the throughput — they could fit a day's events in a single file — but because their existing team knew Kafka and didn't have time to learn something new during their busiest season.
A logistics company tracking container ships in real time uses... a spreadsheet. The operations team updates it manually. They tried building an automated pipeline twice. Both times, the automated system failed in ways that were harder to debug than the spreadsheet. The spreadsheet is "wrong" in a dozen ways, but it's inspectably wrong. You can see the errors.
None of these are "best practices." All of them are correct for their context.

The AI agent hype cycle

If you want to see the best practice trap in its most aggressive form, watch how the data engineering industry is currently responding to AI agents.

Every competitor blog I read lately — Airbyte, Confluent, Kestra — is positioning their product as "AI agent ready." There are deep dives on Model Context Protocol, ontologies for agents, context window management. The implicit message: if you're not architecting for AI agents right now, you're falling behind.

I asked a customer last week if they were looking at AI agents for their data pipelines. "We spent six months trying to get an LLM to generate SQL," he said. "It was 70% accurate on simple queries and 30% accurate on complex ones. The 30% was subtle enough that we didn't catch it until the CEO saw a wrong number in a board deck. We're back to engineers writing SQL."

This isn't an argument against AI. It's an argument against defaulting to AI because it's the current best practice. The teams that benefit from AI agents today have specific characteristics: high query volumes, relatively simple schemas, tolerance for occasional errors, and engineering resources to validate outputs. If that doesn't describe your situation, AI agents aren't your solution yet — no matter how many vendor blog posts suggest otherwise.

How to actually evaluate technology

So if "best practice" isn't a reliable guide, what is?

Here's the framework I use now, both for my own architectural decisions and when advising customers:

Start with your actual constraints. How much data? What arrival patterns? What latency requirements? What team size and expertise? What budget for operations? The answers to these questions eliminate 90% of "industry standard" architectures immediately.

Optimize for debugging, not for elegance. The architecture that produces clean diagrams is often the one that's hardest to debug at 2 AM. Prefer systems where you can trace a single record from source to destination without crossing three different abstraction layers.

Measure operational cost in team attention, not just infrastructure dollars. A distributed system that runs itself but requires a senior engineer to be on call is more expensive than a single server that needs occasional restarts but can be managed by a junior hire.

Plan for the migration you'll actually do, not the migration you should do. Every team has legacy systems they'll never retire. Design for graceful coexistence with old technology rather than revolutionary replacement of it.

When in doubt, start boring. You can always add complexity. Removing it is much harder. The teams I see succeeding are the ones that add technology reluctantly, with clear evidence that simpler approaches have been exhausted.

The counter-argument I'm not making

I want to be clear about what I'm not saying. I'm not arguing for technical conservatism or against trying new things. Some problems genuinely do require complex, distributed, real-time architectures. If you're processing payments at scale, you need exactly-once semantics. If you're serving ML features with sub-100ms latency, you need streaming. If you're Netflix, you need what Netflix needs.

But most companies aren't Netflix. Most data pipelines don't need to handle 10,000 events per second. Most teams don't have a platform engineering group to manage the operational burden of "modern" data infrastructure.

The uncomfortable truth is that the industry has conflated "what successful tech companies do" with "what you should do." Successful tech companies have endless engineering resources, high tolerance for operational pain, and business models that require real-time everything. Your company probably doesn't. Your architecture shouldn't pretend otherwise.

Where layline.io fits (and where it doesn't)

I'll close with something that might surprise you: layline.io is not the right choice for every data integration problem.

If you have a few batch jobs that run reliably on a schedule, and your team is comfortable with your current setup, you probably don't need us. Seriously. The operational overhead of learning a new platform isn't worth it if your current reality is stable and understood.

Where we add value is when you've outgrown simple approaches but want to avoid the complexity tax of stitching together multiple specialized tools. When you need both batch and streaming in the same system. When your team is tired of maintaining separate orchestration, transformation, and monitoring layers. When you want to consolidate around one model instead of managing a coordination seam between three different tools.

Even then, I'd rather you start with a proof of concept that processes a single day's data than an ambitious migration plan. Prove that the simpler approach works for your actual workload before committing to the complex one.

The best practice is the one that works for you. Everything else is just marketing.