<?xml version="1.0" encoding="UTF-8"?>
<rss version="2.0" xmlns:atom="http://www.w3.org/2005/Atom" xmlns:dc="http://purl.org/dc/elements/1.1/">
  <channel>
    <title>DEV Community: Andrew Tan</title>
    <description>The latest articles on DEV Community by Andrew Tan (@andrew_tan_layline).</description>
    <link>https://dev.to/andrew_tan_layline</link>
    <image>
      <url>https://media2.dev.to/dynamic/image/width=90,height=90,fit=cover,gravity=auto,format=auto/https:%2F%2Fdev-to-uploads.s3.us-east-2.amazonaws.com%2Fuploads%2Fuser%2Fprofile_image%2F3880780%2Fa0095aa9-e581-4d26-a573-4c327e5f52ea.jpeg</url>
      <title>DEV Community: Andrew Tan</title>
      <link>https://dev.to/andrew_tan_layline</link>
    </image>
    <atom:link rel="self" type="application/rss+xml" href="https://dev.to/feed/andrew_tan_layline"/>
    <language>en</language>
    <item>
      <title>IBM Owns Confluent. Kafka Users Still Have a Runtime Problem.</title>
      <dc:creator>Andrew Tan</dc:creator>
      <pubDate>Mon, 02 Nov 2026 14:35:47 +0000</pubDate>
      <link>https://dev.to/andrew_tan_layline/ibm-owns-confluent-kafka-users-still-have-a-runtime-problem-1d60</link>
      <guid>https://dev.to/andrew_tan_layline/ibm-owns-confluent-kafka-users-still-have-a-runtime-problem-1d60</guid>
      <description>&lt;p&gt;&lt;em&gt;The Kafka protocol is still open. The commercial center of gravity is not. Seven months after IBM closed Confluent, the questions that matter are dual-run, replay, and who owns batch when the streaming story is watsonx.&lt;/em&gt;&lt;/p&gt;

&lt;h2&gt;
  
  
  Two facts that keep getting mashed together
&lt;/h2&gt;

&lt;p&gt;Apache Kafka is still an open protocol. You can run it on MSK, on a cluster you operate, on Redpanda, on Confluent Platform. The wire format did not get acquired.&lt;/p&gt;

&lt;p&gt;Confluent, the company that turned that protocol into the default commercial Kafka stack, has been an IBM company since &lt;a href="https://newsroom.ibm.com/2026-03-17-ibm-completes-acquisition-of-confluent,-making-real-time-data-the-engine-of-enterprise-ai-and-agents" rel="noopener noreferrer"&gt;March 17, 2026&lt;/a&gt;. IBM paid about &lt;a href="https://newsroom.ibm.com/2026-03-17-ibm-completes-acquisition-of-confluent,-making-real-time-data-the-engine-of-enterprise-ai-and-agents" rel="noopener noreferrer"&gt;$11 billion, $31 a share&lt;/a&gt;. Day one, the story was not "we remain the independent Kafka company." Day one was watsonx.data, IBM MQ, webMethods, and IBM Z. Streaming as fuel for an AI data platform and hybrid automation.&lt;/p&gt;

&lt;p&gt;Treat those as the same event and you will sleep through the next 18 months of product decisions. Separate them and the next conversation with your vendor gets a lot more useful.&lt;/p&gt;

&lt;h2&gt;
  
  
  What actually closed
&lt;/h2&gt;

&lt;p&gt;I already wrote the &lt;a href="https://layline.io/resources/blog/2026-04-17-layline-vs-kafka" rel="noopener noreferrer"&gt;technical Kafka comparison&lt;/a&gt;. This is the concentration piece.&lt;/p&gt;

&lt;p&gt;On close day, IBM put the thesis in the &lt;a href="https://newsroom.ibm.com/2026-03-17-ibm-completes-acquisition-of-confluent,-making-real-time-data-the-engine-of-enterprise-ai-and-agents" rel="noopener noreferrer"&gt;press release title&lt;/a&gt;: real-time data as the engine of enterprise AI and agents. Rob Thomas talked about models and agents acting on what is happening now. Jay Kreps, still CEO in that quote, talked about setting the world's data in motion at IBM scale.&lt;/p&gt;

&lt;p&gt;That is a different product bet than "best Kafka."&lt;/p&gt;

&lt;p&gt;Best Kafka means partition rebalances, connector quality, Flink packaging, cloud SKUs, and whether your consumer lag page still works on a Tuesday. IBM's bet means Confluent Cloud and Platform get pulled toward watsonx, MQ, and the mainframe. The Kafka log is the on-ramp. The destination is IBM's AI and integration suite.&lt;/p&gt;

&lt;p&gt;You can like that bet. Plenty of enterprises already live in that suite. You should not pretend it is the same roadmap Confluent sold as a public company.&lt;/p&gt;

&lt;h2&gt;
  
  
  The founder left. The protocol stayed.
&lt;/h2&gt;

&lt;p&gt;On &lt;a href="https://x.com/jaykreps/status/2087954153582780901" rel="noopener noreferrer"&gt;August 13&lt;/a&gt;, Jay Kreps stepped back after nearly twelve years. Founder transitions after an $11 billion close are normal. They also tell you where the writing happens.&lt;/p&gt;

&lt;p&gt;Roadmaps used to get argued in Mountain View, on a Kafka-company clock. They now get argued inside IBM Software, on a watsonx and mainframe clock. Connector cadence, cloud packaging, and what "supported" means will follow that clock.&lt;/p&gt;

&lt;p&gt;A concrete signal showed up in late October. IBM &lt;a href="https://www.ibm.com/new/announcements/expanding-enterprise-data-engineering-with-apache-flink-stream-processing" rel="noopener noreferrer"&gt;announced watsonx.data integration for Apache Flink&lt;/a&gt;, with a visual canvas and a Python SDK that submit jobs onto Confluent Platform for Apache Flink. That is not a Kafka killer. It is IBM wrapping stream processing in the watsonx.data product line. If you bought Confluent because you wanted an independent streaming company, read that sentence again.&lt;/p&gt;

&lt;p&gt;I am not claiming the product got worse. I do not have a telemetry feed into Confluent Cloud. I am claiming the center of gravity moved, and you can see it in public.&lt;/p&gt;

&lt;h2&gt;
  
  
  Kafka the language, IBM the publisher
&lt;/h2&gt;

&lt;p&gt;Think of Kafka as a language. Lots of people speak it. The grammar is in the open. IBM bought the largest school that taught the commercial dialect, printed the textbooks, and ran the certification booth at the conference.&lt;/p&gt;

&lt;p&gt;You can still speak the language. You can hire people who speak it. You can run a cluster that never sends IBM a dollar.&lt;/p&gt;

&lt;p&gt;What you cannot assume is that the commercial default path (Confluent Cloud, Confluent Platform, the connector catalog you already standardized on) will fight for Kafka independence. It will follow IBM's data and AI platform. Dual-running a topic into watsonx.data will get more love than making it trivial to leave.&lt;/p&gt;

&lt;p&gt;Vendors follow their thesis. Your job is to notice the thesis changed.&lt;/p&gt;

&lt;p&gt;&lt;a href="https://media2.dev.to/dynamic/image/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto%2Cformat=auto/https%3A%2F%2Fdev-to-uploads.s3.us-east-2.amazonaws.com%2Fuploads%2Farticles%2Fw41t8gfwetnrexnbo4g4.jpg" class="article-body-image-wrapper"&gt;&lt;img src="https://media2.dev.to/dynamic/image/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto%2Cformat=auto/https%3A%2F%2Fdev-to-uploads.s3.us-east-2.amazonaws.com%2Fuploads%2Farticles%2Fw41t8gfwetnrexnbo4g4.jpg" width="800" height="533"&gt;&lt;/a&gt;&lt;/p&gt;

&lt;h2&gt;
  
  
  Three questions your next QBR should include
&lt;/h2&gt;

&lt;p&gt;Skip the "are you still committed to Kafka?" slide. Of course they are. Kafka is the asset.&lt;/p&gt;

&lt;p&gt;Ask the operational questions the press release does not cover.&lt;/p&gt;

&lt;h3&gt;
  
  
  Dual-run for the next two years
&lt;/h3&gt;

&lt;p&gt;If IBM folds SKUs, rebrands consoles, or moves you onto a watsonx control plane, you need a period where the old path and the new path produce the same outputs. Dual-run is boring. Dual-run is how you survive a vendor merge without a weekend cutover.&lt;/p&gt;

&lt;h3&gt;
  
  
  Replay when MQ, Z, and a warehouse are in the same incident
&lt;/h3&gt;

&lt;p&gt;Most Kafka teams already know how to rewind a consumer group. The hard case is mixed: a stream, a mainframe transaction, a nightly ERP file, and an Iceberg table someone already queried. IBM's day-one diagram puts those in one family. Your incident on a Tuesday night still needs a person who can rewind one without corrupting the others.&lt;/p&gt;

&lt;h3&gt;
  
  
  Whether batch still needs a second orchestrator
&lt;/h3&gt;

&lt;p&gt;If the answer is "use Airflow for the files and Confluent for the topics," you did not buy a platform. You bought a streaming suite and kept the split you already had. Ask them to show one workflow that replays a late file and a bad topic with the same ownership.&lt;/p&gt;

&lt;p&gt;If your AE cannot answer those without a solution architect and a week of delay, you have your signal.&lt;/p&gt;

&lt;h2&gt;
  
  
  The problem that did not get acquired
&lt;/h2&gt;

&lt;p&gt;IBM bought a streaming company. It did not buy your coordination problem.&lt;/p&gt;

&lt;p&gt;You still have to catch up a batch job that missed the window while the stream kept going. You still have to replay three hours of poison messages without double-writing the warehouse. You still have to join a slowly changing dimension to a live topic. You still have to tell a new hire which system is the source of truth on Friday afternoon.&lt;/p&gt;

&lt;p&gt;None of that lives in the Kafka protocol. The IBM logo on the Confluent console does not make it easier. Some of it gets harder if the commercial stack is being re-aimed at agents and watsonx while your actual workload is still "make the payments file and the fraud topic agree."&lt;/p&gt;

&lt;p&gt;The teams I talk to who are calm about this did not post "Kafka is open source, we are fine" in Slack. They already treat the broker as a transport and keep the runtime (retries, dependencies, replay, approvals) in a place they control.&lt;/p&gt;

&lt;h2&gt;
  
  
  Where we fit
&lt;/h2&gt;

&lt;p&gt;At layline.io we work with teams who speak Kafka and still have to land files at 6pm. We are not a Kafka replacement and we are not an IBM alternative. We are the runtime that sits across batch and streaming so the coordination work does not live inside the streaming vendor's suite.&lt;/p&gt;

&lt;p&gt;If IBM's platform is the right home for you, use it. Just do not confuse "we still produce Kafka records" with "our operational model is unchanged." The records can look the same. The gravity around them cannot.&lt;/p&gt;




&lt;p&gt;&lt;em&gt;If you are on Confluent Cloud or Platform and trying to separate protocol comfort from vendor gravity, &lt;a href="https://layline.io/contact" rel="noopener noreferrer"&gt;get in touch&lt;/a&gt;. We have helped teams map dual-run and replay across batch and stream without waiting for the next rebrand.&lt;/em&gt;&lt;/p&gt;




</description>
      <category>kafka</category>
      <category>ibm</category>
      <category>watsonx</category>
      <category>layline</category>
    </item>
    <item>
      <title>Streamhouse Is the New Buzzword. Here's the Part That Actually Matters.</title>
      <dc:creator>Andrew Tan</dc:creator>
      <pubDate>Tue, 27 Oct 2026 09:21:38 +0000</pubDate>
      <link>https://dev.to/andrew_tan_layline/streamhouse-is-the-new-buzzword-heres-the-part-that-actually-matters-1c7e</link>
      <guid>https://dev.to/andrew_tan_layline/streamhouse-is-the-new-buzzword-heres-the-part-that-actually-matters-1c7e</guid>
      <description>&lt;p&gt;&lt;em&gt;Confluent, Aiven, and Redpanda want you to believe the streamhouse changes everything. It does change some things. But it doesn't fix the problem most teams are actually struggling with.&lt;/em&gt;&lt;/p&gt;




&lt;h2&gt;
  
  
  The word nobody defined
&lt;/h2&gt;

&lt;p&gt;I was on a panel last month when someone used the word "streamhouse" five times in two minutes. Thepny were earnest about it. They said it was "the biggest shift in data architecture since the lakehouse." They said it would "collapse the boundary between streaming and analytics." They said teams that didn't adopt it would be "maintaining legacy batch infrastructure" within eighteen months.&lt;/p&gt;

&lt;p&gt;After the panel, I asked them what a streamhouse actually is.&lt;/p&gt;

&lt;p&gt;They paused. Then they said: "It's when your streaming platform and your warehouse become the same thing." I asked for an example. They said Kafka with Iceberg. I asked how that's different from what Flink already does. They said the streamhouse is "native." I asked what "native" means in this context. They said it means you don't need a separate job to move data from the stream into the warehouse. I asked what happens when the stream needs to be replayed, or backfilled, or joined to a slowly changing dimension table. They said those are "implementation details."&lt;/p&gt;

&lt;p&gt;They were a vendor. They were selling something. But the word itself is now in the wild, and I've heard it from data architects, platform leads, and even a CTO who told me their 2027 strategy is to "streamhouse the analytics stack."&lt;/p&gt;

&lt;p&gt;So let's do what the marketing doesn't: define the thing, separate the real change from the repackaging, and figure out whether this is a category you should care about or just a new label for complexity you already have.&lt;/p&gt;




&lt;h2&gt;
  
  
  What "streamhouse" is supposed to mean
&lt;/h2&gt;

&lt;p&gt;The term comes from a working group announced earlier this year by Confluent, Aiven, Redpanda, StreamNative, and Ververica. The pitch is straightforward: streaming data and analytical data have lived in separate systems for too long. Kafka handles the stream. Snowflake handles the analytics. You build a pipeline between them. The pipeline breaks. Everyone blames the other team.&lt;/p&gt;

&lt;p&gt;The streamhouse proposal is to unify them. Streaming data lands in open table formats — Iceberg, Delta Lake, Paimon — as it arrives. Query engines read those tables directly. No ETL job. No batch sync. No "landing zone" that gets stale between runs. The stream is the table. The table is the stream.&lt;/p&gt;

&lt;p&gt;Confluent calls this Tableflow. Aiven calls it Aiven Streams. The exact branding changes by vendor, but the architecture is consistent:&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;Events arrive in a streaming layer (Kafka, Redpanda, Pulsar).&lt;/li&gt;
&lt;li&gt;A streaming engine (Flink, Spark Structured Streaming, or a vendor-specific runtime) writes them incrementally into an open table format.&lt;/li&gt;
&lt;li&gt;Query engines (Trino, DuckDB, Snowflake, BigQuery) read the table as if it were a normal warehouse table.&lt;/li&gt;
&lt;li&gt;The table is "continuously updated" rather than "periodically refreshed."&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;This is not a scam. There is a real technical achievement here. Making a streaming pipeline produce queryable table files in near-real-time, with schema evolution, compaction, and snapshot isolation, is genuinely hard. The teams building this deserve credit.&lt;/p&gt;

&lt;p&gt;But the architecture is not new. It's a refinement of something Flink, Kinesis Analytics, and even some bespoke Kafka-to-S3 pipelines have been doing for years. What's new is the marketing umbrella: the word "streamhouse," the working group, the promise that this pattern removes the need for separate batch and streaming systems.&lt;/p&gt;

&lt;p&gt;That last part is the stretch.&lt;/p&gt;




&lt;h2&gt;
  
  
  What changes — and what doesn't
&lt;/h2&gt;

&lt;p&gt;Let's separate the real from the aspirational.&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;What actually gets better:&lt;/strong&gt;&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;
&lt;strong&gt;Latency to queryable data drops.&lt;/strong&gt; If you're running a batch sync every hour, moving to continuous micro-batching or true streaming cuts that to seconds or minutes. For use cases where freshness matters — fraud detection, operational dashboards, real-time pricing — this is a genuine improvement.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Schema evolution is cleaner.&lt;/strong&gt; Open table formats handle schema changes better than most bespoke streaming-to-warehouse pipelines. Adding a column doesn't require a migration script. Renaming a field is tracked in metadata. This reduces the "who broke the dashboard" incidents.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;One storage layer, fewer copies.&lt;/strong&gt; If your analytics team can query the same Iceberg files that your streaming pipeline produces, you don't need a separate warehouse extract. Fewer copies means fewer sync jobs, fewer consistency windows, and fewer places for data to diverge.&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;&lt;strong&gt;What stays exactly the same:&lt;/strong&gt;&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;
&lt;strong&gt;Orchestration.&lt;/strong&gt; A streamhouse does not remove the need to coordinate jobs, dependencies, and retries. If your streaming pipeline feeds a dbt model that feeds a dashboard, that DAG still needs to be managed. The streamhouse replaces the batch extraction step. It does not replace the orchestration layer.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Backfills and replays.&lt;/strong&gt; When a bug corrupts three days of streamed data, you need to rewind the log and reprocess. The streamhouse table format makes this possible. It does not make it easy. You still need to manage consumer offsets, handle schema changes that happened mid-stream, and verify that replayed data matches what downstream systems already saw.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Cross-system workflows.&lt;/strong&gt; Most real pipelines touch more than one system. A stream might trigger an API call, update a cache, write to a database, and produce a notification. The streamhouse handles the analytics path. It does not handle the operational path. You still need a runtime that can coordinate across systems with different consistency models and failure modes.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Cost structure.&lt;/strong&gt; Streaming infrastructure is always-on infrastructure. A batch job that runs for ten minutes every hour costs ten minutes of compute per hour. A streaming job that runs continuously costs sixty. The streamhouse shifts cost from sync jobs to streaming compute. Whether that's cheaper depends on your volume, your query patterns, and how well you tune compaction.&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;The streamhouse improves the storage and queryability of streaming data. It does not remove the need for a runtime that handles the messy operational work around that data.&lt;/p&gt;




&lt;h2&gt;
  
  
  The continuous table illusion
&lt;/h2&gt;

&lt;p&gt;The most seductive idea in the streamhouse pitch is the "continuous table." It's a table that never goes stale. You query it and you get the latest data. No batch job. No waiting. The table is alive.&lt;/p&gt;

&lt;p&gt;This is a powerful mental model. It's also slightly misleading.&lt;/p&gt;

&lt;p&gt;A continuous table is not a single object that updates in place. It's a sequence of snapshots, each representing the table at a point in time. When you query it, you're reading one of those snapshots. The query engine chooses the latest one it knows about. The streaming engine is constantly appending new files and committing new snapshots.&lt;/p&gt;

&lt;p&gt;This matters because the gap between "the latest event arrived" and "the latest snapshot is queryable" is not zero. It depends on:&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;
&lt;strong&gt;Commit frequency.&lt;/strong&gt; How often does the streaming engine write a new snapshot? Every second? Every minute? The faster, the more metadata overhead.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Compaction lag.&lt;/strong&gt; The table accumulates small files. Queries slow down. Somebody — or some scheduled job — has to compact those files into larger ones. During compaction, query behavior can change.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Catalog consistency.&lt;/strong&gt; The query engine learns about new snapshots through a catalog. If the catalog is eventually consistent — and most are — a query might read a snapshot that's thirty seconds behind the actual stream. Thirty seconds is not real-time. It's near-real-time wearing a real-time costume.&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;The continuous table is a useful abstraction. But it's still an abstraction over a batch-of-micro-batches process. The files are discrete. The snapshots are discrete. The freshness is bounded by commit intervals, compaction schedules, and catalog latency. Calling it "continuous" obscures the engineering reality, which is: smaller batches, faster commits, same coordination problems.&lt;/p&gt;

&lt;p&gt;&lt;a href="https://media2.dev.to/dynamic/image/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto%2Cformat=auto/https%3A%2F%2Fdev-to-uploads.s3.us-east-2.amazonaws.com%2Fuploads%2Farticles%2Fu47lrha4w1977k4i6a0r.jpg" class="article-body-image-wrapper"&gt;&lt;img src="https://media2.dev.to/dynamic/image/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto%2Cformat=auto/https%3A%2F%2Fdev-to-uploads.s3.us-east-2.amazonaws.com%2Fuploads%2Farticles%2Fu47lrha4w1977k4i6a0r.jpg" width="800" height="533"&gt;&lt;/a&gt;&lt;/p&gt;




&lt;h2&gt;
  
  
  Where teams still end up stitching runtimes together
&lt;/h2&gt;

&lt;p&gt;Here's the scenario the streamhouse marketing doesn't show you:&lt;/p&gt;

&lt;p&gt;You have a streaming pipeline that writes events into an Iceberg table. Your analytics team queries it with Trino. Your ML team reads it with Spark. Your operational team needs to trigger an alert when a specific event pattern occurs — not after the event lands in the table, but when it arrives in the stream.&lt;/p&gt;

&lt;p&gt;The streamhouse handles the analytics and ML paths. But the operational alert needs to run against the stream itself, before the event is committed to a table snapshot. So you stand up a separate Flink job to process the raw stream. Now you have two runtimes: one for the table path, one for the operational path.&lt;/p&gt;

&lt;p&gt;Then the business asks for a weekly report that reconciles the streamed data with the batch data from your ERP system. The ERP data doesn't arrive in a stream. It arrives as a daily CSV dump. So you build a batch job that reads the CSV, joins it to the Iceberg table, and produces the report. Now you have three runtimes.&lt;/p&gt;

&lt;p&gt;Then someone discovers that the streaming pipeline produced bad data for six hours last Tuesday because of a schema drift. You need to rewind the stream, fix the logic, and replay. But the replay has to produce exactly the same table snapshots that downstream queries already saw, or the ML team's training run will be inconsistent. So you coordinate the replay with the ML team's pipeline. Now you're doing runtime orchestration across four systems.&lt;/p&gt;

&lt;p&gt;The streamhouse solved the "how do I query my stream" problem. It did not solve the "how do I coordinate multiple runtimes with different semantics" problem. That problem is still yours.&lt;/p&gt;




&lt;h2&gt;
  
  
  The honest decision framework
&lt;/h2&gt;

&lt;p&gt;If you're evaluating whether the streamhouse pattern makes sense for your team, don't start with the architecture diagram. Start with what you're actually trying to do.&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;Use a streamhouse when:&lt;/strong&gt;&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;Your primary use case is analytics or ML on data that arrives continuously, and your current batch sync is the bottleneck.&lt;/li&gt;
&lt;li&gt;You already run Kafka (or Redpanda, or Pulsar) and you're tired of maintaining separate pipelines to move that data into a warehouse.&lt;/li&gt;
&lt;li&gt;Your query patterns are compatible with open table formats — columnar reads, predicate pushdown, snapshot isolation — and you're not trying to do sub-second operational triggers.&lt;/li&gt;
&lt;li&gt;You have the team to manage compaction, catalog consistency, and schema evolution at streaming velocity.&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;&lt;strong&gt;Don't use a streamhouse when:&lt;/strong&gt;&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;Your "streaming" use case is actually a batch job that runs every fifteen minutes. The streamhouse is not free. If nobody needs data faster than fifteen minutes, you're paying continuous compute for intermittent value.&lt;/li&gt;
&lt;li&gt;You need operational triggers — API calls, cache updates, notifications — that must fire on event arrival, not on table commit. The streamhouse is an analytics pattern, not an operational pattern.&lt;/li&gt;
&lt;li&gt;Your team is already struggling with open table format operations. The streamhouse adds streaming velocity to those operations. If compaction and catalog management are already painful, making them faster doesn't make them easier.&lt;/li&gt;
&lt;li&gt;You're hoping it will replace your orchestration layer. It won't. You'll still need something to manage dependencies, retries, backfills, and cross-system workflows.&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;The streamhouse is a storage and query pattern. It is not a runtime pattern. If your problems are storage and query problems — stale data, sync lag, too many copies — it's worth evaluating. If your problems are runtime problems — coordination, failure recovery, cross-system semantics — the streamhouse doesn't help and may add complexity.&lt;/p&gt;




&lt;h2&gt;
  
  
  What the vendors are really selling
&lt;/h2&gt;

&lt;p&gt;I don't think the streamhouse working group is being dishonest. I think they're being selective. They're showing the use cases where the pattern shines — real-time analytics, continuous ML feature stores, operational dashboards — and quietly omitting the use cases where it doesn't.&lt;/p&gt;

&lt;p&gt;What they're really selling is a shift in center of gravity. For the last decade, the warehouse was the center of the data universe. Everything flowed into it. The streamhouse proposal says the stream is the center, and the warehouse is just one consumer of many.&lt;/p&gt;

&lt;p&gt;This is a meaningful shift for some organizations. If your business is event-driven — fraud detection, IoT, real-time trading — the stream-first model makes sense. But most organizations are not event-driven. They're schedule-driven. They run reports. They update dashboards. They retrain models overnight. For these teams, moving the center of gravity to the stream is a solution in search of a problem.&lt;/p&gt;

&lt;p&gt;The vendors know this. That's why the marketing emphasizes "future-proofing" and "modern data architecture" rather than specific operational improvements. The streamhouse is positioned as an inevitable evolution, like the cloud or the lakehouse before it. The subtext is: if you're not doing this, you're falling behind.&lt;/p&gt;

&lt;p&gt;Maybe. But "falling behind" what? If your batch pipelines are reliable, your stakeholders are happy, and your costs are predictable, adopting a streamhouse because a vendor says it's inevitable is not modernization. It's architecture by FOMO.&lt;/p&gt;




&lt;h2&gt;
  
  
  Where we fit in
&lt;/h2&gt;

&lt;p&gt;At layline.io, we don't sell a streamhouse. We sell a runtime that handles both batch and streaming in the same pipeline, on the infrastructure you already have.&lt;/p&gt;

&lt;p&gt;The reason this matters: a lot of teams are evaluating the streamhouse because they want batch and streaming to coexist without maintaining two separate stacks. That's a real problem. But the streamhouse solves it by unifying storage around the stream. We solve it by unifying runtime around the workflow.&lt;/p&gt;

&lt;p&gt;If your analytics team needs a warehouse table and your operations team needs a stream trigger, both paths can run through the same pipeline logic, the same observability, and the same failure recovery. The storage format becomes a configuration choice — Iceberg when it helps, Parquet when it doesn't, a message bus when you need operational latency — rather than an architectural commitment.&lt;/p&gt;

&lt;p&gt;The streamhouse is a useful pattern for a specific set of problems. It's not a universal architecture, and it's not a replacement for the operational runtime that keeps your pipelines working at 2 AM. The teams that succeed with it are the ones who adopt it for the right reasons — query latency, reduced sync overhead, cleaner schema evolution — and don't expect it to solve problems it was never designed to touch.&lt;/p&gt;

&lt;p&gt;The buzzword will fade. The underlying technology — continuous ingestion into open table formats — will stick around and improve. The question is whether your team needed it before the marketing told you that you did.&lt;/p&gt;




&lt;p&gt;&lt;em&gt;If your team is trying to separate the streamhouse signal from the noise — or if you're looking for a runtime that handles batch and streaming without forcing you to pick a storage religion — &lt;a href="https://layline.io/contact" rel="noopener noreferrer"&gt;get in touch&lt;/a&gt;. We've helped teams evaluate this exact tradeoff, and the answer is usually simpler than the architecture diagrams suggest.&lt;/em&gt;&lt;/p&gt;

</description>
      <category>streamhouse</category>
      <category>dataengineering</category>
      <category>realtimedata</category>
      <category>dataarchitecture</category>
    </item>
    <item>
      <title>dbt Is Leaving the Warehouse. Are Your Pipelines Ready?</title>
      <dc:creator>Andrew Tan</dc:creator>
      <pubDate>Tue, 06 Oct 2026 14:22:46 +0000</pubDate>
      <link>https://dev.to/andrew_tan_layline/dbt-is-leaving-the-warehouse-are-your-pipelines-ready-1339</link>
      <guid>https://dev.to/andrew_tan_layline/dbt-is-leaving-the-warehouse-are-your-pipelines-ready-1339</guid>
      <description>&lt;p&gt;dbt built the modern analytics workflow. Now the transformation layer is moving upstream — and the gap between SQL models and runtime execution is becoming the bottleneck nobody planned for.&lt;/p&gt;

&lt;h2&gt;
  
  
  The scheduled run that used to be enough
&lt;/h2&gt;

&lt;p&gt;For most of the last decade, the analytics workflow looked like this:&lt;/p&gt;

&lt;p&gt;Extract data from your sources. Load it into a warehouse. Write SQL models in dbt. Schedule them to run every few hours. Build dashboards on top. When the business asks why the numbers look off, you check the freshness badge and discover the job failed six hours ago.&lt;/p&gt;

&lt;p&gt;It wasn't perfect, but it was predictable. The contract between analytics and engineering was clear: models are built in the warehouse, on a schedule, in SQL. dbt made that contract elegant. Version-controlled models, automated testing, dependency graphs, documentation generated from code. For batch analytics, it was a genuine leap forward.&lt;/p&gt;

&lt;p&gt;Then the business started asking for fresher data. Not next-day fresh. Not hourly fresh. Event-driven fresh. The kind of fresh where a customer updates their profile and the recommendation engine knows about it before they click away.&lt;/p&gt;

&lt;p&gt;And suddenly, the scheduled run isn't enough.&lt;/p&gt;




&lt;h2&gt;
  
  
  Why dbt's success created a runtime gap
&lt;/h2&gt;

&lt;p&gt;dbt's core insight was separating model logic from execution infrastructure. You write the SQL. dbt handles the DAG, the tests, the documentation, the materialization strategy. The actual compute runs wherever you point it — Snowflake, BigQuery, Redshift, Databricks. dbt doesn't care. It's a compiler and orchestrator for SQL models, not a runtime.&lt;/p&gt;

&lt;p&gt;That separation was genius for batch workloads. It meant analytics engineers could own modeling logic without managing clusters, partitioning strategies, or memory tuning. The warehouse handled all of that. The analytics engineer focused on semantics: what does this model mean, how is it tested, who depends on it.&lt;/p&gt;

&lt;p&gt;But that separation assumes something important: that the execution layer can handle the workload you're throwing at it. And for scheduled batch jobs against a modern warehouse, that's true. For real-time processing against streaming data, it's not.&lt;/p&gt;

&lt;p&gt;The gap isn't in the SQL. dbt's SQL is still SQL. The gap is in everything that happens between the event and the model result:&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;Freshness SLA. A dbt model running every fifteen minutes is still fifteen minutes behind. In a streaming context, fifteen minutes is a batch job wearing a smaller costume.&lt;/li&gt;
&lt;li&gt;Out-of-order events. Streamed data arrives late, duplicated, or out of sequence. A SQL model that assumes ordered input produces wrong results without warning.&lt;/li&gt;
&lt;li&gt;Stateful operations. Window functions, sessionization, deduplication — these need to maintain state across events. dbt's materialization strategies (table, incremental, ephemeral) weren't designed for event-time state management.&lt;/li&gt;
&lt;li&gt;Operational joins. Joining a stream of orders to a slowly changing dimension table in real time is a different problem than joining two warehouse tables on &lt;code&gt;customer_id&lt;/code&gt;. The dimension might change mid-query. The stream doesn't wait.&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;These aren't edge cases. They're the defining characteristics of stream processing. And dbt, by design, delegates all of them to the execution layer.&lt;/p&gt;




&lt;h2&gt;
  
  
  What the vendors are actually selling
&lt;/h2&gt;

&lt;p&gt;If you've been to a data conference in the last year, you've seen the marketing shift. Confluent is pushing dbt-native modeling on Flink. Fivetran launched dbt Wizard with AI-assisted model generation. Snowflake announced Dynamic Tables, BigQuery has Materialized Views, Databricks is doubling down on Delta Live Tables.&lt;/p&gt;

&lt;p&gt;The message is consistent: you can keep your dbt workflow and just... make it real-time. The SQL stays the same. The DAG stays the same. The only thing that changes is the speed.&lt;/p&gt;

&lt;p&gt;This is approximately half true.&lt;/p&gt;

&lt;p&gt;Yes, you can run SQL on streaming data. Flink SQL, Spark Structured Streaming, and a growing list of stream processors support SQL syntax that looks familiar. Yes, you can version-control those SQL files and build DAGs. Some tools even support dbt-style testing and documentation.&lt;/p&gt;

&lt;p&gt;What they don't tell you is that the other half of the problem — the runtime half — doesn't go away. It just changes shape.&lt;/p&gt;

&lt;p&gt;When you move from scheduled batch to continuous streaming, you inherit a new set of concerns that dbt never had to solve:&lt;/p&gt;

&lt;div class="table-wrapper-paragraph"&gt;&lt;table&gt;
&lt;thead&gt;
&lt;tr&gt;
&lt;th&gt;Batch assumption&lt;/th&gt;
&lt;th&gt;Streaming reality&lt;/th&gt;
&lt;/tr&gt;
&lt;/thead&gt;
&lt;tbody&gt;
&lt;tr&gt;
&lt;td&gt;Data is complete when the job starts&lt;/td&gt;
&lt;td&gt;Data is never complete; late arrivals are normal&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;Failures are caught at the end of the run&lt;/td&gt;
&lt;td&gt;Failures must be detected and handled per-event&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;Schema changes happen between runs&lt;/td&gt;
&lt;td&gt;Schema changes happen mid-stream&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;Reprocessing means rerunning a job&lt;/td&gt;
&lt;td&gt;Reprocessing means rewinding a stream and replaying&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;Cost is proportional to data volume&lt;/td&gt;
&lt;td&gt;Cost is proportional to infrastructure uptime&lt;/td&gt;
&lt;/tr&gt;
&lt;/tbody&gt;
&lt;/table&gt;&lt;/div&gt;

&lt;p&gt;The SQL might look the same. But the system running it is solving an entirely different set of problems.&lt;/p&gt;




&lt;h2&gt;
  
  
  The ownership question nobody wants to answer
&lt;/h2&gt;

&lt;p&gt;Here's where it gets organizational. dbt created a clean boundary: analytics engineers own the models, platform engineers own the warehouse. When a model fails, it's usually a SQL problem or a data quality problem. When the warehouse is slow, it's a platform problem. The separation of concerns mapped neatly to a separation of teams.&lt;/p&gt;

&lt;p&gt;Streaming breaks that boundary.&lt;/p&gt;

&lt;p&gt;When a real-time pipeline fails, is it a SQL problem or an infrastructure problem? If events are arriving out of order, does the analytics engineer rewrite the window logic, or does the platform engineer reconfigure the stream processor's watermark settings? If a late-arriving event corrupts a materialized view, who fixes it — the person who wrote the SQL or the person who manages the checkpointing?&lt;/p&gt;

&lt;p&gt;I've seen teams handle this three ways, and only one of them works:&lt;/p&gt;

&lt;p&gt;Option 1: Analytics engineers learn stream processing. They become fluent in checkpointing, backpressure, event time vs. processing time, and exactly-once semantics. This works for small teams with senior people. It doesn't scale.&lt;/p&gt;

&lt;p&gt;Option 2: Platform engineers own everything downstream of Kafka. The analytics engineers write SQL specs, and the platform team implements them in Flink or Spark. This preserves the separation but creates a translation layer. SQL specs are ambiguous about event-time behavior. The platform team makes assumptions. Those assumptions become bugs six months later.&lt;/p&gt;

&lt;p&gt;Option 3: Separate the modeling logic from runtime responsibility. Analytics engineers own what the model means — the business logic, the tests, the semantics. Platform engineers own how it runs — the execution engine, the state backend, the failure recovery. Both sides agree on a contract: the model expects ordered input within a bounded lateness, and the runtime guarantees that contract or surfaces a clear error.&lt;/p&gt;

&lt;p&gt;This third option is harder to set up. It requires both teams to agree on interfaces and failure modes upfront. But it's the only one that scales without turning your analytics engineers into distributed systems specialists or your platform team into mind readers.&lt;/p&gt;




&lt;h2&gt;
  
  
  What "real-time dbt" actually requires
&lt;/h2&gt;

&lt;p&gt;If your team is serious about moving dbt-style models into real-time pipelines, here is what you actually need — not what's in the marketing deck:&lt;/p&gt;

&lt;p&gt;A runtime that understands event time. Not just processing time. Event time. The difference between "when did this arrive" and "when did this happen" is the difference between correct results and subtly wrong results that look fine on a dashboard.&lt;/p&gt;

&lt;p&gt;Explicit state management. Windowing, deduplication, and sessionization all require state. That state needs to be checkpointed, recoverable, and queryable for debugging. If you can't inspect what the system thought at 2:47 PM last Tuesday, you can't debug a production incident.&lt;/p&gt;

&lt;p&gt;Schema evolution with teeth. Adding a column is easy. Handling a type change, a rename, or a semantic shift in a column's meaning is hard. Your pipeline needs to detect these changes, decide whether they're safe, and either adapt or stop with a clear error.&lt;/p&gt;

&lt;p&gt;Replay and backfill as first-class operations. In batch, reprocessing is rerunning a job. In streaming, it's rewinding a log and replaying events through the same logic. If your real-time pipeline can't replay exactly, you can't recover from bugs, you can't test changes against historical data, and you can't prove compliance.&lt;/p&gt;

&lt;p&gt;Cost visibility by workload. Batch costs are easy to reason about: this job processed this much data and took this long. Streaming costs are continuous: the job is always running, always consuming resources, and the cost doesn't correlate cleanly with business output. You need telemetry that connects infrastructure spend to pipeline behavior.&lt;/p&gt;

&lt;p&gt;None of these are SQL features. They're runtime features. And they're the difference between a demo that runs for ten minutes and a system that runs for ten months.&lt;/p&gt;

&lt;p&gt;&lt;a href="/images/blog/2026-10-05/inline1.jpg" class="article-body-image-wrapper"&gt;&lt;img src="/images/blog/2026-10-05/inline1.jpg" alt="A team of engineers standing at the boundary between a calm warehouse room and a chaotic streaming pipeline room, with one person trying to hold a SQL model that is stretching and deforming between the two spaces"&gt;&lt;/a&gt;&lt;/p&gt;

&lt;h2&gt;
  
  
  &lt;img src="https://media2.dev.to/dynamic/image/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto%2Cformat=auto/https%3A%2F%2Fdev-to-uploads.s3.us-east-2.amazonaws.com%2Fuploads%2Farticles%2F2lwk5b2sykuiiwm9bbdu.jpg" width="800" height="533"&gt;
&lt;/h2&gt;

&lt;h2&gt;
  
  
  The honest pitch
&lt;/h2&gt;

&lt;p&gt;dbt changed how analytics teams work. It brought software engineering practices — version control, testing, documentation — to a discipline that badly needed them. That contribution is real and lasting.&lt;/p&gt;

&lt;p&gt;But dbt was built for a world where data moves in batches, warehouses are the center of gravity, and "fresh" means "updated this hour." The industry is moving toward a world where data moves continuously, models are applied in flight, and "fresh" means "updated this millisecond."&lt;/p&gt;

&lt;p&gt;That doesn't make dbt obsolete. It makes dbt incomplete for a growing set of use cases.&lt;/p&gt;

&lt;p&gt;The vendors selling "real-time dbt" are responding to a real demand. But what they're selling is usually a SQL interface on a stream processor, not a solution to the runtime problems that stream processing introduces. The SQL is the easy part. The state management, failure recovery, schema evolution, and operational observability are the hard parts. And those hard parts don't disappear just because the SQL looks familiar.&lt;/p&gt;

&lt;p&gt;If you're evaluating one of these tools, don't ask "Can it run my dbt models faster?" Ask "What happens when a node restarts mid-window?" "How do I replay last Tuesday's data through a model I changed yesterday?" "What does the system do when schema changes at 2 AM?"&lt;/p&gt;

&lt;p&gt;The answers to those questions will tell you whether you're buying a faster batch tool or a real streaming runtime.&lt;/p&gt;




&lt;h2&gt;
  
  
  Where we fit in
&lt;/h2&gt;

&lt;p&gt;At layline.io, we built a runtime that handles both batch and streaming in the same pipeline. Not two separate systems with a SQL veneer over each. One system where the same team can build scheduled warehouse loads and real-time event processing without switching tools, contexts, or mental models.&lt;/p&gt;

&lt;p&gt;The analytics engineers keep ownership of model semantics. The platform team keeps ownership of execution infrastructure. But both sides work in the same environment, with the same observability, and the same guarantees around replay, state recovery, and schema evolution.&lt;/p&gt;

&lt;p&gt;dbt taught the industry that modeling logic deserves rigor. We're building on that idea — and adding the runtime rigor that real-time data demands.&lt;/p&gt;

</description>
      <category>dbt</category>
      <category>realtime</category>
      <category>streamingdata</category>
      <category>dataengine</category>
    </item>
    <item>
      <title>Iceberg Is Easy to Demo. The Maintenance Bill Arrives Later.</title>
      <dc:creator>Andrew Tan</dc:creator>
      <pubDate>Thu, 24 Sep 2026 08:08:26 +0000</pubDate>
      <link>https://dev.to/andrew_tan_layline/iceberg-is-easy-to-demo-the-maintenance-bill-arrives-later-fgn</link>
      <guid>https://dev.to/andrew_tan_layline/iceberg-is-easy-to-demo-the-maintenance-bill-arrives-later-fgn</guid>
      <description>&lt;p&gt;Open table formats solve portability. They do not solve operations. The real work starts after the launch post: compaction, retention, multi-catalog sprawl, and keeping batch and streaming paths consistent.&lt;/p&gt;

&lt;h2&gt;
  
  
  The demo that never ages
&lt;/h2&gt;

&lt;p&gt;Here is how the Iceberg pitch usually goes:&lt;/p&gt;

&lt;p&gt;An engineer stands in front of a screen and runs a query against a Parquet dataset. Then they run the same query against an Iceberg table. The results are identical. The audience nods. Then the engineer shows time travel—rolling back to a previous snapshot—and the room actually murmurs. Somebody asks about schema evolution. The engineer adds a column, rewrites nothing, and the old queries still work. The committee is sold.&lt;/p&gt;

&lt;p&gt;Six months later, the same team is in a different room. This one has no screen. Just a spreadsheet, a growing list of alerts, and a question nobody asked during the demo:&lt;/p&gt;

&lt;p&gt;Who owns the compaction?&lt;/p&gt;

&lt;p&gt;The demo showed what Iceberg makes possible. It did not show what Iceberg makes &lt;em&gt;your&lt;/em&gt; problem.&lt;/p&gt;




&lt;h2&gt;
  
  
  What the spec promises vs. what the team inherits
&lt;/h2&gt;

&lt;p&gt;Open table formats are a genuine advance. Portability across engines, snapshot isolation, partition evolution, and hidden partitioning are real capabilities that solve real pain. If you are trying to escape a vendor-locked warehouse, Iceberg, Delta Lake, and Hudi are the best paths out.&lt;/p&gt;

&lt;p&gt;But the spec is not the system. The spec says how the metadata should be laid out. It does not say how your team should keep that metadata from growing without bound, how to coordinate compaction across multiple writers, or what happens when two different query engines disagree on which snapshot to read.&lt;/p&gt;

&lt;p&gt;Here is what the demo skips:&lt;/p&gt;

&lt;h3&gt;
  
  
  Compaction is not automatic
&lt;/h3&gt;

&lt;p&gt;Every insert, update, and delete creates new files and new metadata entries. Left alone, a high-velocity table accumulates thousands of small files. Query performance degrades. Metadata files bloat. The table that looked fast in the demo starts timing out in production. Somebody has to schedule compaction, monitor it, tune it, and handle the failures when two jobs try to rewrite the same partition.&lt;/p&gt;

&lt;h3&gt;
  
  
  Catalog sprawl is real
&lt;/h3&gt;

&lt;p&gt;Iceberg tables need a catalog: Hive, Glue, Nessie, Polaris, a custom REST service. Each catalog has its own consistency model, its own authentication, its own upgrade cycle. A team that adopts Iceberg to avoid vendor lock-in often ends up managing two or three catalog systems instead of one warehouse. The lock-in shifts from the storage format to the catalog layer.&lt;/p&gt;

&lt;h3&gt;
  
  
  Retention policy is a distributed problem
&lt;/h3&gt;

&lt;p&gt;When a snapshot expires, Iceberg marks the data as unreachable. But the underlying files still exist in object storage until something deletes them. That "something" is your problem. If you set aggressive retention to save on storage, you might lose the ability to roll back when a downstream job produces bad results. If you keep everything, your storage bill compounds while your S3 bucket turns into an archaeological dig.&lt;/p&gt;

&lt;h3&gt;
  
  
  Batch and streaming see different tables
&lt;/h3&gt;

&lt;p&gt;A batch job writing to Iceberg produces large, well-structured files. A streaming job produces small, frequent files. If both paths write to the same table, the query planner has to handle two radically different file layouts. The streaming path needs frequent compaction to stay readable. The batch path needs stable files to avoid recomputation. Coordinating these two rhythms in one table is harder than the architecture diagrams suggest.&lt;/p&gt;

&lt;p&gt;The demo showed a single writer and a single reader. Production rarely works like that.&lt;/p&gt;




&lt;h2&gt;
  
  
  Why "open" creates new fragmentation
&lt;/h2&gt;

&lt;p&gt;The promise of open table formats is interoperability. Query the same data from Spark, Trino, Flink, DuckDB, Snowflake, BigQuery. In practice, each engine supports a different subset of the spec at a different maturity level.&lt;/p&gt;

&lt;p&gt;One engine supports position deletes but not equality deletes. Another supports time travel but only for tables written by its own catalog. A third supports partition evolution but requires a specific metadata version that breaks older readers. The table is "open" in theory. In practice, it is coupled to the specific combination of engines and catalog versions your team happens to run.&lt;/p&gt;

&lt;p&gt;This is not a criticism of the projects themselves. Iceberg, Delta, and Hudi are moving fast and improving rapidly. The issue is that teams adopt them expecting liberation and discover a new kind of operational surface area. Instead of one vendor to blame, they have a matrix of version compatibilities to manage.&lt;/p&gt;

&lt;p&gt;The hidden cost is cognitive load. Your data engineers now need to understand not just their pipelines, but the compaction scheduling, catalog consistency model, metadata format version, and engine-specific behaviors of every tool that touches the table. That expertise does not come from a demo.&lt;/p&gt;

&lt;p&gt;&lt;a href="https://media2.dev.to/dynamic/image/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto%2Cformat=auto/https%3A%2F%2Fdev-to-uploads.s3.us-east-2.amazonaws.com%2Fuploads%2Farticles%2Fqugrl2ziqlo68tq5pdf5.jpg" class="article-body-image-wrapper"&gt;&lt;img src="https://media2.dev.to/dynamic/image/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto%2Cformat=auto/https%3A%2F%2Fdev-to-uploads.s3.us-east-2.amazonaws.com%2Fuploads%2Farticles%2Fqugrl2ziqlo68tq5pdf5.jpg" width="800" height="533"&gt;&lt;/a&gt;&lt;/p&gt;




&lt;h2&gt;
  
  
  The checklist nobody runs before adopting
&lt;/h2&gt;

&lt;p&gt;If your team is evaluating an open table format, here are the questions that matter more than query performance in a benchmark:&lt;/p&gt;

&lt;h3&gt;
  
  
  Who owns compaction, and what happens when it fails?
&lt;/h3&gt;

&lt;p&gt;Compaction is not a one-time setup. It is a continuous background process that competes for the same compute resources as your production queries. If compaction falls behind, queries slow down. If compaction corrupts a partition, recovery is manual and stressful. You need an owner, a runbook, and a way to detect when compaction is not keeping up.&lt;/p&gt;

&lt;h3&gt;
  
  
  What is your catalog exit strategy?
&lt;/h3&gt;

&lt;p&gt;Catalogs are the real lock-in point. If you commit to Glue today, can you migrate to Nessie or Polaris later without rewriting table paths and reconfiguring every downstream job? Most teams do not test this until they are forced to.&lt;/p&gt;

&lt;h3&gt;
  
  
  How do you handle late data and backfills?
&lt;/h3&gt;

&lt;p&gt;Batch backfills and streaming late arrivals both rewrite historical partitions. Open table formats handle this better than raw Parquet, but they do not eliminate the coordination problem. If a backfill runs while a streaming job is appending to the same partition, you need to understand isolation semantics, retry behavior, and exactly what each engine does when it sees a conflict.&lt;/p&gt;

&lt;h3&gt;
  
  
  What is your metadata growth plan?
&lt;/h3&gt;

&lt;p&gt;Metadata files are small but they multiply. A table with daily snapshots and hourly compaction can generate thousands of metadata files per month. Object storage is cheap, but LIST operations are not free. Some query engines load the full metadata tree into memory. At a certain scale, metadata itself becomes a performance bottleneck.&lt;/p&gt;

&lt;h3&gt;
  
  
  Who gets paged when a query returns wrong results?
&lt;/h3&gt;

&lt;p&gt;Snapshot isolation is great until someone reads from the wrong snapshot because the catalog was briefly inconsistent. Or because a streaming job committed an incomplete batch. Or because two engines interpreted the same metadata file differently. Debugging these issues requires expertise in the format, the catalog, and the specific engine. The on-call rotation just got deeper.&lt;/p&gt;

&lt;p&gt;If you cannot answer these questions with something more specific than "we'll figure it out," you are not adopting a technology. You are taking on a new operational domain.&lt;/p&gt;




&lt;h2&gt;
  
  
  Where the tradeoff is worth it
&lt;/h2&gt;

&lt;p&gt;I want to be fair. There are situations where open table formats are the clear right choice:&lt;/p&gt;

&lt;h3&gt;
  
  
  You are actively escaping vendor lock-in.
&lt;/h3&gt;

&lt;p&gt;If your warehouse provider is raising prices, deprecating features, or limiting egress, the portability of an open format is worth the operational overhead. The alternative is staying trapped.&lt;/p&gt;

&lt;h3&gt;
  
  
  You genuinely need time travel and rollback.
&lt;/h3&gt;

&lt;p&gt;Some workloads—especially in regulated industries or financial services—require the ability to reconstruct historical state exactly. The snapshot model is not a nice-to-have here. It is a compliance requirement.&lt;/p&gt;

&lt;h3&gt;
  
  
  You run multiple compute engines on the same data.
&lt;/h3&gt;

&lt;p&gt;If your analytics team uses Spark, your BI team uses Trino, and your ML pipeline uses DuckDB, a shared open table format eliminates the extract-transform-load dance between systems. The coordination cost is real, but it is lower than maintaining three separate copies of the same dataset.&lt;/p&gt;

&lt;h3&gt;
  
  
  You have the team for it.
&lt;/h3&gt;

&lt;p&gt;If you have engineers who understand metadata formats, compaction strategies, and catalog consistency models, the operational burden is manageable. If you do not, you are outsourcing the expertise to consultants and hoping they stay available.&lt;/p&gt;




&lt;h2&gt;
  
  
  Where layline.io fits
&lt;/h2&gt;

&lt;p&gt;At layline.io, we do not sell a table format. We sell a processing runtime that handles both batch and streaming workloads on the infrastructure you already have. That includes open table formats when they make sense, and traditional storage when they do not.&lt;/p&gt;

&lt;p&gt;The reason this matters: a lot of teams adopt Iceberg because they need batch and streaming to coexist, and they have been told that open table formats are the only way to unify them. That is not true. The unification happens at the processing layer, not the storage layer. If your runtime can write well-structured files in batch mode and handle micro-batches in streaming mode—while managing compaction, backfills, and late data in the same workflow—the storage format becomes a configuration choice, not an architectural commitment.&lt;/p&gt;

&lt;p&gt;We see teams who adopted Iceberg for the right reasons and then discovered that the hard part was never the format. It was the operational coordination around it: keeping batch and streaming paths consistent, handling schema changes without breaking downstream consumers, and making sure the same business logic produces the same results regardless of when the data arrives.&lt;/p&gt;

&lt;p&gt;That is the problem we focus on. The table format is a detail. The operational model is what determines whether the system works at 2 AM on a Tuesday.&lt;/p&gt;




&lt;h2&gt;
  
  
  The question to ask before the demo
&lt;/h2&gt;

&lt;p&gt;Next time a vendor shows you a slick Iceberg demo—time travel, partition evolution, engine switching that looks effortless—ask them this:&lt;/p&gt;

&lt;p&gt;"Show me the compaction schedule. Show me the catalog failover. Show me what happens when a streaming job and a batch backfill hit the same partition. Show me the storage bill after six months of metadata growth. And show me who gets paged when a query engine reads a snapshot that was partially written."&lt;/p&gt;

&lt;p&gt;If the answer is a reference to the documentation, you are looking at the easy part. The hard part is what you will own for the next three years.&lt;/p&gt;

&lt;p&gt;Open table formats are not a scam. They are a real, valuable technology with a genuine operational cost that the marketing rarely mentions. The teams that succeed are the ones who price that cost upfront, assign ownership before the first table is created, and treat the format as one component of a larger operational system—not as a magic layer that makes infrastructure problems disappear.&lt;/p&gt;

&lt;p&gt;The demo is the beginning. The maintenance bill is where the story actually starts.&lt;/p&gt;




&lt;p&gt;&lt;em&gt;If your team is weighing open table formats and trying to understand the full cost of ownership, &lt;a href="https://layline.io/contact" rel="noopener noreferrer"&gt;get in touch&lt;/a&gt;. We work with teams on exactly this problem—and the operational reality is usually more manageable than the fear, once you know what to plan for.&lt;/em&gt;&lt;/p&gt;

</description>
      <category>dataengineering</category>
      <category>datagovernance</category>
      <category>datacompliance</category>
      <category>datapipelines</category>
    </item>
    <item>
      <title>Compliance Fails in Transit: Why Data Governance Has to Move Into the Pipeline</title>
      <dc:creator>Andrew Tan</dc:creator>
      <pubDate>Tue, 15 Sep 2026 11:04:53 +0000</pubDate>
      <link>https://dev.to/andrew_tan_layline/compliance-fails-in-transit-why-data-governance-has-to-move-into-the-pipeline-4o18</link>
      <guid>https://dev.to/andrew_tan_layline/compliance-fails-in-transit-why-data-governance-has-to-move-into-the-pipeline-4o18</guid>
      <description>&lt;h2&gt;
  
  
  The audit that never finds the real problem
&lt;/h2&gt;

&lt;p&gt;Here's a scene that plays out in regulated industries every quarter:&lt;/p&gt;

&lt;p&gt;The compliance team runs their governance scan. The data catalog shows classification tags on every warehouse table. The lakehouse has column-level security policies. The BI tool enforces row-level access. Everyone signs off. The auditors leave happy.&lt;/p&gt;

&lt;p&gt;Meanwhile, a CDC feed is replicating customer PII from a transactional database to an analytics cluster. Nobody classified the stream itself — they classified the destination table, which is fine, except the stream passes through a staging topic where three different services subscribe. One of them is an AI enrichment job that sends fragments of the data to an external LLM API. The API call doesn't show up in the data catalog because the catalog only catalogs what lands, not what moves.&lt;/p&gt;

&lt;p&gt;The governance program is impeccable at rest. It's nearly blind in motion.&lt;/p&gt;

&lt;p&gt;This isn't a tooling failure. It's a category error. We've built governance around the assumption that data lives in tables and files, and that if we control the endpoints, we control the story. But modern pipelines don't work like that. Data is in constant motion — across brokers, through transformations, into model training sets, out to partner APIs — and most governance frameworks haven't caught up.&lt;/p&gt;




&lt;h2&gt;
  
  
  Why at-rest governance misses the hard cases
&lt;/h2&gt;

&lt;p&gt;Batch-era governance made sense for batch-era architecture. Data moved in scheduled loads. You knew when it arrived. You could scan it, classify it, and apply policy before anyone queried it. The pipeline was basically a delivery truck, and governance happened at the loading dock.&lt;/p&gt;

&lt;p&gt;Streaming and CDC broke that model. Data moves continuously. It gets joined in flight. It gets enriched by external services. It gets filtered, split, and routed to multiple consumers before it ever touches a governed destination. By the time it lands, the sensitive parts may have already leaked, been copied, or sent somewhere the catalog can't see.&lt;/p&gt;

&lt;p&gt;The gap shows up in specific, painful ways:&lt;/p&gt;

&lt;p&gt;Classification after landing is often too late. If a CDC stream carries unmasked credit card numbers through three intermediate topics before they reach the warehouse, the warehouse column-level policy is a Band-Aid on a bleed that happened upstream.&lt;/p&gt;

&lt;p&gt;Batch governance tools don't understand streaming semantics. A data catalog can tell you what's in a table. It can't tell you that a streaming join is pulling PII from one topic and correlating it with behavioral data from another, creating a new composite dataset that nobody reviewed.&lt;/p&gt;

&lt;p&gt;Lineage breaks at the seams. Most lineage tools trace table-to-table relationships. They don't trace the API call that enriches a record mid-pipeline, or the model that trains on a snapshot of the stream, or the reverse ETL job that pushes derived data back to a CRM. The map looks complete until you follow a record through its actual lifecycle.&lt;/p&gt;

&lt;p&gt;Retention policies ignore movement-time copies. You set a 90-day retention on the warehouse. Great. But the stream consumer that materialized a view in Elasticsearch? The S3 bucket that the pipeline writes to for replay? The partner API that received a copy? Retention is only as strong as its weakest replica, and most governance programs don't know where the replicas are.&lt;/p&gt;




&lt;h2&gt;
  
  
  What governance-in-motion actually means
&lt;/h2&gt;

&lt;p&gt;Moving governance into the pipeline doesn't mean replacing your data catalog or your lakehouse security model. It means extending policy enforcement to the places where data is actually moving.&lt;/p&gt;

&lt;p&gt;Here are the four capabilities that matter:&lt;/p&gt;

&lt;h3&gt;
  
  
  Policy at the boundary
&lt;/h3&gt;

&lt;p&gt;Instead of classifying data after it arrives, classify it when it enters the pipeline. A CDC feed from a customer database should carry its classification tags with it — PII, financial data, health records — and those tags should persist through every reshape, join, and routing decision. If a downstream consumer tries to send tagged data to an unapproved destination, the pipeline should block it, not log it and move on.&lt;/p&gt;

&lt;p&gt;This sounds obvious, but most pipelines don't do it. The metadata that matters for governance — classification, consent flags, retention requirements — is usually stripped out during normalization or stored in a separate catalog that the runtime doesn't consult.&lt;/p&gt;

&lt;h3&gt;
  
  
  Approval gates for high-risk flows
&lt;/h3&gt;

&lt;p&gt;Some data movements shouldn't happen without a second look. A pipeline that starts replicating a new table to an external analytics tool. A stream that begins including a newly added column with geolocation data. An AI workflow that wants to send customer transcripts to a third-party model.&lt;/p&gt;

&lt;p&gt;These aren't failures. They're normal operations. But they're also the moments where governance risk concentrates. The right model isn't to block everything and wait for a ticket. It's to let low-risk flows proceed automatically and flag high-risk ones for approval — with the approval gate embedded in the pipeline itself, not in a separate workflow tool that the engineer may or may not remember to use.&lt;/p&gt;

&lt;h3&gt;
  
  
  Retention and deletion that follow the data
&lt;/h3&gt;

&lt;p&gt;When a customer exercises their right to be forgotten, or when a retention period expires, the request needs to reach every copy of the data — not just the warehouse table. That includes stream replays, materialized views, model training snapshots, and partner API caches.&lt;/p&gt;

&lt;p&gt;In practice, this means the pipeline runtime needs to track where data was sent and maintain the mapping between a record's identity and its replicas. Most streaming platforms don't do this. They treat each message as independent, with no memory of where it came from or where it went. Governance in motion requires a different model: messages carry identity, and the runtime maintains provenance.&lt;/p&gt;

&lt;h3&gt;
  
  
  Lineage that includes transformations, not just tables
&lt;/h3&gt;

&lt;p&gt;Table-to-table lineage is useful for batch pipelines. For streaming and CDC, it's incomplete. You need to know that a particular enrichment step appended third-party data, that a join correlated two datasets that were supposed to stay separate, or that a filter silently dropped records that should have been retained for compliance.&lt;/p&gt;

&lt;p&gt;This means lineage can't be a post-hoc scan of query logs. It has to be built into the runtime, capturing what each step did to the data as it happens.&lt;/p&gt;




&lt;h2&gt;
  
  
  The rollout path: start with interfaces
&lt;/h2&gt;

&lt;p&gt;Nobody rebuilds their governance model in one project. The teams that succeed start small and expand.&lt;/p&gt;

&lt;p&gt;The right place to begin is the pipeline boundary — the points where data enters or leaves your system. CDC feeds from production databases. API calls to external services. Reverse ETL jobs that push data back to operational tools. These are the highest-risk, highest-visibility interfaces, and they're where governance failures show up first.&lt;/p&gt;

&lt;p&gt;Pick one interface. Add classification tagging at the source. Add a policy check before the data exits. Add lineage tracking for that one flow. Prove it works. Then expand to the next interface.&lt;/p&gt;

&lt;p&gt;The teams I see doing this well share one trait: they treat governance as a runtime concern, not a documentation concern. The policy isn't written in a wiki and hoped for. It's enforced by the pipeline, tested in CI, and versioned with the code. When the policy changes, the pipeline changes. When the pipeline changes, the policy is revalidated.&lt;/p&gt;




&lt;h2&gt;
  
  
  Where this gets harder than it sounds
&lt;/h2&gt;

&lt;p&gt;I want to be honest about the obstacles, because "just move governance into the pipeline" is easy to say and hard to do.&lt;/p&gt;

&lt;p&gt;Existing tools aren't built for it. Most data catalogs, security scanners, and lineage tools were designed for batch warehouses. They expect scheduled scans, not continuous streams. They expect tables, not topics. Extending them to cover motion usually means custom integration work that the vendor doesn't support.&lt;/p&gt;

&lt;p&gt;Performance matters. Adding classification checks, policy lookups, and lineage logging to a streaming pipeline adds latency. For high-throughput flows, the overhead has to be negligible — which means policy decisions need to be cached, evaluated asynchronously, or pushed to the edge of the pipeline where they don't block the critical path.&lt;/p&gt;

&lt;p&gt;Organizational ownership is fuzzy. Data governance is often owned by a compliance team that doesn't write pipeline code. Platform engineering owns the runtime but doesn't set policy. Getting these teams to agree on what "governance in motion" means, who maintains it, and who gets paged when it breaks is often harder than the technical implementation.&lt;/p&gt;

&lt;p&gt;The standards are still forming. There's no universal protocol for attaching governance metadata to streaming records. No standard API for pipeline policy enforcement. The vendors are building their own models, and they don't interoperate well. If you're betting on a specific approach, you're partly betting on which vendor's model wins.&lt;/p&gt;

&lt;p&gt;&lt;a href="https://media2.dev.to/dynamic/image/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto%2Cformat=auto/https%3A%2F%2Fdev-to-uploads.s3.us-east-2.amazonaws.com%2Fuploads%2Farticles%2Fs766rjb1nxd988dkltcp.jpg" class="article-body-image-wrapper"&gt;&lt;img src="https://media2.dev.to/dynamic/image/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto%2Cformat=auto/https%3A%2F%2Fdev-to-uploads.s3.us-east-2.amazonaws.com%2Fuploads%2Farticles%2Fs766rjb1nxd988dkltcp.jpg" alt=" " width="800" height="533"&gt;&lt;/a&gt;&lt;/p&gt;




&lt;h2&gt;
  
  
  Where layline.io fits
&lt;/h2&gt;

&lt;p&gt;I won't pretend we solve every part of this. At layline.io, we've focused on the runtime layer: a processing engine for both batch and streaming that can carry metadata through transformations, enforce policies at pipeline boundaries, and maintain lineage across the full flow.&lt;/p&gt;

&lt;p&gt;The bet is that governance shouldn't be a separate system sitting next to your pipeline, parsing logs and hoping it catches everything. It should be part of the pipeline itself — embedded in the same runtime that moves the data, enforced by the same code paths that transform it, visible in the same observability layer that tells you whether the pipeline is healthy.&lt;/p&gt;

&lt;p&gt;This matters because the alternative is fragmentation. One tool for cataloging, another for streaming, another for batch, another for lineage, another for policy management. Each integration point is a seam where governance can leak. The teams I talk to are tired of seams.&lt;/p&gt;

&lt;p&gt;We're not the only ones building in this direction. Confluent is pushing stream governance. Airbyte is framing sovereignty as an architectural problem. The whole market is moving toward the same conclusion: data in motion needs the same controls as data at rest. We're building our piece of that stack.&lt;/p&gt;




&lt;h2&gt;
  
  
  The question to ask your governance team
&lt;/h2&gt;

&lt;p&gt;Next time your compliance team signs off on a clean audit, ask them this:&lt;/p&gt;

&lt;p&gt;"Can you show me where customer PII goes between the production database and the warehouse? Not the endpoints — the path. Every topic, every enrichment call, every replica."&lt;/p&gt;

&lt;p&gt;If the answer is a diagram with two boxes and an arrow, you don't have governance. You have hope.&lt;/p&gt;

&lt;p&gt;The good news is that fixing it doesn't require ripping everything out. It requires moving the control point from the destination to the journey. Start at the boundaries. Add policy to the pipeline. Make lineage part of the runtime. The tools are getting better, the patterns are getting clearer, and the teams that figure this out first will have a genuine operational advantage — not just cleaner audits, but fewer incidents, faster recovery, and data infrastructure they can actually trust.&lt;/p&gt;

</description>
      <category>dataengineering</category>
      <category>datagovernance</category>
      <category>datacompliance</category>
      <category>streamprocessing</category>
    </item>
    <item>
      <title>Fresh Enough for an AI Agent? The Data Freshness Budget Nobody Defines</title>
      <dc:creator>Andrew Tan</dc:creator>
      <pubDate>Tue, 08 Sep 2026 13:09:25 +0000</pubDate>
      <link>https://dev.to/andrew_tan_layline/fresh-enough-for-an-ai-agent-the-data-freshness-budget-nobody-defines-18pg</link>
      <guid>https://dev.to/andrew_tan_layline/fresh-enough-for-an-ai-agent-the-data-freshness-budget-nobody-defines-18pg</guid>
      <description>&lt;p&gt;&lt;em&gt;Most teams talk about AI context as a storage problem. The harder question is whether the data is fresh enough for the action the agent is about to take.&lt;/em&gt;&lt;/p&gt;

&lt;p&gt;A lot of AI infrastructure conversations are still aimed at the wrong bottleneck.&lt;/p&gt;

&lt;p&gt;Teams debate vector databases, prompt caching, long-term memory, MCP servers, and which model should sit behind the agent. All of that matters. None of it answers the question that actually determines whether an agent is safe to use in production.&lt;/p&gt;

&lt;p&gt;How fresh is the data when the agent decides to do something?&lt;/p&gt;

&lt;p&gt;That question sounds boring. It is not boring when the agent sends the wrong refund, approves the wrong order, escalates the wrong incident, or calls the wrong tool because it is looking at a customer record from twenty minutes ago.&lt;/p&gt;

&lt;p&gt;Most data teams already understand quality, lineage, and schema drift. Freshness gets treated like a nice-to-have until agents enter the picture. Then freshness stops being a dashboard preference and turns into an operational boundary.&lt;/p&gt;

&lt;h2&gt;
  
  
  Context is not the same as permission
&lt;/h2&gt;

&lt;p&gt;This is the part many teams skip.&lt;/p&gt;

&lt;p&gt;An agent can have plenty of context and still have the wrong context for the action you want it to take. Ten million rows in a warehouse do not help if the one field that matters changed three minutes ago and your sync runs every hour.&lt;/p&gt;

&lt;p&gt;That is the difference between informative data and actionable data.&lt;/p&gt;

&lt;p&gt;Informative data helps an agent explain what happened last week. Actionable data lets it decide what should happen right now.&lt;/p&gt;

&lt;p&gt;Those are not the same workload.&lt;/p&gt;

&lt;p&gt;A support copilot summarizing the last five tickets can tolerate some lag. An agent that decides whether a refund already happened cannot.&lt;/p&gt;

&lt;p&gt;A sales assistant drafting an account brief can work from nightly CRM syncs. An agent that routes a live lead based on current product usage probably needs data that is a few minutes old at most.&lt;/p&gt;

&lt;p&gt;A finance bot preparing a monthly variance summary can stay batch. An agent that freezes a suspicious payout cannot.&lt;/p&gt;

&lt;p&gt;The mistake is treating all AI context as one category called "AI-ready data."&lt;/p&gt;

&lt;p&gt;It is not one category. It is a stack of decisions with different freshness requirements.&lt;/p&gt;

&lt;h2&gt;
  
  
  The freshness budget
&lt;/h2&gt;

&lt;p&gt;The cleanest way to think about this is a freshness budget.&lt;/p&gt;

&lt;p&gt;Every agent task has a maximum acceptable lag between what happened in the source system and what the agent sees when it acts.&lt;/p&gt;

&lt;p&gt;That lag budget depends on the consequence of being wrong.&lt;/p&gt;

&lt;p&gt;Here is a simple version:&lt;/p&gt;

&lt;div class="table-wrapper-paragraph"&gt;&lt;table&gt;
&lt;thead&gt;
&lt;tr&gt;
&lt;th&gt;Agent task&lt;/th&gt;
&lt;th&gt;Typical freshness budget&lt;/th&gt;
&lt;th&gt;What happens if you miss it&lt;/th&gt;
&lt;/tr&gt;
&lt;/thead&gt;
&lt;tbody&gt;
&lt;tr&gt;
&lt;td&gt;Weekly account summary&lt;/td&gt;
&lt;td&gt;24 hours&lt;/td&gt;
&lt;td&gt;Slightly stale narrative&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;Internal KPI Q&amp;amp;A&lt;/td&gt;
&lt;td&gt;1 to 4 hours&lt;/td&gt;
&lt;td&gt;Confusing answers, low trust&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;Sales lead routing&lt;/td&gt;
&lt;td&gt;5 to 15 minutes&lt;/td&gt;
&lt;td&gt;Bad prioritization, slower follow-up&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;Customer support refund decision&lt;/td&gt;
&lt;td&gt;Under 5 minutes&lt;/td&gt;
&lt;td&gt;Duplicate refunds, policy mistakes&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;Fraud or risk intervention&lt;/td&gt;
&lt;td&gt;Seconds to 1 minute&lt;/td&gt;
&lt;td&gt;Real money loss&lt;/td&gt;
&lt;/tr&gt;
&lt;/tbody&gt;
&lt;/table&gt;&lt;/div&gt;

&lt;p&gt;This is not a universal table. It is a forcing function.&lt;/p&gt;

&lt;p&gt;Most teams never define these numbers at all. They just say they want "real-time AI" or "AI-ready pipelines" and hope the stack sorts itself out later.&lt;/p&gt;

&lt;p&gt;It will not.&lt;/p&gt;

&lt;p&gt;If you do not define a freshness budget, the default budget becomes "whatever the pipeline already does." That is usually an accident, not a design choice.&lt;/p&gt;

&lt;h2&gt;
  
  
  Batch is still fine for a lot of AI work
&lt;/h2&gt;

&lt;p&gt;I do not think the answer is to push every workload into streaming.&lt;/p&gt;

&lt;p&gt;That is expensive. It also creates a new class of operational problems if the team does not need it.&lt;/p&gt;

&lt;p&gt;Some AI use cases are perfectly happy on batch:&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;Writing internal summaries&lt;/li&gt;
&lt;li&gt;Preparing research briefs&lt;/li&gt;
&lt;li&gt;Tagging support conversations for trend analysis&lt;/li&gt;
&lt;li&gt;Drafting renewal prep notes&lt;/li&gt;
&lt;li&gt;Enriching planning docs&lt;/li&gt;
&lt;li&gt;Creating weekly executive recaps&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;These tasks benefit from completeness more than immediacy.&lt;/p&gt;

&lt;p&gt;You usually want the full record, not the newest event from fifteen seconds ago.&lt;/p&gt;

&lt;p&gt;This matters because a lot of AI messaging right now frames the future as one giant real-time system. That is a good way to overspend.&lt;/p&gt;

&lt;p&gt;The better question is narrower: where does low-latency data actually change the quality of the action?&lt;/p&gt;

&lt;p&gt;If the answer is nowhere, keep it batch.&lt;/p&gt;

&lt;p&gt;If the answer is specific interfaces, move those interfaces first.&lt;/p&gt;

&lt;h2&gt;
  
  
  The dangerous middle
&lt;/h2&gt;

&lt;p&gt;The real trouble is not clearly batch and not truly real-time.&lt;/p&gt;

&lt;p&gt;It is the mushy middle where teams have pipelines that update every hour, or every thirty minutes, or whenever a connector feels like catching up, and then they hand that data to an agent with permission to act.&lt;/p&gt;

&lt;p&gt;That is where mistakes get expensive.&lt;/p&gt;

&lt;p&gt;Hourly freshness sounds decent until you map it to the workflow.&lt;/p&gt;

&lt;p&gt;If a customer upgrades at 10:02 and the agent sees the old plan until 11:00, you have almost an hour where it can deny an entitlement the customer already paid for.&lt;/p&gt;

&lt;p&gt;If an order is canceled at 2:11 and your inventory assistant does not learn that until 3:00, it can reorder stock you no longer need.&lt;/p&gt;

&lt;p&gt;If a chargeback flag lands in one system before another, your risk agent may see a healthy account and approve the next transaction with full confidence.&lt;/p&gt;

&lt;p&gt;Nothing is obviously broken in those examples. The jobs ran. The tables updated. The dashboard probably looks fine.&lt;/p&gt;

&lt;p&gt;The problem is that the action window is tighter than the data window.&lt;/p&gt;

&lt;p&gt;That gap is where agent failures live.&lt;/p&gt;

&lt;p&gt;&lt;a href="https://media2.dev.to/dynamic/image/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto%2Cformat=auto/https%3A%2F%2Fdev-to-uploads.s3.us-east-2.amazonaws.com%2Fuploads%2Farticles%2Firfj69a60p7ssm7bet1x.jpg" class="article-body-image-wrapper"&gt;&lt;img src="https://media2.dev.to/dynamic/image/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto%2Cformat=auto/https%3A%2F%2Fdev-to-uploads.s3.us-east-2.amazonaws.com%2Fuploads%2Farticles%2Firfj69a60p7ssm7bet1x.jpg" width="800" height="420"&gt;&lt;/a&gt;&lt;/p&gt;

&lt;h2&gt;
  
  
  Approval gates are a freshness strategy too
&lt;/h2&gt;

&lt;p&gt;There is another point that gets lost in the rush to automate.&lt;/p&gt;

&lt;p&gt;Sometimes the right answer is not faster data. Sometimes the right answer is an approval gate.&lt;/p&gt;

&lt;p&gt;If an agent works on stale or ambiguous context, you do not always need to block the use case forever. You may just need to change the last step.&lt;/p&gt;

&lt;p&gt;Let the agent gather data. Let it draft the response. Let it recommend the action. Then require a human approval if the freshness budget is not met or if the decision touches money, compliance, access, or customer-facing risk.&lt;/p&gt;

&lt;p&gt;This is not a failure of automation. It is part of the design.&lt;/p&gt;

&lt;p&gt;Good AI systems do not just think about model quality. They think about when not to act alone.&lt;/p&gt;

&lt;p&gt;That is especially true when the underlying data arrives through a mix of batch syncs, CDC pipelines, event streams, and external APIs that all move at different speeds.&lt;/p&gt;

&lt;h2&gt;
  
  
  You need freshness by interface, not by platform slogan
&lt;/h2&gt;

&lt;p&gt;A lot of vendor positioning blurs this on purpose.&lt;/p&gt;

&lt;p&gt;The promise usually sounds something like this: connect all your data, feed your agents, unlock real-time decisions.&lt;/p&gt;

&lt;p&gt;Fine. But which decisions?&lt;/p&gt;

&lt;p&gt;You do not need one giant freshness target for the whole company. You need freshness targets by interface.&lt;/p&gt;

&lt;p&gt;Start with the places where an agent crosses from advice into action:&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;approving or denying something&lt;/li&gt;
&lt;li&gt;sending a customer communication&lt;/li&gt;
&lt;li&gt;changing access or entitlements&lt;/li&gt;
&lt;li&gt;moving money&lt;/li&gt;
&lt;li&gt;opening or closing incidents&lt;/li&gt;
&lt;li&gt;triggering downstream systems automatically&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;Those interfaces deserve explicit lag limits, monitoring, fallback behavior, and ownership.&lt;/p&gt;

&lt;p&gt;If the pipeline misses the budget, what should happen?&lt;/p&gt;

&lt;p&gt;Maybe the action pauses.&lt;/p&gt;

&lt;p&gt;Maybe the agent can still draft but not send.&lt;/p&gt;

&lt;p&gt;Maybe it can operate on a narrow safe subset of tools.&lt;/p&gt;

&lt;p&gt;Maybe it needs a human in the loop.&lt;/p&gt;

&lt;p&gt;The point is to define this before the first incident teaches it to you.&lt;/p&gt;

&lt;h2&gt;
  
  
  What this means for the data stack
&lt;/h2&gt;

&lt;p&gt;Once you define freshness budgets, the stack conversation gets easier.&lt;/p&gt;

&lt;p&gt;You can stop arguing about batch versus streaming as ideology.&lt;/p&gt;

&lt;p&gt;Some interfaces need event-driven processing. Some need CDC with tight SLAs. Some are fine with scheduled syncs. Some need a hybrid path where the same workflow handles both historical backfills and low-latency updates.&lt;/p&gt;

&lt;p&gt;That last category is the one that gets painful fast if your architecture splits batch and streaming into separate systems.&lt;/p&gt;

&lt;p&gt;Now you are maintaining two versions of the same business logic. One answers historical questions. One powers live actions. They drift. Your agent gets inconsistent state depending on which path it touched.&lt;/p&gt;

&lt;p&gt;That is why I think the better long-term design is not "real-time everywhere." It is one runtime that can handle both batch and streaming, plus the orchestration around them, without forcing the team to rebuild the workflow every time a freshness requirement changes.&lt;/p&gt;

&lt;p&gt;That is also where &lt;strong&gt;layline.io&lt;/strong&gt; fits. The point is not to turn every pipeline into a stream. The point is to let teams tighten freshness where the action requires it, keep batch where it is enough, and manage both in one operational model.&lt;/p&gt;

&lt;h2&gt;
  
  
  The practical test
&lt;/h2&gt;

&lt;p&gt;If your team is rolling out AI agents right now, ask one question for every agent action:&lt;/p&gt;

&lt;p&gt;What is the maximum age of data we can tolerate before this action becomes unsafe, wrong, or embarrassing?&lt;/p&gt;

&lt;p&gt;Write the number down.&lt;/p&gt;

&lt;p&gt;If nobody can answer it, you do not have an AI problem yet. You have a requirements problem.&lt;/p&gt;

&lt;p&gt;And if the answer is "it depends," that is fine. Break it down by interface until it stops depending.&lt;/p&gt;

&lt;p&gt;The teams that do this well will not be the ones with the loudest AI stack. They will be the ones that know which decisions need seconds, which need minutes, which can wait until tomorrow, and where a human still belongs in the loop.&lt;/p&gt;

&lt;p&gt;That sounds less exciting than the usual agent demos.&lt;/p&gt;

&lt;p&gt;It is also the difference between a helpful system and one that creates a new kind of on-call rotation.&lt;/p&gt;

</description>
      <category>agents</category>
      <category>dataengineering</category>
    </item>
    <item>
      <title>Why 85% of Enterprises Aren't Ready for AI</title>
      <dc:creator>Andrew Tan</dc:creator>
      <pubDate>Tue, 01 Sep 2026 13:50:43 +0000</pubDate>
      <link>https://dev.to/andrew_tan_layline/why-85-of-enterprises-arent-ready-for-ai-669</link>
      <guid>https://dev.to/andrew_tan_layline/why-85-of-enterprises-arent-ready-for-ai-669</guid>
      <description>&lt;p&gt;&lt;em&gt;Fivetran's 2026 benchmark found 85% of enterprises running agentic AI on infrastructure that isn't ready. The model isn't the problem. The data pipelines feeding it are.&lt;/em&gt;&lt;/p&gt;

&lt;p&gt;Here's a pattern showing up in an increasing number of places:&lt;/p&gt;

&lt;p&gt;A company spends six months evaluating LLMs: They run benchmarks, negotiate contracts, build a proof of concept. The model looks great. Then they deploy it to production and the agents start hallucinating in ways that have nothing to do with the model at all.&lt;/p&gt;

&lt;p&gt;The AI is fine, but the data feeding it isn't.&lt;/p&gt;

&lt;p&gt;This is what Fivetran's Enterprise Data Infrastructure Benchmark Report 2026 is actually measuring when it says 85% of enterprises aren't ready for agentic AI. Not that their models are wrong, but that their pipelines are wrong.&lt;/p&gt;




&lt;h2&gt;
  
  
  What "not ready" actually means
&lt;/h2&gt;

&lt;p&gt;The phrase "AI-ready infrastructure" gets used to sell a lot of things. It usually means something vague about cloud scalability. That's not what the Fivetran report is measuring.&lt;/p&gt;

&lt;p&gt;The three concrete problems they identified:&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;Stale data.&lt;/strong&gt; The agent is reasoning over yesterday's state. For a customer service agent, that means it doesn't know a refund was already issued. For a fraud detection agent, it's working from patterns that are 18 hours old. Batch pipelines running hourly or nightly can't support agents that need to act on what's happening now.&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;No schema enforcement.&lt;/strong&gt; Source systems change constantly. Columns get renamed, types get widened, new fields appear. If your pipeline doesn't enforce schema contracts downstream, an agent can receive malformed data that looks valid — and confidently act on it.&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;No observability.&lt;/strong&gt; Most pipeline monitoring tells you whether a job ran. It doesn't tell you whether the data is correct. An agent can be consuming data from a pipeline that's technically running but silently dropping 20% of events. You won't know until someone notices the agent is acting strangely, and by then the damage is done.&lt;/p&gt;

&lt;p&gt;These aren't new problems. They're the same data quality issues that have plagued analytics teams for years. What changed is the blast radius. A data quality problem in a BI dashboard is a bad chart. A data quality problem in an agentic system is an autonomous decision made on bad information.&lt;/p&gt;




&lt;h2&gt;
  
  
  The stack mismatch
&lt;/h2&gt;

&lt;p&gt;The AI-ready data stack problem is a deep mismatch in architecture assumptions.&lt;/p&gt;

&lt;p&gt;Most enterprise data stacks were built around batch processing. Nightly ETL jobs. Daily warehouse refreshes. Dashboards that update every morning. The entire system was optimized for throughput over freshness.&lt;/p&gt;

&lt;p&gt;AI agents have different requirements. They need data that's current, not just accurate. They need to act on what's happening in the last few seconds or minutes, not the last twelve hours. And when they're wrong, they need the system to catch it — not just log it and move on.&lt;/p&gt;

&lt;p&gt;&lt;a href="https://media2.dev.to/dynamic/image/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto%2Cformat=auto/https%3A%2F%2Fdev-to-uploads.s3.us-east-2.amazonaws.com%2Fuploads%2Farticles%2F8qrfpdalqw85vg6ehncw.jpg" class="article-body-image-wrapper"&gt;&lt;img src="https://media2.dev.to/dynamic/image/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto%2Cformat=auto/https%3A%2F%2Fdev-to-uploads.s3.us-east-2.amazonaws.com%2Fuploads%2Farticles%2F8qrfpdalqw85vg6ehncw.jpg" width="800" height="420"&gt;&lt;/a&gt;&lt;/p&gt;

&lt;p&gt;The companies that are ready — the 15% — didn't necessarily replace their entire stack. Most of them shifted to real-time where it matters and kept batch where it makes sense. The customer order stream runs in real-time. The quarterly cost accounting still runs nightly. The difference is intentionality: they made explicit decisions about which data needs freshness and built pipelines accordingly.&lt;/p&gt;




&lt;h2&gt;
  
  
  The three properties that actually matter
&lt;/h2&gt;

&lt;p&gt;There's a lot of noise about what "AI-ready" infrastructure requires. Usually it's a vendor's checklist that conveniently maps to their product. So here's the version that maps to the actual failure modes:&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;Freshness.&lt;/strong&gt; Not just "real-time for everything" — that's expensive and often unnecessary. But for the data your agents actually act on, you need to know the lag and have guarantees about it. If your customer data is 4 minutes old, fine — as long as your agent knows that and acts on it. What breaks agents is when they assume data is current and it isn't.&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;Consistency.&lt;/strong&gt; Agents often combine data from multiple sources to make a decision. If those sources are running on different schedules or at different freshness levels, you get subtle inconsistencies that are hard to debug. A customer lookup says the account is active; the transaction stream hasn't caught up and shows it as pending closure. The agent makes a decision that's correct for each source individually but wrong for the combined state.&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;Observability at the event level.&lt;/strong&gt; Pipeline monitoring tells you about jobs. Agent reliability requires monitoring individual events. Are events being processed or dropped? Is the schema matching what the agent expects? Are there bursts that overwhelm downstream consumers? This is a different class of monitoring than most teams have built.&lt;/p&gt;




&lt;h2&gt;
  
  
  What this means for the build vs. buy decision
&lt;/h2&gt;

&lt;p&gt;The AI wave is surfacing a bill that's been accumulating in a lot of engineering orgs: the cost of treating data infrastructure as a solved problem.&lt;/p&gt;

&lt;p&gt;Teams that built custom batch pipelines two years ago and called it done are now facing a hard choice: retrofit real-time onto a system that wasn't designed for it, or rebuild. Neither option is cheap. Retrofitting tends to produce the two-pipeline problem — a batch system for history, a streaming system for real-time, two codebases doing roughly the same thing with slightly different logic and perpetually diverging results.&lt;/p&gt;

&lt;p&gt;The teams handling this well are the ones who don't have to make that choice. When batch and streaming run as the same pipelines with the same tooling, switching a workflow from hourly to real-time is a configuration change, not a rewrite. The observability, the schema enforcement, the failure handling — it comes with the platform, not as a custom build on top.&lt;/p&gt;

&lt;p&gt;That's what &lt;strong&gt;layline.io&lt;/strong&gt; is built around. Not just real-time for its own sake, but the ability to make explicit, intentional decisions about data freshness across the stack — without maintaining two separate systems to get there.&lt;/p&gt;




&lt;h2&gt;
  
  
  The question to sit with
&lt;/h2&gt;

&lt;p&gt;The 85% number is striking. But the more interesting question is: how many of those enterprises &lt;em&gt;know&lt;/em&gt; they're in that 85%?&lt;/p&gt;

&lt;p&gt;The teams that are in trouble aren't usually the ones with obviously broken pipelines. They're the ones with pipelines that appear to work — jobs running, dashboards loading, no active incidents — but with silent reliability issues that only show up when an AI agent starts making consequential decisions on the data.&lt;/p&gt;

&lt;p&gt;If your agents are acting strangely and you've already ruled out the model, look at the data.&lt;/p&gt;

</description>
      <category>aireadiness</category>
      <category>datapipeline</category>
      <category>realtimedata</category>
      <category>enterpriseai</category>
    </item>
    <item>
      <title>The Hidden Costs of Building Your Own Batch-Streaming Integration Layer</title>
      <dc:creator>Andrew Tan</dc:creator>
      <pubDate>Tue, 25 Aug 2026 10:36:20 +0000</pubDate>
      <link>https://dev.to/andrew_tan_layline/the-hidden-costs-of-building-your-own-batch-streaming-integration-layer-225l</link>
      <guid>https://dev.to/andrew_tan_layline/the-hidden-costs-of-building-your-own-batch-streaming-integration-layer-225l</guid>
      <description>&lt;h2&gt;
  
  
  The Hidden Costs of Building Your Own Batch-Streaming Integration Layer
&lt;/h2&gt;

&lt;p&gt;&lt;em&gt;With AI-assisted coding, building your own data pipelines looks cheaper than ever. But the real costs aren't in the initial build—they're in the maintenance, the on-call rotations, and the accumulated complexity that compounds over time.&lt;/em&gt;&lt;/p&gt;

&lt;p&gt;Here's a conversation that keeps happening:&lt;/p&gt;

&lt;blockquote&gt;
&lt;p&gt;&lt;strong&gt;Engineering Manager:&lt;/strong&gt; "We need a new data pipeline for the customer analytics project."&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;Senior Engineer:&lt;/strong&gt; "I can build that. With Cursor and Copilot, I can have the core logic done in a couple of days."&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;EM:&lt;/strong&gt; "What about maintenance?"&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;SE:&lt;/strong&gt; "It's just a Python script with some Airflow orchestration. How hard can it be?"&lt;/p&gt;
&lt;/blockquote&gt;

&lt;p&gt;Three months later, the engineer who built it is on vacation, the pipeline is failing silently, and nobody can figure out why the customer segment counts don't match the source system. The "simple Python script" has grown to 2,400 lines, touches three different databases, and has exactly zero documentation about what the business logic is actually supposed to do.&lt;/p&gt;

&lt;p&gt;The AI coding revolution has made the &lt;em&gt;build&lt;/em&gt; decision feel almost free. What it hasn't changed is the &lt;em&gt;own&lt;/em&gt; decision — and that's where most of the cost lives.&lt;/p&gt;




&lt;h2&gt;
  
  
  The Honest Accounting
&lt;/h2&gt;

&lt;p&gt;When teams estimate the cost of building their own data integration layer, they usually model something like this:&lt;/p&gt;

&lt;div class="table-wrapper-paragraph"&gt;&lt;table&gt;
&lt;thead&gt;
&lt;tr&gt;
&lt;th&gt;Cost Item&lt;/th&gt;
&lt;th&gt;Estimated&lt;/th&gt;
&lt;/tr&gt;
&lt;/thead&gt;
&lt;tbody&gt;
&lt;tr&gt;
&lt;td&gt;Initial development&lt;/td&gt;
&lt;td&gt;2-3 weeks of engineer time&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;Infrastructure&lt;/td&gt;
&lt;td&gt;Existing Kubernetes cluster&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;Maintenance&lt;/td&gt;
&lt;td&gt;"Just keep it running"&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;&lt;strong&gt;Total first-year cost&lt;/strong&gt;&lt;/td&gt;
&lt;td&gt;&lt;strong&gt;~$30K loaded&lt;/strong&gt;&lt;/td&gt;
&lt;/tr&gt;
&lt;/tbody&gt;
&lt;/table&gt;&lt;/div&gt;

&lt;p&gt;Here's what the spreadsheet actually looks like after twelve months:&lt;/p&gt;

&lt;div class="table-wrapper-paragraph"&gt;&lt;table&gt;
&lt;thead&gt;
&lt;tr&gt;
&lt;th&gt;Cost Item&lt;/th&gt;
&lt;th&gt;Actual&lt;/th&gt;
&lt;/tr&gt;
&lt;/thead&gt;
&lt;tbody&gt;
&lt;tr&gt;
&lt;td&gt;Initial development&lt;/td&gt;
&lt;td&gt;4 weeks (scope crept)&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;Infrastructure&lt;/td&gt;
&lt;td&gt;$8K/year in compute, storage, network&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;On-call burden&lt;/td&gt;
&lt;td&gt;15-20 hours/month paging, debugging, fixing&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;Schema drift incidents&lt;/td&gt;
&lt;td&gt;3 major, 8 minor (data quality failures)&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;Failed retry handling&lt;/td&gt;
&lt;td&gt;Built ad-hoc, never quite right&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;Documentation debt&lt;/td&gt;
&lt;td&gt;Still zero, now critical&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;Knowledge silo risk&lt;/td&gt;
&lt;td&gt;One engineer understands it&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;&lt;strong&gt;Total first-year cost&lt;/strong&gt;&lt;/td&gt;
&lt;td&gt;&lt;strong&gt;~$85K loaded + opportunity cost&lt;/strong&gt;&lt;/td&gt;
&lt;/tr&gt;
&lt;/tbody&gt;
&lt;/table&gt;&lt;/div&gt;

&lt;p&gt;The gap isn't because engineers are bad at estimation. It's because the spreadsheet only captures the work you can see upfront. The real costs accumulate invisibly: the 2 AM pages, the "quick fixes" that become permanent, the subtle data corruption that takes days to detect.&lt;/p&gt;




&lt;h2&gt;
  
  
  The Two-Pipeline Problem
&lt;/h2&gt;

&lt;p&gt;There's a specific failure mode that hits teams building their own batch-streaming infrastructure: the divergence problem.&lt;/p&gt;

&lt;p&gt;You start with batch. It's straightforward. You write a job that runs every hour, extracts data, transforms it, loads it somewhere. Works fine.&lt;/p&gt;

&lt;p&gt;Then the business asks for real-time. "Can we get this data in seconds instead of hours?"&lt;/p&gt;

&lt;p&gt;So you build a streaming pipeline. Kafka, maybe Flink or Spark Streaming. It consumes the same source data and delivers to the same destination. But the transformation logic is different — streaming has different constraints, different state management, different failure modes. You can't just port the batch code over.&lt;/p&gt;

&lt;p&gt;Now you have two pipelines doing roughly the same thing. They produce slightly different results because the batch join is outer and the streaming join is inner, or because the batch job handles late data differently than the streaming window. When someone asks why the numbers don't match, you have to debug both systems.&lt;/p&gt;

&lt;p&gt;Six months in, you've got:&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;Two codebases to maintain&lt;/li&gt;
&lt;li&gt;Two sets of infrastructure to monitor
&lt;/li&gt;
&lt;li&gt;Two failure modes to understand&lt;/li&gt;
&lt;li&gt;Two on-call rotations (or one very unhappy person)&lt;/li&gt;
&lt;li&gt;And one persistent question: why can't we just have one pipeline?&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;&lt;a href="https://media2.dev.to/dynamic/image/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto%2Cformat=auto/https%3A%2F%2Fdev-to-uploads.s3.us-east-2.amazonaws.com%2Fuploads%2Farticles%2Fyti3d0p3ka80mhsfs635.jpg" class="article-body-image-wrapper"&gt;&lt;img src="https://media2.dev.to/dynamic/image/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto%2Cformat=auto/https%3A%2F%2Fdev-to-uploads.s3.us-east-2.amazonaws.com%2Fuploads%2Farticles%2Fyti3d0p3ka80mhsfs635.jpg" width="800" height="420"&gt;&lt;/a&gt;&lt;/p&gt;

&lt;p&gt;The honest answer: because batch and streaming are genuinely different paradigms, and most DIY stacks aren't built to unify them.&lt;/p&gt;




&lt;h2&gt;
  
  
  The Hidden Complexity Multipliers
&lt;/h2&gt;

&lt;p&gt;Beyond the obvious costs, there are three complexity multipliers that don't show up in initial estimates:&lt;/p&gt;

&lt;h3&gt;
  
  
  Schema Evolution
&lt;/h3&gt;

&lt;p&gt;Your source system changes. A column gets renamed. A type gets widened. A new nullable field appears. In a managed platform, this is handled. In your custom pipeline, it's a code change, a deployment, and a prayer that you didn't break downstream consumers.&lt;/p&gt;

&lt;p&gt;The real cost isn't the change itself. It's the coordination: notifying every team that consumes this data, updating their schemas, testing the integration, rolling back if something goes wrong. A two-hour code change becomes a two-week project.&lt;/p&gt;

&lt;h3&gt;
  
  
  Failure Handling at Scale
&lt;/h3&gt;

&lt;p&gt;A simple retry loop is easy. Exponential backoff, a dead letter queue, some alerting — you can build that in an afternoon.&lt;/p&gt;

&lt;p&gt;But production failure handling is fractal. What happens when the destination is down for an hour? What happens when a message is too large? What happens when a schema mismatch causes a parse failure? What happens when the same event gets delivered twice? What happens when network partitions create split-brain situations?&lt;/p&gt;

&lt;p&gt;Each edge case needs handling. Each handler needs testing. Each test needs maintenance. The "simple retry logic" grows into a distributed systems concern that nobody on the team has deep expertise in.&lt;/p&gt;

&lt;h3&gt;
  
  
  Observability Gaps
&lt;/h3&gt;

&lt;p&gt;You need to know: Is the pipeline running? Is it keeping up with the source? Are events being processed or dropped? What's the latency? What's the error rate? What's the cost per million events?&lt;/p&gt;

&lt;p&gt;Building this visibility isn't just adding a metrics endpoint. It's designing the right metrics, building the dashboards, setting the right alerts (not too noisy, not too quiet), and training the team to interpret them. It's another system to build, maintain, and debug.&lt;/p&gt;




&lt;h2&gt;
  
  
  When Building Actually Makes Sense
&lt;/h2&gt;

&lt;p&gt;I want to be fair. There are situations where building your own integration layer is the right call:&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;You have extremely specific requirements&lt;/strong&gt; that no vendor handles well — unusual data formats, custom security constraints, exotic deployment environments.&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;You have the team for it&lt;/strong&gt; — distributed systems engineers who've operated Kafka at scale, who understand exactly-once semantics, who've debugged backpressure problems at 3 AM.&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;It's a genuine differentiator&lt;/strong&gt; — the data processing layer is core to your product, not just infrastructure. You're not building a pipeline; you're building a competitive advantage.&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;You're at a scale where vendor costs exceed build costs&lt;/strong&gt; — though be honest about what "build cost" includes. Most teams underestimate by 2-3x.&lt;/p&gt;

&lt;p&gt;For everyone else, the calculation usually favors buying — if you account for the full cost of ownership.&lt;/p&gt;




&lt;h2&gt;
  
  
  The Vendor Evaluation That Actually Matters
&lt;/h2&gt;

&lt;p&gt;If you're comparing vendors, the feature matrix is the wrong place to start. Most platforms have similar capabilities on paper. What matters is the operational model:&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;How do they handle the 2 AM problem?&lt;/strong&gt; When something breaks in production, who gets paged? Is it your team debugging their infrastructure, or their team debugging your pipeline?&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;What's the migration path if you leave?&lt;/strong&gt; Data pipelines are sticky. Understand what it costs to extract your logic and move it elsewhere.&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;Do they unify batch and streaming?&lt;/strong&gt; Or will you end up with two pipelines anyway, just in someone else's infrastructure?&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;What's the real TCO?&lt;/strong&gt; Include training, integration time, the cost of waiting for features you need, and the opportunity cost of engineering time spent managing the platform.&lt;/p&gt;




&lt;h2&gt;
  
  
  Where layline.io Fits
&lt;/h2&gt;

&lt;p&gt;I won't pretend this is an unbiased take. At &lt;strong&gt;layline.io&lt;/strong&gt;, we built a platform specifically for teams who've done the honest accounting and decided that building isn't the right call.&lt;/p&gt;

&lt;p&gt;The core bet: batch and streaming shouldn't be separate pipelines. They should be the same workflows, the same tooling, the same team. When you need real-time, you don't rebuild. You adjust a configuration.&lt;/p&gt;

&lt;p&gt;The operational burden sits with us. Schema evolution, failure handling, observability — that's the platform's job, not yours. Your team focuses on the business logic, not the distributed systems plumbing.&lt;/p&gt;

&lt;p&gt;Is it cheaper than building your own? That depends on how honestly you account for the build cost. If you're counting two weeks of development and calling it done, probably not. If you're including the on-call rotation, the maintenance burden, the schema drift incidents, and the opportunity cost of engineers not building product features — then usually, yes.&lt;/p&gt;




&lt;h2&gt;
  
  
  The Question to Ask
&lt;/h2&gt;

&lt;p&gt;Before your team commits to building, ask this:&lt;/p&gt;

&lt;p&gt;"If we build this ourselves, who owns the 2 AM page when it breaks six months from now? And do they know what they're signing up for?"&lt;/p&gt;

&lt;p&gt;If the answer is clear and everyone understands the commitment, build away. If there's hesitation, or if the answer is "we'll figure that out later," do the honest accounting. The numbers might surprise you.&lt;/p&gt;

</description>
      <category>dataengineering</category>
      <category>buildvsbuy</category>
      <category>dataintegration</category>
      <category>batchstreaming</category>
    </item>
    <item>
      <title>The News Everyone Saw Coming</title>
      <dc:creator>Andrew Tan</dc:creator>
      <pubDate>Tue, 25 Aug 2026 09:19:24 +0000</pubDate>
      <link>https://dev.to/andrew_tan_layline/the-news-everyone-saw-coming-3a1g</link>
      <guid>https://dev.to/andrew_tan_layline/the-news-everyone-saw-coming-3a1g</guid>
      <description>&lt;p&gt;&lt;em&gt;Dagster joining Prefect signals the end of standalone orchestrators. The winners will be unified platforms that combine orchestration and processing — and that's exactly where the market is heading.&lt;/em&gt;&lt;/p&gt;

&lt;p&gt;In August 2026, Dagster Labs announced it would be joining forces with Prefect. Two of the most visible modern orchestration tools — both born as reactions to Airflow's limitations — are now under one roof. The press releases talk about "combining strengths" and "accelerating the future of data workflows."&lt;/p&gt;

&lt;p&gt;The reality is simpler: the standalone orchestrator market is consolidating, and fast.&lt;/p&gt;

&lt;p&gt;This isn't a surprise to anyone who's been watching. Venture funding for orchestration-only startups dried up two years ago. The category leaders have been searching for exits or additional funding rounds with increasingly defensive terms. Customers have been asking harder questions about roadmaps, pricing stability, and long-term viability.&lt;/p&gt;

&lt;p&gt;What's different now is the clarity. Dagster and Prefect joining isn't just another acquisition. It's confirmation that standalone orchestration — scheduling tasks, managing dependencies, handling retries — isn't a sustainable standalone business. The tools that survive will be the ones that do more.&lt;/p&gt;

&lt;h2&gt;
  
  
  The Pattern: What Happens After the Press Release
&lt;/h2&gt;

&lt;p&gt;Vendor consolidation in enterprise software follows a predictable script. The announcements are always optimistic. The outcomes for customers are more mixed.&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;Pricing changes, usually upward.&lt;/strong&gt; The combined entity needs to show returns. "Synergies" often translate to reduced discount flexibility, new tier structures, or module-based pricing that used to be included. The Talend customers who saw renewal jumps after the Qlik acquisition aren't outliers. They're the norm.&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;Roadmap shifts, sometimes dramatically.&lt;/strong&gt; Features that don't serve the combined product strategy get deprioritized. The Dagster asset model and the Prefect flow model may both survive, or one may become the "legacy" approach that receives maintenance-only updates. Teams betting on specific differentiators find themselves on the wrong side of architectural bets.&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;Integration debt accumulates.&lt;/strong&gt; The tools don't merge instantly. For 12-24 months, customers run on "combined" platforms that are really two separate codebases with integration layers. Bug fixes take longer because they have to work across both systems. Documentation drifts out of sync. The migration path from "old" to "new" is promised but delayed.&lt;/p&gt;

&lt;p&gt;None of this is malicious. It's just what happens when point solutions in a shrinking market try to survive. The standalone orchestrator category is consolidating because the economics stopped working.&lt;/p&gt;

&lt;h2&gt;
  
  
  Why the Winners Will Be Unified Platforms
&lt;/h2&gt;

&lt;p&gt;Here's the part the consolidation story misses: the tools that survive won't be orchestrators at all. They'll be platforms that happen to include orchestration.&lt;/p&gt;

&lt;p&gt;The standalone orchestrators tried to differentiate on scheduling models, developer experience, or observability. They treated orchestration as the product. But orchestration was never the end goal — it was always a means to an end. Teams don't wake up wanting better task schedulers. They wake up wanting reliable data pipelines.&lt;/p&gt;

&lt;p&gt;Modern data infrastructure is moving toward unified platforms for a simple reason: the split between "orchestration" and "processing" is artificial. When your orchestrator (Airflow, Dagster, Prefect) is separate from your processing engine (Spark, dbt, custom Python), you pay a coordination tax. Multiple mental models. Multiple monitoring systems. Multiple failure modes at the integration seams.&lt;/p&gt;

&lt;p&gt;The platforms that are winning — Databricks, Snowflake, and a new generation of unified data infrastructure — don't treat orchestration as a separate concern. It's built in. Your workflows schedule themselves, retry on failure, enforce dependencies, and trigger downstream work without a separate coordination layer.&lt;/p&gt;

&lt;p&gt;This is where the market is heading. Not more standalone orchestrators. Fewer seams between orchestration and execution.&lt;/p&gt;

&lt;h2&gt;
  
  
  What This Means for Teams Making Choices Now
&lt;/h2&gt;

&lt;p&gt;If you're running production workflows on Dagster, Prefect, or any other orchestration tool facing consolidation pressure, you have an opportunity. The market transition creates a window to move to something better — not just different.&lt;/p&gt;

&lt;p&gt;Here's what to look for in a platform that will survive the consolidation:&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;Unified batch and streaming in one runtime.&lt;/strong&gt; The split between "batch orchestrator" and "streaming processor" is another artificial seam that's collapsing. Teams need both. Maintaining separate tools for scheduled jobs and real-time flows doesn't make sense anymore.&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;Orchestration integrated with processing, not bolted on.&lt;/strong&gt; The scheduler should understand your data, not just your task dependencies. When a step fails, you want the system to know what data was affected, not just that a task returned a non-zero exit code.&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;Sustainable business model, not venture-scale growth targets.&lt;/strong&gt; The consolidation is happening because the standalone orchestrator market couldn't support venture-scale returns. Look for platforms with clear paths to profitability, reasonable pricing models, and business structures that don't require acquisition or IPO to survive.&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;Clear migration paths from the tools being consolidated.&lt;/strong&gt; The best platforms right now are the ones actively helping teams migrate from Dagster, Prefect, and Airflow — not because they're orchestrators, but because they're proving they can replace the entire coordination layer with something simpler.&lt;/p&gt;

&lt;p&gt;&lt;a href="https://media2.dev.to/dynamic/image/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto%2Cformat=auto/https%3A%2F%2Fdev-to-uploads.s3.us-east-2.amazonaws.com%2Fuploads%2Farticles%2Fpp46dfts283by7ezzrsu.jpg" class="article-body-image-wrapper"&gt;&lt;img src="https://media2.dev.to/dynamic/image/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto%2Cformat=auto/https%3A%2F%2Fdev-to-uploads.s3.us-east-2.amazonaws.com%2Fuploads%2Farticles%2Fpp46dfts283by7ezzrsu.jpg" width="800" height="420"&gt;&lt;/a&gt;&lt;/p&gt;

&lt;h2&gt;
  
  
  Where We Fit in This Transition
&lt;/h2&gt;

&lt;p&gt;At layline.io, we've been building what the market is moving toward: a unified platform for both batch and streaming data processing where orchestration is intrinsic, not external.&lt;/p&gt;

&lt;p&gt;We didn't set out to build a better orchestrator. We set out to eliminate the need for separate orchestration entirely. When your processing engine can schedule itself, retry intelligently, and maintain lineage without a separate coordination layer, the "orchestrator" category becomes a legacy concept.&lt;/p&gt;

&lt;p&gt;The consolidation of standalone orchestrators validates this direction. The market is telling us what we already knew: teams are tired of maintaining separate scheduling layers on top of their actual data work. They want infrastructure that handles the full lifecycle — from event ingestion through transformation to delivery — without handoffs between systems.&lt;/p&gt;

&lt;p&gt;For teams currently on Dagster or Prefect, this is actually good news. The consolidation creates urgency to evaluate alternatives, and the alternatives have gotten significantly better. A platform that handles both your scheduled batch jobs and your real-time event processing, with unified observability and no coordination seams, isn't a risky bet on a new category. It's the stable, proven direction the whole market is moving.&lt;/p&gt;

&lt;h2&gt;
  
  
  The Consolidation Creates Opportunity
&lt;/h2&gt;

&lt;p&gt;Dagster and Prefect joining forces won't be the last move in this market. Kestra will face the same pressure. Airflow's position is stable but not growing. The standalone orchestrator category is shrinking toward a few acquired survivors and gradual absorption into platforms.&lt;/p&gt;

&lt;p&gt;This isn't a crisis for data teams. It's a clearing of the landscape. The fragmentation of the last five years — five different orchestrators, three different streaming systems, separate monitoring for each — is giving way to consolidation around unified platforms.&lt;/p&gt;

&lt;p&gt;The teams that come out ahead will be the ones that treat this transition as an upgrade opportunity, not a migration burden. The platforms you're moving to are better than the tools you're leaving. They're simpler to operate, cheaper to maintain, and designed for the workloads you're actually running.&lt;/p&gt;

&lt;p&gt;The consolidation should excite you if you've been waiting for the data infrastructure market to mature. The standalone tool era is ending. The unified platform era is beginning. And that's exactly what most data teams actually need.&lt;/p&gt;




&lt;p&gt;&lt;em&gt;If you're evaluating your orchestration strategy or considering alternatives to consolidated vendors, &lt;a href="https://layline.io/contact" rel="noopener noreferrer"&gt;get in touch&lt;/a&gt;. We're helping teams migrate from standalone orchestrators to unified platforms — and the results are consistently better than expected.&lt;/em&gt;&lt;/p&gt;




</description>
      <category>dagster</category>
      <category>prefect</category>
      <category>dataintegration</category>
      <category>dataorchestration</category>
    </item>
    <item>
      <title>CDC Is the Plumbing Everyone Forgets Until It Breaks</title>
      <dc:creator>Andrew Tan</dc:creator>
      <pubDate>Mon, 03 Aug 2026 14:20:52 +0000</pubDate>
      <link>https://dev.to/andrew_tan_layline/cdc-is-the-plumbing-everyone-forgets-until-it-breaks-2i08</link>
      <guid>https://dev.to/andrew_tan_layline/cdc-is-the-plumbing-everyone-forgets-until-it-breaks-2i08</guid>
      <description>&lt;p&gt;Change Data Capture is the invisible layer enabling real-time analytics and event-driven systems — but most teams only think about it after their first production incident&lt;/p&gt;

&lt;h2&gt;
  
  
  The Invisible Layer That Everything Depends On
&lt;/h2&gt;

&lt;p&gt;Real-time dashboards. Event-driven microservices. Data lakes that stay current. Behind every one of these modern data architectures sits a component that most teams don't think much about: &lt;strong&gt;Change Data Capture&lt;/strong&gt;.&lt;/p&gt;

&lt;p&gt;CDC's job is simple enough — watch database transaction logs and emit events whenever data changes. New order? Event. Status update? Event. Customer deletion? Event. The concept is elegant, and when it works, it just works.&lt;/p&gt;

&lt;p&gt;But there's a problem. CDC is the plumbing of modern data infrastructure: invisible when it functions, catastrophic when it fails, and somehow always an afterthought in architecture reviews. Teams spend weeks debating Kafka topologies and Spark configurations, then slap in a CDC connector with default settings and move on.&lt;/p&gt;

&lt;p&gt;Six months later, the call comes. The dashboard is six hours behind. The inventory sync is showing yesterday's data. The CEO is asking why customers can buy products that don't exist. And nobody can figure out why — because the CDC connector is "healthy" according to the monitoring dashboard.&lt;/p&gt;

&lt;p&gt;This pattern plays out across the industry with remarkable consistency. The issue isn't that CDC is fundamentally unreliable. It's that the gap between what teams assume it does and what it actually does is wide enough to hide production incidents until they become business problems.&lt;/p&gt;

&lt;p&gt;&lt;a href="https://media2.dev.to/dynamic/image/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto%2Cformat=auto/https%3A%2F%2Fdev-to-uploads.s3.us-east-2.amazonaws.com%2Fuploads%2Farticles%2Fyj2jtbwm8hzre299qso9.jpg" class="article-body-image-wrapper"&gt;&lt;img src="https://media2.dev.to/dynamic/image/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto%2Cformat=auto/https%3A%2F%2Fdev-to-uploads.s3.us-east-2.amazonaws.com%2Fuploads%2Farticles%2Fyj2jtbwm8hzre299qso9.jpg" width="800" height="533"&gt;&lt;/a&gt;&lt;/p&gt;

&lt;h2&gt;
  
  
  What CDC Actually Does (And What Teams Assume It Does)
&lt;/h2&gt;

&lt;p&gt;At its core, Change Data Capture watches your database transaction log and emits events whenever data changes. Insert a row? Event. Update a field? Event. Delete a record? Event. The concept is beautifully simple.&lt;/p&gt;

&lt;p&gt;But the simplicity is deceptive. Here's what CDC &lt;strong&gt;actually&lt;/strong&gt; captures versus what teams &lt;strong&gt;assume&lt;/strong&gt; it captures:&lt;/p&gt;

&lt;div class="table-wrapper-paragraph"&gt;&lt;table&gt;
&lt;thead&gt;
&lt;tr&gt;
&lt;th&gt;What teams assume&lt;/th&gt;
&lt;th&gt;What actually happens&lt;/th&gt;
&lt;/tr&gt;
&lt;/thead&gt;
&lt;tbody&gt;
&lt;tr&gt;
&lt;td&gt;"Every change is captured immediately"&lt;/td&gt;
&lt;td&gt;There's latency. Sometimes milliseconds, sometimes seconds, sometimes longer if the connector is backlogged.&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;"The events are in the same order as the transactions"&lt;/td&gt;
&lt;td&gt;Not necessarily. Parallel replication, commit ordering, and eventual consistency can scramble sequences.&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;"Schema changes are handled gracefully"&lt;/td&gt;
&lt;td&gt;Adding a column? Fine. Renaming one? Dropping one? Changing a type? Your CDC pipeline may need manual intervention.&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;"It's just a log tail, what could go wrong?"&lt;/td&gt;
&lt;td&gt;Connector crashes, replication slot exhaustion, disk space issues on the source DB, network partitions...&lt;/td&gt;
&lt;/tr&gt;
&lt;/tbody&gt;
&lt;/table&gt;&lt;/div&gt;

&lt;p&gt;The gap between assumption and reality is where incidents breed.&lt;/p&gt;

&lt;h2&gt;
  
  
  The Three Failure Modes Nobody Talks About
&lt;/h2&gt;

&lt;p&gt;After watching a dozen CDC implementations go sideways, I've noticed three failure patterns that don't get enough attention in the tutorials and vendor demos.&lt;/p&gt;

&lt;h3&gt;
  
  
  1. The Schema Drift Trap
&lt;/h3&gt;

&lt;p&gt;Your application team adds a new column to the &lt;code&gt;orders&lt;/code&gt; table. It's a harmless change — a nullable &lt;code&gt;delivery_notes&lt;/code&gt; field. They deploy on Tuesday. By Thursday, your data warehouse has incomplete records because the CDC connector is still using the old schema and silently dropping the new field.&lt;/p&gt;

&lt;p&gt;The worst part? The connector doesn't fail. It just produces events that are &lt;em&gt;technically&lt;/em&gt; valid but &lt;em&gt;practically&lt;/em&gt; wrong. Your data quality monitors don't catch it because the schema validator thinks everything is fine. You only discover the gap when someone asks why the delivery notes report is blank for half the week.&lt;/p&gt;

&lt;h3&gt;
  
  
  2. The Replication Slot Bomb
&lt;/h3&gt;

&lt;p&gt;PostgreSQL users, this one's for you. CDC connectors use "replication slots" to track which WAL (Write-Ahead Log) entries they've processed. If your connector goes down — or even just slows down significantly — those slots hold onto log entries. The database can't reclaim that disk space.&lt;/p&gt;

&lt;p&gt;I've seen teams wake up to production databases at 95% disk capacity because a flaky CDC connector was holding replication slots hostage. The fix is a manual cleanup job that feels terrifying to run at 2 AM. The prevention? Monitoring and alerting that most teams don't set up until after the first incident.&lt;/p&gt;

&lt;h3&gt;
  
  
  3. The Consumer Coupling Problem
&lt;/h3&gt;

&lt;p&gt;CDC emits a firehose of events. Every microservice, analytics job, and data warehouse sync that cares about database changes taps into that stream. It's elegant and decoupled — until it isn't.&lt;/p&gt;

&lt;p&gt;What happens when one slow consumer can't keep up? Backpressure propagates. The CDC connector buffers, then drops, then crashes. Or worse: it keeps running but falls behind, and your "real-time" pipeline has a 20-minute lag that nobody notices because the metrics dashboard shows "connector healthy."&lt;/p&gt;

&lt;p&gt;The fix is usually some form of buffering (Kafka, Kinesis, a message queue) between the CDC source and the consumers. But now you've added latency and another piece of infrastructure to manage. The simple plumbing has become a complex subsystem.&lt;/p&gt;

&lt;h2&gt;
  
  
  Sizing for Reality, Not for Hope
&lt;/h2&gt;

&lt;p&gt;Here's a fictional conversation:&lt;/p&gt;

&lt;blockquote&gt;
&lt;p&gt;&lt;strong&gt;Me:&lt;/strong&gt; "How many transactions per second does your CDC need to handle?"&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;Them:&lt;/strong&gt; "Oh, maybe a few hundred during peak."&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;Me:&lt;/strong&gt; "And what's your biggest table?"&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;Them:&lt;/strong&gt; "About fifty million rows."&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;Me:&lt;/strong&gt; "What happens when you run a bulk update on that table?"&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;Them:&lt;/strong&gt; "...We do those sometimes."&lt;/p&gt;
&lt;/blockquote&gt;

&lt;p&gt;CDC connectors aren't sized for your average transaction volume. They're sized for your &lt;strong&gt;worst-case&lt;/strong&gt; transaction volume. That quarterly data cleanup job that touches ten million rows? That generates ten million CDC events in a burst. If your connector can't handle the spike, you get lag, backpressure, or dropped events.&lt;/p&gt;

&lt;p&gt;The teams that do this well plan for bursts from day one. They set up monitoring on replication lag, not just connector health. They test their failure modes: what happens if the connector restarts mid-bulk-update? What happens if the destination is down for an hour?&lt;/p&gt;

&lt;h2&gt;
  
  
  Design Decisions That Make CDC Manageable
&lt;/h2&gt;

&lt;p&gt;CDC doesn't have to be a ticking time bomb. Here are the patterns I've seen work in production:&lt;/p&gt;

&lt;h3&gt;
  
  
  Separate CDC Infrastructure from Analytics Infrastructure
&lt;/h3&gt;

&lt;p&gt;Don't run your CDC connector on the same cluster as your Spark jobs or your BI queries. When the analytics team runs a heavy join that saturates the network, your CDC events shouldn't suffer. Give CDC its own lane.&lt;/p&gt;

&lt;h3&gt;
  
  
  Idempotent Consumers Are Non-Negotiable
&lt;/h3&gt;

&lt;p&gt;CDC events can be duplicated. Connectors restart, network partitions happen, at-least-once delivery is the default. If your downstream consumer can't handle "process this order update twice," you're going to have data corruption. Build idempotency in from the start.&lt;/p&gt;

&lt;h3&gt;
  
  
  Schema Registries Save Sanity
&lt;/h3&gt;

&lt;p&gt;Use a schema registry (Confluent Schema Registry, AWS Glue, or similar) to track changes to your event schemas. When the application team changes a table, the schema change flows through the registry and your consumers can adapt programmatically instead of breaking silently.&lt;/p&gt;

&lt;h3&gt;
  
  
  Monitor What Matters
&lt;/h3&gt;

&lt;p&gt;"Connector is running" is the wrong metric. Monitor:&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;
&lt;strong&gt;Replication lag&lt;/strong&gt; (how far behind is the CDC from the database?)&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Event processing rate&lt;/strong&gt; (are we keeping up with production?)&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Schema change events&lt;/strong&gt; (did something change in the source we need to know about?)&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Dead letter queue depth&lt;/strong&gt; (what couldn't be processed and why?)&lt;/li&gt;
&lt;/ul&gt;

&lt;h2&gt;
  
  
  Where layline.io Fits: CDC Without the Footguns
&lt;/h2&gt;

&lt;p&gt;At &lt;strong&gt;layline.io&lt;/strong&gt;, we've watched teams struggle with CDC enough that we built a dedicated &lt;a href="https://layline.io/solutions/etl-elt" rel="noopener noreferrer"&gt;Debezium Source Asset&lt;/a&gt; directly into the platform. The goal isn't to reinvent CDC — Debezium is excellent — but to wrap it in the reliability and observability that production systems need.&lt;/p&gt;

&lt;p&gt;Instead of running a standalone connector that you have to babysit, layline.io gives you:&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;Visual pipeline design&lt;/strong&gt; that includes CDC sources as first-class citizens. You see the data flow from database to destination on a single canvas. When something breaks, you know exactly where.&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;Built-in backpressure handling&lt;/strong&gt; through Apache Pekko's actor-model streaming. When downstream systems slow down, layline.io throttles gracefully instead of dropping events or crashing connectors.&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;Unified retry and error handling&lt;/strong&gt; across the entire pipeline. CDC events that fail to process don't vanish into a log file — they go through the same retry mechanisms as every other data source.&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;Schema-aware transformation&lt;/strong&gt; that can adapt to changes in the source database without manual intervention. Add a column, rename a field, change a type — the pipeline adjusts instead of breaking.&lt;/p&gt;

&lt;p&gt;The broader point: CDC is too important to be an afterthought. It deserves the same engineering rigor as the rest of your data infrastructure. Whether you use layline.io or build your own stack, treat CDC like the critical component it is — not like plumbing you can ignore until the basement floods.&lt;/p&gt;

</description>
      <category>changedatacapture</category>
      <category>realtimedata</category>
      <category>dataengineering</category>
      <category>databasereplication</category>
    </item>
    <item>
      <title>Your Data Warehouse Is Not Your Data Pipeline</title>
      <dc:creator>Andrew Tan</dc:creator>
      <pubDate>Fri, 31 Jul 2026 11:14:14 +0000</pubDate>
      <link>https://dev.to/andrew_tan_layline/your-data-warehouse-is-not-your-data-pipeline-4n6</link>
      <guid>https://dev.to/andrew_tan_layline/your-data-warehouse-is-not-your-data-pipeline-4n6</guid>
      <description>&lt;p&gt;Teams keep forcing their warehouse to do integration work it was never designed for. The result is ballooning costs, opaque failures, and architectures that become harder to maintain the more they ‘succeed.’ Here’s the case for separating data movement from analytics storage.&lt;/p&gt;

&lt;h2&gt;
  
  
  The expensive truth about modern data stacks
&lt;/h2&gt;

&lt;p&gt;Spend enough time around data platform teams and you hear the same story. A company builds out its "modern data stack" — warehouse, processing layer, orchestrator — and everything looks clean on the architecture diagram. Then the warehouse bill starts to climb. Ingestion jobs fail more often than anyone expected. And every time something breaks, it takes half a day to figure out whether the problem is in the load, the reshape, the orchestrator, or the warehouse itself.&lt;/p&gt;

&lt;p&gt;At some point, someone on the team says the quiet part out loud: "I think we built a really expensive integration tool by accident."&lt;/p&gt;

&lt;p&gt;They are usually right.&lt;/p&gt;




&lt;h2&gt;
  
  
  The category error
&lt;/h2&gt;

&lt;p&gt;A data warehouse is a query and storage engine. It is optimized for one thing: answering analytical questions fast over large datasets.&lt;/p&gt;

&lt;p&gt;A data pipeline is a movement and processing runtime. It is optimized for something different: getting data from where it is to where it needs to be, in the right shape, at the right time, reliably.&lt;/p&gt;

&lt;p&gt;Those are different jobs. But in the last decade, we've quietly asked the warehouse to do both.&lt;/p&gt;

&lt;p&gt;It started innocently. Warehouses got better at loading data. Then they got stored procedures. Then dbt turned SQL into a processing layer. Then orchestrators started triggering warehouse queries to move data between tables. And before anyone named it, the warehouse had become the default integration layer.&lt;/p&gt;

&lt;p&gt;The result is predictable. The warehouse is excellent at analytics. It is mediocre at integration. And when you force it to do integration at scale, you pay for it in three currencies: cost, reliability, and architectural fragility.&lt;/p&gt;




&lt;h2&gt;
  
  
  What goes wrong when the warehouse becomes the pipeline
&lt;/h2&gt;

&lt;h3&gt;
  
  
  The compute bill becomes a surprise
&lt;/h3&gt;

&lt;p&gt;Warehouse compute is priced for analytical queries. Analysts run a few big queries, wait for results, and go make decisions. The compute is bursty and human-paced.&lt;/p&gt;

&lt;p&gt;Integration workloads don't look like that. They run continuously or on tight schedules. They move millions of rows. They run the same conversions over and over. They don't pause to let humans read dashboards.&lt;/p&gt;

&lt;p&gt;When you run this kind of workload inside a warehouse, the meter spins differently. It is common for a "simple" hourly sync to consume more credits than the entire analytics workload. Not because the warehouse is bad, but because it's the wrong engine for the job.&lt;/p&gt;

&lt;h3&gt;
  
  
  Failures become opaque
&lt;/h3&gt;

&lt;p&gt;A pipeline has a clear job: take data from A, transform it, deliver it to B. When it fails, you want to know which step failed and why.&lt;/p&gt;

&lt;p&gt;When the warehouse is the pipeline, failure is distributed across layers. Was the load slow because the warehouse was overloaded? Did the orchestrator lose its connection? Did the reshape query hit a timeout? Is the data wrong because of the source, the conversion, or a change to the warehouse execution plan?&lt;/p&gt;

&lt;p&gt;Debugging becomes archaeology. You dig through query history, orchestrator logs, and warehouse metrics, trying to reconstruct what actually happened. The tools are all there. The clarity isn't.&lt;/p&gt;

&lt;h3&gt;
  
  
  Latency is whatever the warehouse decides
&lt;/h3&gt;

&lt;p&gt;If your pipeline is a series of warehouse queries, your latency is bounded by warehouse scheduling. A query waits in a queue. It compiles. It runs. Maybe it gets preempted. Maybe it scales up. Maybe it doesn't.&lt;/p&gt;

&lt;p&gt;For batch analytics, this is fine. No one cares if a nightly report finishes at 3 AM or 3:15 AM.&lt;/p&gt;

&lt;p&gt;For operational use cases, it's not fine. Fraud detection, inventory updates, customer-facing dashboards — these need minutes or seconds, not warehouse-queue time. When the warehouse is your pipeline, you inherit its pace. And its pace is designed for analysts, not operations.&lt;/p&gt;

&lt;h3&gt;
  
  
  Lock-in deepens
&lt;/h3&gt;

&lt;p&gt;The more integration logic lives inside the warehouse, the harder it becomes to leave. Your rewrites are in warehouse-specific SQL dialects. Your orchestration is tied to warehouse sessions. Your data quality rules run as warehouse queries. Even your cost visibility is warehouse-shaped.&lt;/p&gt;

&lt;p&gt;This isn't a conspiracy. It's just what happens when one tool becomes responsible for too many jobs. The migration cost grows until it feels easier to stay unhappy than to leave.&lt;/p&gt;




&lt;h2&gt;
  
  
  What clean separation looks like
&lt;/h2&gt;

&lt;p&gt;The fix isn't to throw out the warehouse. The warehouse is good at what it does. The fix is to let it do what it does and stop asking it to do everything else.&lt;/p&gt;

&lt;p&gt;In practice, that usually means two platforms, not one:&lt;/p&gt;

&lt;h3&gt;
  
  
  Integration and orchestration runtime
&lt;/h3&gt;

&lt;p&gt;This is where data moves, gets reshaped, gets validated, and gets routed to the right consumers. It also schedules pipelines, retries failures, enforces dependencies, and triggers downstream work — both inside the platform and in external systems. It runs on an engine designed for continuous data flow, not query latency.&lt;/p&gt;

&lt;h3&gt;
  
  
  Warehouse
&lt;/h3&gt;

&lt;p&gt;This is where data is stored and queried. It receives clean, ready-to-query data from the integration layer. It doesn't worry about how the data got there, when the next load arrives, or what to do if a job fails. It just answers questions.&lt;/p&gt;

&lt;p&gt;Logically, you can still think of integration and orchestration as separate concerns. Operationally, they often belong in the same runtime. A pipeline that can move data but can't schedule itself, retry itself, or trigger the next step is only half useful. The best platforms combine both.&lt;/p&gt;

&lt;p&gt;When these concerns are separated from the warehouse, each tool gets simpler. The integration layer is optimized for throughput and reliability. The orchestrator is optimized for dependency management and failure recovery. The warehouse is optimized for query performance.&lt;/p&gt;

&lt;p&gt;Most importantly, problems stay in their lane. When ingestion fails, you look at the integration runtime. When a report is wrong, you look at the warehouse. When a job doesn't run, you look at the orchestrator — which, in a clean setup, is part of the same runtime that moves the data.&lt;/p&gt;

&lt;p&gt;&lt;a href="https://media2.dev.to/dynamic/image/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto%2Cformat=auto/https%3A%2F%2Fdev-to-uploads.s3.us-east-2.amazonaws.com%2Fuploads%2Farticles%2F6v0esvijnzbdvk0isz1l.jpg" class="article-body-image-wrapper"&gt;&lt;img src="https://media2.dev.to/dynamic/image/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto%2Cformat=auto/https%3A%2F%2Fdev-to-uploads.s3.us-east-2.amazonaws.com%2Fuploads%2Farticles%2F6v0esvijnzbdvk0isz1l.jpg" width="800" height="420"&gt;&lt;/a&gt;&lt;/p&gt;




&lt;h2&gt;
  
  
  When warehouse-as-pipeline is actually fine
&lt;/h2&gt;

&lt;p&gt;I don't want to overstate this. For some teams, the warehouse-as-pipeline pattern works fine.&lt;/p&gt;

&lt;p&gt;If you're small, your data volumes are low, your reshaping is simple, and your latency requirements are "tomorrow is fine," then keeping everything in one place is a reasonable tradeoff. The operational simplicity is worth more than the architectural purity.&lt;/p&gt;

&lt;p&gt;The problems start when the pattern keeps scaling past its natural limit. A team that outgrows it usually knows. The bills get weird. The failures get mysterious. The idea of adding a real-time use case becomes a multi-month project instead of a configuration change.&lt;/p&gt;

&lt;p&gt;The question isn't whether the pattern is bad. The question is whether it's still the right pattern for where you are now.&lt;/p&gt;




&lt;h2&gt;
  
  
  The migration path nobody takes
&lt;/h2&gt;

&lt;p&gt;Most teams imagine this separation as a rip-and-replace project. It doesn't have to be.&lt;/p&gt;

&lt;p&gt;The better approach is to extract the movement layer first. Pick one data source. Instead of loading it directly into the warehouse and then reshaping it there, move it through a dedicated integration runtime first. Clean it. Validate it. Then write the clean data to the warehouse.&lt;/p&gt;

&lt;p&gt;The warehouse doesn't change much. The analysts keep querying the same tables. But now those tables are fed by a pipeline that is designed for feeding tables.&lt;/p&gt;

&lt;p&gt;Once one source is moved, the pattern repeats. Source by source. Pipeline by pipeline. Over time, the warehouse stops being the integration hub and becomes what it was meant to be: the analytics hub.&lt;/p&gt;

&lt;p&gt;Teams that do this successfully don't start with the hardest pipeline. They start with a boring one. The boring pipelines teach you the pattern without the risk. The hard pipelines get easier once the pattern is in place.&lt;/p&gt;




&lt;h2&gt;
  
  
  Where layline.io fits
&lt;/h2&gt;

&lt;p&gt;I'll be direct: this is the architectural bet behind layline.io.&lt;/p&gt;

&lt;p&gt;We built a data processing platform that handles the integration and orchestration layer — both batch and streaming — without making the warehouse do the heavy lifting. Pipelines move data, reshape it, validate it, and deliver it. They also schedule themselves, retry on failure, enforce dependencies, and trigger downstream workflows inside layline or in external systems.&lt;/p&gt;

&lt;p&gt;The warehouse stores the data and queries it. Each tool does its own job.&lt;/p&gt;

&lt;p&gt;Because layline handles both batch and streaming in the same runtime, you don't end up with one tool for your hourly loads and another tool for your real-time events. Same workflows. Same observability. Same team. And because orchestration is built in, you don't need a separate orchestrator sitting on top, coordinating between layline and everything else.&lt;/p&gt;

&lt;p&gt;That's not a pitch for everyone. If your warehouse-as-pipeline setup is working and your bills are sane, you don't need us. But if you're staring at a tripled warehouse bill and wondering how a "simple" sync got so expensive, the separation we're describing is probably what you're actually looking for.&lt;/p&gt;




&lt;h2&gt;
  
  
  The question to ask your team
&lt;/h2&gt;

&lt;p&gt;Pick your three most expensive warehouse workloads. Not the biggest analytical queries — the ones that run all day, moving and reshaping data.&lt;/p&gt;

&lt;p&gt;Ask: are these workloads answering business questions, or are they just getting data into a shape where it can answer business questions?&lt;/p&gt;

&lt;p&gt;If the answer is the second one, you've got integration work running in an analytics engine. That's not a moral failing. It's a very common architecture. But it's also a very fixable one.&lt;/p&gt;

&lt;p&gt;The warehouse is a powerful tool. It just isn't the only tool.&lt;/p&gt;

</description>
      <category>datapipeline</category>
      <category>datawarehouse</category>
      <category>dataintegration</category>
      <category>dataorchestration</category>
    </item>
    <item>
      <title>The AI Productivity Gap: Why the Numbers Don't Add Up</title>
      <dc:creator>Andrew Tan</dc:creator>
      <pubDate>Mon, 06 Jul 2026 13:00:31 +0000</pubDate>
      <link>https://dev.to/andrew_tan_layline/the-ai-productivity-gap-why-the-numbers-dont-add-up-2e0g</link>
      <guid>https://dev.to/andrew_tan_layline/the-ai-productivity-gap-why-the-numbers-dont-add-up-2e0g</guid>
      <description>&lt;p&gt;&lt;em&gt;Every enterprise dashboard claims AI is transforming the business. The actual productivity numbers tell a very different story — and understanding why matters for every team making AI investment decisions.&lt;/em&gt;&lt;/p&gt;




&lt;p&gt;There's a gap between the story being told about AI in the enterprise and what companies are actually experiencing on the ground. You could watch this play out across industries for a while now, and the pattern is consistent enough that it's worth naming directly.&lt;/p&gt;

&lt;p&gt;The pitch is familiar: AI tools will automate the repetitive work, amplify your team's output, and ultimately let you do more with less. The reality, for most organizations, looks quite different. The executives I speak with are largely describing the same experience — AI projects that showed early promise in demos and pilots, then ran into friction when exposed to the noise of real production environments.&lt;/p&gt;

&lt;p&gt;This isn't an argument against AI adoption. It's an argument for being precise about where AI actually delivers value versus where it adds cost and complexity without a corresponding return.&lt;/p&gt;




&lt;h2&gt;
  
  
  The deployment failure pattern
&lt;/h2&gt;

&lt;p&gt;The first thing that gets lost in AI coverage is how often production deployments fail quietly.&lt;/p&gt;

&lt;p&gt;Announcements of AI initiatives tend to generate press. The quiet rollbacks that follow tend not to. But when you talk to operations teams candidly, the reversal pattern is common — systems that worked in controlled testing, connected to clean data and well-defined inputs, that degraded when exposed to the variability of real customers, real data, and real edge cases.&lt;/p&gt;

&lt;p&gt;Customer-facing AI deployments have been particularly prone to this. The tolerance for errors in customer interactions is low, and the compounding effect of getting things wrong repeatedly erodes trust faster than any initial efficiency gain can offset. Teams that replaced human capacity with AI and then had to reverse course found themselves spending months rebuilding, often with more urgency than before.&lt;/p&gt;

&lt;p&gt;The lesson isn't that AI customer interaction tools don't work — it's that the failure modes are underestimated during the planning phase, and the cost of a failed rollout exceeds the projected savings even when the initial deployment looked promising.&lt;/p&gt;




&lt;h2&gt;
  
  
  The accuracy ceiling
&lt;/h2&gt;

&lt;p&gt;Why do production deployments fail at rates that don't match pre-deployment expectations? The answer is largely in how AI capability is measured versus how it needs to perform.&lt;/p&gt;

&lt;p&gt;Benchmarks and vendor demos select for conditions where AI performs best. Production environments don't. The gap between benchmark accuracy and real-world accuracy is consistently larger than teams expect, particularly for anything involving ambiguous inputs, unusual edge cases, or tasks requiring contextual judgment.&lt;/p&gt;

&lt;p&gt;In software development — which has been the proving ground for AI productivity claims — the productivity story is more nuanced than the marketing suggests. AI tools are genuinely useful for certain well-scoped tasks: generating boilerplate, explaining unfamiliar code, drafting documentation. But the secondary costs of AI-assisted development are underweighted: code review cycles get longer when you can't assume the same level of reliability you'd expect from an experienced engineer, security review becomes more necessary, and debugging AI-introduced errors can consume more time than writing equivalent code from scratch.&lt;/p&gt;

&lt;p&gt;The net productivity effect, in practice, is much closer to neutral than the adoption narrative suggests. The teams I've seen extract real value from AI coding tools have been disciplined about scope — using AI in a narrow, well-supervised lane and keeping human judgment in the loop for anything that matters.&lt;/p&gt;

&lt;p&gt;There's also a question of whether reliability improves sufficiently with more capable models. The structural challenge is that AI systems are fundamentally probabilistic — they approximate, they extrapolate, and their confidence doesn't reliably track their accuracy. Newer models are better, but the same category of failures persists. The question isn't whether AI will ever be reliable enough, it's whether the current generation is reliable enough for the specific task you're considering, and that requires honest evaluation rather than optimistic extrapolation.&lt;/p&gt;




&lt;h2&gt;
  
  
  The real cost equation
&lt;/h2&gt;

&lt;p&gt;Even setting aside the reliability question, the economics of AI deployment have shifted in ways that deserve scrutiny.&lt;/p&gt;

&lt;p&gt;When AI tools first entered the enterprise, pricing was structured to drive adoption — flat subscriptions that made ROI calculations appear straightforward. Many of those pricing models were, in retrospect, being offered well below the actual cost of providing the service. As the market has matured and providers have moved toward pricing that reflects real operational costs, the economics look quite different from the projections that justified many initial investments.&lt;/p&gt;

&lt;p&gt;The teams that made commitments based on early pricing are now navigating a different cost environment. Usage-based pricing models mean that scaling up AI adoption increases costs non-linearly. The math that justified a pilot may not survive contact with production usage volumes.&lt;/p&gt;

&lt;p&gt;There's also the indirect cost of integration overhead, maintenance, and the ongoing work of keeping AI systems calibrated as underlying models and APIs change. These costs are consistently underestimated in project planning and rarely appear in the productivity gain calculations that AI vendors highlight.&lt;/p&gt;

&lt;p&gt;The honest ROI calculation for AI adoption needs to include the full cost picture: inference at realistic usage levels, integration and maintenance overhead, the cost of failures and rollbacks, and the opportunity cost of the engineering time spent managing AI systems rather than building product.&lt;/p&gt;




&lt;h2&gt;
  
  
  What this means for data infrastructure
&lt;/h2&gt;

&lt;p&gt;The AI productivity story has a specific texture in this space worth unpacking.&lt;/p&gt;

&lt;p&gt;The appeal of AI for data workflows is real: generating transformation logic, scaffolding pipeline boilerplate, navigating unfamiliar APIs. If AI could reliably handle these tasks, the productivity gains would be meaningful. The challenge is that data pipelines have near-zero tolerance for silent errors. A transformation that produces plausible-but-wrong output isn't just a bug — it's a corruption that propagates downstream before anyone notices.&lt;/p&gt;

&lt;p&gt;The teams that handle this well use AI as a first-draft accelerator for well-defined, reviewable tasks, with automated validation and human review before anything touches production. That's a meaningfully different model from "AI replaces the engineer" — it's more like a junior colleague who needs supervision. That framing leads to better outcomes than treating AI as a reliable autonomous agent.&lt;/p&gt;

&lt;p&gt;&lt;a href="https://media2.dev.to/dynamic/image/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto%2Cformat=auto/https%3A%2F%2Fdev-to-uploads.s3.us-east-2.amazonaws.com%2Fuploads%2Farticles%2Fbp01iy8dyay5yf8cr3ea.jpg" class="article-body-image-wrapper"&gt;&lt;img src="https://media2.dev.to/dynamic/image/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto%2Cformat=auto/https%3A%2F%2Fdev-to-uploads.s3.us-east-2.amazonaws.com%2Fuploads%2Farticles%2Fbp01iy8dyay5yf8cr3ea.jpg" alt="Data engineer reviewing pipeline workflow on dual monitors with AI code assistant panel open" width="800" height="533"&gt;&lt;/a&gt;&lt;/p&gt;

&lt;p&gt;What doesn't work is using AI in the parts of data engineering where precision is non-negotiable and errors are hard to detect — schema transformations, data quality rules, anything that feeds downstream analytics that people make decisions with. The productivity gains in that zone tend to be negative once you account for the debugging and remediation work.&lt;/p&gt;




&lt;h2&gt;
  
  
  Calibrating the expectation
&lt;/h2&gt;

&lt;p&gt;At layline.io, we've watched our customers navigate these trade-offs, and the pattern among teams that do it well is consistent: they're systematic about where AI helps and where it doesn't, they insist on validation at every stage, and they treat AI output the same way they treat any external input — with appropriate skepticism until it's been verified.&lt;/p&gt;

&lt;p&gt;The AI productivity gap isn't closing on its own. The teams that navigate it well are the ones being precise about where AI genuinely adds value — and staying disciplined about everything else.&lt;/p&gt;

&lt;p&gt;A few questions that have proven useful before any AI deployment in data workflows:&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;What does a failure look like, and how quickly would we detect it?&lt;/strong&gt; Silent errors in pipelines are categorically more dangerous than visible failures. If the answer to "how would we detect it?" is "we'd notice when the numbers look off," that's not a detection mechanism.&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;What's the full cost at production scale?&lt;/strong&gt; Usage-based pricing means the economics at pilot scale don't predict the economics at full deployment. Model it before you commit.&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;What's the rollback path?&lt;/strong&gt; Given how often AI deployments require reversal, any adoption that doesn't include a tested rollback path is taking on more risk than the productivity potential justifies.&lt;/p&gt;

&lt;p&gt;The upside of AI in data infrastructure is real. So is the downside of getting it wrong. The teams that capture the upside are the ones who go in with clear eyes about both.&lt;/p&gt;




&lt;p&gt;&lt;em&gt;Building data infrastructure where reliability isn't optional? &lt;a href="https://layline.io/product" rel="noopener noreferrer"&gt;Take a look at layline.io&lt;/a&gt; — the &lt;a href="https://layline.io/get-started" rel="noopener noreferrer"&gt;Community Edition is free&lt;/a&gt; to explore.&lt;/em&gt;&lt;/p&gt;

</description>
    </item>
  </channel>
</rss>
