DEV Community: Lucas Ehara

The Real-Time Fetish: Why You (Probably) Don't Need Streaming

Lucas Ehara — Thu, 06 Aug 2026 18:23:02 +0000

In modern Data Engineering, there is an unspoken fetish for "Real-Time." If you ask any business stakeholder how fast they need their dashboard to update, the default answer will always be: "As fast as possible."

This drives well-intentioned engineers to design incredibly complex architectures. We spin up Kafka clusters, implement Flink, and wrestle with latency, late-arriving data, and tumbling windows. All to have data flowing in milliseconds. But the harsh reality is that the vast majority of companies are building Ferraris just to sit in rush-hour traffic.

1. The Actionability Gap (The Golden Question)

The biggest mistake when choosing a streaming architecture isn't technical; it's a business mistake. Before implementing real-time pipelines, the only question that matters is:

"Does the company have the operational capacity to make a decision in milliseconds?"

If you are building a credit card fraud detection system or a live e-commerce recommendation engine, yes, every millisecond counts. But if the data is feeding a financial dashboard that the executive board only reviews during their Monday morning meeting, updating that screen every second is a colossal waste of money and effort. Real-time data has zero value if the human action is batch.

2. The Hidden Complexity and the Cloud Bill

Batch processing is forgiving. If a pipeline fails at 3 AM, you trigger a rerun, and by 8 AM, everything is fine. Batch is cheap, predictable, and easy to debug.

Streaming, on the other hand, is unforgiving. Handling application state, event duplication (exactly-once semantics), out-of-order events, and sudden traffic spikes requires a senior engineering team dedicated solely to keeping the infrastructure alive. Furthermore, the cloud bill for 24/7 continuous processing is orders of magnitude higher than spinning up your compute clusters on a schedule.

3. "Micro-Batch" Solves 99% of Your Problems

There is a perfect middle ground that the hype industry tries to ignore: the micro-batch.

Running your data transformation pipelines every hour or even every 15 minutes delivers the "real-time" feeling to the end-user while maintaining the simplicity, reliability, and low cost of batch processing.

Conclusion: Scalable Boredom

Good engineering isn't about using the most complex technology available; it's about solving the business problem with the simplest possible solution.

Embracing batch processing doesn't make you a dinosaur. It makes you a mature professional who protects the company's budget and your own team's sanity. When in doubt, start with D-1 (previous day's data). When the business can mathematically prove that data latency is costing them real money, then you build your streaming architecture.

Stop Throwing Spark at Everything: The Case for Tech Agnostic Data Engineering

Lucas Ehara — Mon, 13 Jul 2026 12:44:41 +0000

Recently, the industry has developed a reflex. When faced with a new pipeline request, the immediate answer is often to spin up Databricks or write a complex PySpark job. It’s the shiny, distributed computing hammer that makes us feel like we’re doing "real" big data.

If you are processing terabytes of streaming data or training heavy machine learning models, Spark is a lifesaver. The problem starts when this distributed reflex replaces actual architectural pragmatism, and we start using sledgehammers to hang picture frames.

1. The Complexity Tax (The Real Cost of Overkill)

Distributed computing introduces distributed problems. Modern cloud data warehouses are incredibly powerful and often more than capable of handling what many companies consider "big data."

When you force a heavy tool onto a moderate dataset, you spend more time managing cluster configurations, overhead, and driver memory than actually delivering data. Often, a well designed SQL pipeline can handle the workload perfectly. Sometimes, solving a severe table lock issue on a massive dataset doesn't require a brand new execution it just requires stepping back and implementing a smart, sequential batch update strategy. If your first instinct is to scale the tech instead of fixing the approach, you aren't engineering; you're just paying a complexity tax.

2. The Trap of Vendor Driven Logic

A data pipeline is a mechanism to deliver business value, not an excuse to showcase a framework. When you tightly couple your entire mental model and transformation logic to the specific nuances of one platform, you lose flexibility.

Being tech agnostic means understanding the fundamental patterns: idempotency, robust data modeling, and clear, modular transformations. If you build your logic agnostically, you own the architecture. If tomorrow the business needs to optimize costs and shift workloads away from a heavy Spark cluster back to native warehouse processing, a tech agnostic foundation makes that migration a planned engineering task, not a complete rewrite.

Conclusion: Engineering Requires Pragmatism

Save the heavy artillery for the heavy problems. When you are responsible for a company's data architecture, your loyalty must be to the solution, not the stack.

Regardless of the vendors in the market, the act of understanding the true data volume and applying the simplest, most effective tool isn't backward. It's elegant engineering.

You can love a specific technology. But to build sustainable data platforms, you must remain agnostic to it.

Why "Vibe Coding" Your Data Warehouse is a Terrible Idea

Lucas Ehara — Sat, 11 Apr 2026 19:16:15 +0000

Recently, the term "vibe coding" has been everywhere. It’s that practice of sitting down, opening your editor (often with an AI assistant), and just writing code in the flow state. You test, iterate, and build without a massive upfront blueprint.

If you are spinning up a landing page or writing a quick Python automation script, vibe coding is liberating. The problem starts when this culture tries to infiltrate Data Engineering, specifically, Data Warehouse modeling.

The Trap of "Just Writing SQL"

Modern cloud data warehouses like AWS Redshift make it incredibly trivial to transform data on the fly. You get a request, you write a massive CREATE TABLE AS SELECT (CTAS) or a Materialized View with some intuitive joins, and push it to production. The stakeholder is happy today. The vibe is great.

But the reality of data engineering catches up to you, and it hits hardest in two specific areas:

1. The Business Disconnect (The Real Cost)

A Data Warehouse is not an IT project; it is the mathematical and historical representation of a company's business processes.

When you vibe code, you skip the whiteboard. You make technical assumptions about business concepts. You assume a user_id means the same thing across all schemas. But without talking to stakeholders and designing a unified conceptual model, you end up creating isolated data silos.

Suddenly, the Marketing dashboard shows 5,000 "active clients," but the Finance report shows 4,200. You spend the next three weeks in endless alignment meetings trying to debug why the numbers don't match, only to realize the core logic was fundamentally different. Once the business users realize the dashboards conflict, trust in the data team evaporates. And in data engineering, trust is the only currency that matters.

2. The Nightmare of Time and History

Software engineering mostly deals with the current state. Data engineering deals with time.

Vibe coding focuses on solving the problem for today's snapshot. But what happens tomorrow when a business rule changes? What if a customer moves to a different sales region? If you didn't plan for table granularity and Slowly Changing Dimensions (SCDs) beforehand, your intuitive "vibe coded" joins will silently overwrite the past.

Suddenly, last year's Q3 revenue report changes because a client's status changed today. You broke the immutability of the past. Trying to retroactively fix historical data in a massive Redshift cluster because you didn't design your dimension tables properly isn't just hard, it's a nightmare that stalls the entire data team.

Conclusion: Engineering Requires Foundations

Save the vibe coding for your side projects. When you are responsible for a company's data architecture, you need blueprints.

Regardless of the technology you chose, the act of understanding the business rules and designing the logical models before writing the first line of SQL isn't bureaucracy. It’s fundamental.

You can code on a vibe. But you must architect your data.