Stop Throwing Spark at Everything: The Case for Tech Agnostic Data Engineering

Lucas Ehara — Mon, 13 Jul 2026 12:44:41 +0000

Recently, the industry has developed a reflex. When faced with a new pipeline request, the immediate answer is often to spin up Databricks or write a complex PySpark job. It’s the shiny, distributed computing hammer that makes us feel like we’re doing "real" big data.

If you are processing terabytes of streaming data or training heavy machine learning models, Spark is a lifesaver. The problem starts when this distributed reflex replaces actual architectural pragmatism, and we start using sledgehammers to hang picture frames.

1. The Complexity Tax (The Real Cost of Overkill)

Distributed computing introduces distributed problems. Modern cloud data warehouses are incredibly powerful and often more than capable of handling what many companies consider "big data."

When you force a heavy tool onto a moderate dataset, you spend more time managing cluster configurations, overhead, and driver memory than actually delivering data. Often, a well designed SQL pipeline can handle the workload perfectly. Sometimes, solving a severe table lock issue on a massive dataset doesn't require a brand new execution it just requires stepping back and implementing a smart, sequential batch update strategy. If your first instinct is to scale the tech instead of fixing the approach, you aren't engineering; you're just paying a complexity tax.

2. The Trap of Vendor Driven Logic

A data pipeline is a mechanism to deliver business value, not an excuse to showcase a framework. When you tightly couple your entire mental model and transformation logic to the specific nuances of one platform, you lose flexibility.

Being tech agnostic means understanding the fundamental patterns: idempotency, robust data modeling, and clear, modular transformations. If you build your logic agnostically, you own the architecture. If tomorrow the business needs to optimize costs and shift workloads away from a heavy Spark cluster back to native warehouse processing, a tech agnostic foundation makes that migration a planned engineering task, not a complete rewrite.

Conclusion: Engineering Requires Pragmatism

Save the heavy artillery for the heavy problems. When you are responsible for a company's data architecture, your loyalty must be to the solution, not the stack.

Regardless of the vendors in the market, the act of understanding the true data volume and applying the simplest, most effective tool isn't backward. It's elegant engineering.

You can love a specific technology. But to build sustainable data platforms, you must remain agnostic to it.

Why "Vibe Coding" Your Data Warehouse is a Terrible Idea

Lucas Ehara — Sat, 11 Apr 2026 19:16:15 +0000

Recently, the term "vibe coding" has been everywhere. It’s that practice of sitting down, opening your editor (often with an AI assistant), and just writing code in the flow state. You test, iterate, and build without a massive upfront blueprint.

If you are spinning up a landing page or writing a quick Python automation script, vibe coding is liberating. The problem starts when this culture tries to infiltrate Data Engineering, specifically, Data Warehouse modeling.

The Trap of "Just Writing SQL"

Modern cloud data warehouses like AWS Redshift make it incredibly trivial to transform data on the fly. You get a request, you write a massive CREATE TABLE AS SELECT (CTAS) or a Materialized View with some intuitive joins, and push it to production. The stakeholder is happy today. The vibe is great.

But the reality of data engineering catches up to you, and it hits hardest in two specific areas:

1. The Business Disconnect (The Real Cost)

A Data Warehouse is not an IT project; it is the mathematical and historical representation of a company's business processes.

When you vibe code, you skip the whiteboard. You make technical assumptions about business concepts. You assume a user_id means the same thing across all schemas. But without talking to stakeholders and designing a unified conceptual model, you end up creating isolated data silos.

Suddenly, the Marketing dashboard shows 5,000 "active clients," but the Finance report shows 4,200. You spend the next three weeks in endless alignment meetings trying to debug why the numbers don't match, only to realize the core logic was fundamentally different. Once the business users realize the dashboards conflict, trust in the data team evaporates. And in data engineering, trust is the only currency that matters.

2. The Nightmare of Time and History

Software engineering mostly deals with the current state. Data engineering deals with time.

Vibe coding focuses on solving the problem for today's snapshot. But what happens tomorrow when a business rule changes? What if a customer moves to a different sales region? If you didn't plan for table granularity and Slowly Changing Dimensions (SCDs) beforehand, your intuitive "vibe coded" joins will silently overwrite the past.

Suddenly, last year's Q3 revenue report changes because a client's status changed today. You broke the immutability of the past. Trying to retroactively fix historical data in a massive Redshift cluster because you didn't design your dimension tables properly isn't just hard, it's a nightmare that stalls the entire data team.

Conclusion: Engineering Requires Foundations

Save the vibe coding for your side projects. When you are responsible for a company's data architecture, you need blueprints.

Regardless of the technology you chose, the act of understanding the business rules and designing the logical models before writing the first line of SQL isn't bureaucracy. It’s fundamental.

You can code on a vibe. But you must architect your data.

DEV Community: Lucas Ehara