DEV Community: Aniket Abhishek Soni

Migrating Petabyte-Scale Parquet to Iceberg Without Dropping a Single Row

Aniket Abhishek Soni — Fri, 04 Sep 2026 11:09:57 +0000

Roughly 70% of companies that migrate their data lakes to Iceberg end up with "zombie" datasets: half-migrated, out-of-sync, and consuming double the storage costs. They treat the migration like a switch-flip, and that is exactly how you cause a P0 incident.

Why I chose this topic: I’ve been burned by "big bang" migrations in healthcare environments where downtime is measured in lost compliance certifications. After three failed attempts at manual synchronization, I settled on a shadow-table strategy that treats the migration as a background process, not a deployment window.

Most engineers interact with the Hive metastore daily. They treat it like a source of truth, but it’s actually a glorified index of files that has no idea if a Parquet file is corrupted, moved, or partially deleted. We rely on it to tell us where our data lives, yet we treat it as an immutable oracle. When you move to Iceberg, you aren’t just changing a table format; you’re replacing a loose collection of files with a transactional state machine.

How it actually works

You don't migrate by moving data. You migrate by shadowing the write path.

The strategy is simple: keep your existing Parquet pipeline as the "Primary," and introduce a secondary "Shadow" sink that writes to an Iceberg table simultaneously.

First, you need a dual-write mechanism. If you’re using Spark, don't try to manage this in your application logic. Use a Kafka Connect sink or a structured streaming job that reads from your source (e.g., Kinesis or Kafka) and commits to both targets.

In your Spark job, keep your Parquet sink as-is. Add the Iceberg sink:

// The primary Parquet path
df.write.format("parquet").mode("append").save(parquetPath)

// The shadow Iceberg path
df.writeTo("iceberg_db.shadow_table")
  .tableProperty("write.format.default", "parquet")
  .tableProperty("format-version", "2")
  .append()

The magic happens in the metadata. Iceberg creates a metadata/ folder inside your table directory. This is where the snapshot IDs live. Even if your source data is identical, the two tables will eventually diverge if your schema evolution isn't handled perfectly.

To ensure parity, you need an automated reconciliation job. Run a daily Spark job that performs a MINUS or EXCEPT operation between the two datasets:

-- Reconciliation check
SELECT * FROM parquet_table EXCEPT SELECT * FROM iceberg_table;

If that query returns rows, your shadow write failed or your schema translation is buggy. Do not proceed to the switch-over phase until that query returns zero rows for seven consecutive days.

Photo by Nicolas Arnold on Unsplash

The tradeoffs nobody mentions

This strategy is not free.

First, the storage cost. You are effectively doubling your storage footprint for the duration of the migration. In a healthcare context, this means keeping two copies of PII-encrypted data. Ensure your cloud lifecycle policies are set to expire the Parquet files only after you have fully decommissioned the old table.

Second, the "Ghost Latency." If your Spark job is writing to two different destinations, the overall job duration will be capped by the slower of the two. Iceberg’s commit process—which involves checking for conflicts in the catalog—can add 5-15 seconds to your job. If you have sub-second SLA requirements for your pipelines, this might be a non-starter.

Third, the metadata overhead. If you are migrating a table with 100,000+ partitions, the initial metadata load in Iceberg can cause your Driver OOM (Out of Memory) errors. You’ll need to set spark.driver.memory significantly higher than you think, often into the 16GB-32GB range, just to handle the snapshot state management during the initial transition.

Photo by Celine Nadon on Unsplash

When to reach for it (and when not to)

Reach for the shadow-table strategy if you have a "live" table—one that is being queried by BI tools or downstream microservices 24/7. This is non-negotiable in financial services where a missing row in a ledger report is a regulatory disaster.

Don't reach for this if your data is static. If you have a table that only gets updated once a month, just perform an INSERT OVERWRITE into a new Iceberg table during a maintenance window. The shadow strategy is complex because it’s meant for high-velocity, high-concurrency environments. If you don't have the write volume to justify the complexity, you are just inviting more failure points into your architecture.

Also, avoid this if your Parquet files are highly fragmented (millions of tiny 1KB files). Migrating "bad" data into Iceberg just gives you a "clean" format for "dirty" data. Use the migration window as an opportunity to perform a compaction job before you point your production queries to the new Iceberg table.

Conclusion

Migrating to Iceberg is less about the technical transition and more about the psychological shift from "file-based data" to "transactional data." By shadowing your writes, you buy yourself the time to catch the edge cases—like subtle schema mismatches or timezone shifts in timestamp columns—that would otherwise blow up in your face on a Monday morning.

The goal isn't to be fast. The goal is to reach a state where you can point your production traffic to the new table, delete the old Parquet files, and have your stakeholders remain completely unaware that anything happened under the hood.

Tags: #data #engineering #iceberg #migration

Cover photo by Valentin Lacoste on Unsplash.

Don't put an LLM in charge of your production database

Aniket Abhishek Soni — Fri, 04 Sep 2026 11:09:23 +0000

Last June, a junior analyst pushed a "helpful" GenAI assistant to our internal Tableau-connected lakehouse. Within forty minutes, the agent generated a SELECT * across a 40-terabyte partitioned table joined against a cross-region S3 bucket. The query didn't just fail; it locked the Databricks SQL Warehouse, blew our monthly compute budget in a single afternoon, and triggered a PagerDuty incident that ruined my kid’s birthday dinner. That query cost us $4,200 in DBU burn and an hour of downtime for our actual business stakeholders.

Why I chose this topic: I’m tired of seeing engineers treat Text-to-SQL as a magic wand rather than a dangerous, non-deterministic interface. I wrote this because production-grade data governance requires moving past "prompt engineering" into hard, infrastructure-level constraints.

The decision you are facing isn't whether to use LLMs for data—it's whether you want to build an expensive, unreliable hallucination engine or a system that actually respects your data perimeter. You are currently choosing between three architectural patterns: the "Raw Agent" (blind trust), the "Semantic Layer" (the guardrail approach), and the "Frozen Schema" (the brute-force approach).

The contenders

The "Raw Agent" is what happens when you just point a LangChain SQLDatabaseChain at your Unity Catalog metastore. It’s the "move fast and break things" approach, except you're breaking your company's P&L.

The "Semantic Layer" uses an intermediate abstraction—think dbt Semantic Layer or a specialized metric store—that acts as a firewall between the LLM and the raw SQL. The LLM talks to the model, not the table.

The "Frozen Schema" is the nuclear option. You don't give the LLM the entire catalog. You give it a strictly curated, subsetted DDL definition of exactly three tables, with no write permissions, and a LIMIT clause hard-coded into the underlying execution proxy.

Photo by Zulfugar Karimov on Unsplash

Latency and the cost of non-determinism

In production, latency is a feature, not a bug. If your Text-to-SQL agent takes 15 seconds to parse, plan, and execute, your business users will go back to asking the data team for CSVs.

The Raw Agent is a nightmare here. If you provide a schema with 500 tables, the token count for the system prompt alone will set you back significant latency on every single request. Using gpt-4o, you’re looking at 2-3 seconds of TTFT (Time To First Token) just to generate a JOIN that will likely fail because it missed a join key on a non-indexed column.

The Semantic Layer wins on efficiency. By exposing only core metrics—revenue_by_region, churn_rate_monthly—the LLM has a search space of 20 variables instead of 20,000 columns. You can cache these responses in Redis for common queries. With a 300ms retrieval time from cache, you’re looking at a sub-second user experience that feels like a real product.

Failure modes and the "Hallucination Gap"

The biggest failure mode in Text-to-SQL is the "Syntactically Correct, Semantically Wrong" query. The LLM might write a perfect JOIN statement that executes without error, but calculates gross_margin by subtracting shipping_costs from revenue when your actual business logic requires (revenue - cost_of_goods_sold) - shipping_costs.

Raw Agents fail silently. They give the user the wrong number, and the user makes a million-dollar decision based on it. There is no error message because the SQL is valid.

The Semantic Layer forces the LLM to use pre-defined logic. You move the source of truth from the LLM’s "knowledge" into your dbt project. When the LLM asks for "margin," it hits a view that already contains the logic. If the LLM tries to add a column that doesn't exist, the query parser throws an AnalysisException before the data is even touched.

Photo by Logan Voss on Unsplash

Infrastructure and operational burden

Governance is where most of these projects die. If you aren't using Row-Level Security (RLS) and Column-Level Security (CLS), you are one prompt away from a data breach.

With the Raw Agent, you have to implement RLS on the warehouse side, but you’re still exposing your entire schema. You need to keep your INFORMATION_SCHEMA restricted. If you use Databricks, you’re looking at complex GRANT hierarchies. It’s brittle. If you add a new column for PII, you have to remember to hide it from the LLM’s system prompt, or you're leaking sensitive data.

The Semantic Layer shifts the burden to the platform team. You maintain the semantic layer, and the agent acts as a client of that layer. This is the only way to scale. You don't grant the LLM a connection to the raw tables; you grant it a service account connection to the semantic interface. The service account has SELECT access only to the views you've blessed.

What I'd pick, and why

I’d pick the Semantic Layer every single time. If you’re building in a regulated industry—healthcare or finance—you don't have a choice.

Here is my non-negotiable stack:

The Guardrail: A dedicated Python intermediary (FastAPI) that sits between the LLM and the SQL engine.
The Constraint: Strict LIMIT enforcement on every query. If the LLM doesn't include LIMIT 100, the FastAPI layer injects it automatically before sending it to the warehouse.
The Identity: Never use a generic "data" service account. Use short-lived tokens mapped to the user requesting the data. If the user doesn't have access to the patients table, the agent can't generate a query that hits it.

The caveat? It’s harder to build. You have to write the code that maps natural language intent to specific semantic metrics. It’s not "plug and play." But in production, "plug and play" just means "plug in and pray."

Don't let an LLM write raw SQL against your production lakehouse. Build a layer, wrap it in strict permissions, and force the agent to play by your rules. Your PagerDuty rotation will thank you.

Cover photo by Tyler on Unsplash.

Is Your 'Exactly-Once' Streaming Pipeline Actually Lying To You?

Aniket Abhishek Soni — Tue, 01 Sep 2026 03:13:16 +0000

Three years ago, I spent two weeks debugging a reconciliation issue where a healthcare provider’s balance sheet was off by exactly $42,000 every Tuesday. We were running Spark 2.4, blindly trusting that checkpointLocation was a magical "undo" button for any downstream failure. We had duplicate writes during network partitions, and our downstream SQL database was a mess of upserts gone wrong.

Today, if a node dies, my pipeline recovers, reprocesses the micro-batch, and the end state of my data remains identical to a clean run. I stopped trusting the marketing copy on the Spark docs and started reading the source code of the connectors. If you think "exactly-once" is a toggle you flip in your Spark configuration, you’re about to lose your job—or at least your sleep.

Why the common approach falls short

Most of my peers treat outputMode("append") and a checkpoint path as a set-it-and-forget-it deployment strategy. They assume that because the Spark UI says "Exactly-Once," the entire system—from Kafka to the final OLAP table—is bulletproof.

It isn’t. Spark’s "exactly-once" guarantee is strictly internal. It guarantees that the state of your streaming query is consistent. If Spark crashes, it can resume from the last offset stored in your HDFS or S3 checkpoint directory. The problem is that Spark doesn't control the sink. If your sink isn't idempotent, Spark will write the same data twice during a retry, and your "exactly-once" pipeline just turned into an "at-least-twice" nightmare.

Consider a standard write to S3 as Parquet. If a task fails halfway through writing a file, Spark leaves behind a partial file. When the task retries, it writes again. Unless you are using the S3A committers correctly—specifically the DirectoryStagingCommitter—you end up with garbage data that ruins your downstream partition. I’ve seen production pipelines where the cumulative error from "zombie" files grew to terabytes, slowing down queries by an order of magnitude.

Photo by Dimitri Karastelev on Unsplash

The illusion of atomicity in sinks

If you’re writing to a standard RDBMS using foreachBatch, you are effectively on your own. Spark doesn't know about your SQL transaction. You have to handle it manually.

df.writeStream
  .foreachBatch { (batchDF: DataFrame, batchId: Long) =>
    batchDF.persist()
    batchDF.write.format("jdbc").mode("append").save(...)
    batchDF.unpersist()
  }
  .start()

If the save operation succeeds but the Spark driver crashes before it can commit the offset to the checkpoint, the stream will restart, re-read the batch, and write the data again. Your JDBC sink now has duplicates.

To fix this, you must implement a "batch-id-aware" write pattern. You need a metadata table in your database that stores the latest batchId processed. Inside foreachBatch, you wrap the write in a transaction:

// Pseudocode for the pattern that actually works
batchDF.write.jdbc(url, "dest_table", properties)
metadataDF.write.jdbc(url, "processed_batches", properties)

If you don't atomize the write and the metadata update, you are just praying that the network gods are in a good mood. Relying on "exactly-once" without managing your sink's idempotency is like locking your front door but leaving the garage wide open.

The Kafka-to-Kafka trap

When you’re doing stream-to-stream processing (e.g., Kafka to Kafka), Spark Structured Streaming is actually quite good at exactly-once, provided you use the built-in Kafka source and sink. It uses the Kafka transactional producer API.

However, the moment you deviate from the "supported" path, you break the chain. I once saw a team try to enrich Kafka data by calling a REST API inside a mapPartitions block. When a retry occurred, the API was called again. If that API performed an action (like charging a credit card), the user got charged twice.

Exactly-once in Spark is a state-management feature, not a global distributed transaction protocol. If your pipeline involves external side effects, you must implement your own idempotency keys. I force every single upstream producer to generate a UUID for every event. In the Spark job, I use that UUID to check against a Redis cache before performing any side effect. If the key exists, I drop the event. It’s the only way to sleep at night.

Photo by Egor Komarov on Unsplash

The objections (and my answers)

The common pushback I hear is: "But the Spark documentation says it guarantees exactly-once, so why should I build all this extra infrastructure?"

My answer is simple: Spark’s guarantee is scoped to the Spark application's internal state. It is not a contract with your external database, your external API, or your storage layer. If you treat it as a universal guarantee, you are ignoring the physics of distributed systems. The "distributed snapshot" that Spark takes is only useful if the external system can participate in that snapshot. Most cannot.

Another argument I hear is that the performance hit of idempotency checks is too high. "We can’t do a Redis lookup for every single event in a high-throughput stream."

If you can’t afford an idempotency check, you can’t afford a duplicate. In healthcare or financial services, "exactly-once" isn't a performance optimization; it’s a regulatory requirement. If you’re processing insurance claims and you duplicate a payment, the cost of fixing that data is a thousand times higher than the cost of a few milliseconds of Redis latency. Optimize for correctness first, then optimize for throughput.

Conclusion

Stop using the term "exactly-once" as a shorthand for "I don't have to worry about data quality." You always have to worry.

Spark Structured Streaming is a phenomenal engine, but it is not a magical black box that prevents business logic errors. Exactly-once is an internal property of how Spark manages offsets and state. Once that data leaves the Spark executor to hit a sink, the "exactly-once" guarantee is only as good as your integration.

Use foreachBatch to control your sink. Use idempotency keys for side effects. Check your checkpoint directories for garbage files. If you aren't doing these things, your pipeline is not exactly-once; it’s just a broken system waiting for a high-volume day to reveal its flaws.

Tags: #spark #streaming #data #engineering

Cover photo by Jilbert Ebrahimi on Unsplash.

Is Your Silver Layer Just a Slow-Motion Train Wreck?

Aniket Abhishek Soni — Sun, 30 Aug 2026 05:46:19 +0000

It was 3:15 AM on a Tuesday when the PagerDuty alerts started screaming. Our Bronze-to-Silver processing job, which usually took forty minutes, had been running for six hours. The cluster was pegged, the spill-to-disk metrics were off the charts, and we were locking out the downstream BI dashboard for our compliance team. By the time I manually killed the job and truncated the Silver table to perform a full-refresh, we were three hours behind our SLA. That incident cost the firm roughly $40,000 in regulatory reporting penalties and a very awkward meeting with the CTO. The culprit? A "simple" merge operation on a 5TB table that finally hit a partition skew tipping point.

Why I chose this topic: I spent three years watching engineers treat Delta tables like static files, only to be surprised when their jobs collapsed under volume. I’m writing this because incremental processing isn't a "nice to have" anymore—it’s the only way to keep your sanity in production environments.

Most of you are currently standing at a crossroads. You have a massive Bronze (raw) layer, and you need to feed a Silver (cleansed) layer. You’re either running full-table overwrites, which is a financial death trap as your data grows, or you’re hacking together custom watermarks that break the second a schema changes or a late-arriving record shows up. You’re choosing between "simple but expensive" and "complex but fragile."

The contenders

You have three primary ways to move data from Bronze to Silver in the Delta ecosystem.

First, the Full Overwrite. You read the entire source, apply your logic, and df.write.mode("overwrite") the target. It’s clean, it’s idempotent, and it’s mathematically guaranteed to bankrupt you if your data volume doubles annually.

Second, the Custom Watermark. You track a max_processed_timestamp in a control table, filter your source by event_time > last_processed, and perform a merge into the target. It’s the "classic" approach. It works until you have an update to a record that happened three days ago, which your watermark filter will conveniently ignore.

Third, Change Data Feed (CDF). You enable delta.enableChangeDataFeed = true on your Delta table. Delta Lake then tracks every row-level change (inserts, updates, deletes) in a hidden log. You treat the table like a stream, consuming only the deltas.

Photo by Michael Evans on Unsplash

The cost of doing business

Let’s talk about the bill. Full Overwrites are the most expensive because you pay for the compute to shuffle the entire dataset for every run. If you have 10TB of data and only 1GB changed, you are paying for the other 9.999TB of sorting and shuffling.

Custom Watermarks feel cheaper, but the hidden cost is the merge operation. When you run a MERGE INTO, Spark has to perform a full shuffle of the target table to identify the files containing matching keys. If your Silver table is partitioned by date, and your updates are scattered across two years of data, a single MERGE will force Spark to rewrite every single partition in your table.

CDF is the clear winner here. Because CDF gives you a stream of only the changes, your Silver layer transformation becomes a pure append-only operation if you design it right. You are reading a few hundred megabytes of deltas instead of terabytes of static data. Your compute clusters can be 1/10th the size, and your job duration drops from hours to minutes.

Ops burden and the "missing update" problem

The real nightmare in production is the "missing update." With Watermarks, if a record arrives late or an upstream system triggers a retroactive correction, your job will never see it unless you implement a "look-back" window—which basically forces you back into a partial full-refresh, re-processing data you’ve already handled.

CDF solves this by design. Because CDF captures the transaction log, it doesn't care if an update comes in two hours late. It will show up in the change feed as a new version of the row. You don't have to write complex logic to handle late-arriving data; you just process the stream.

However, CDF comes with an ops tax. You must manage table properties. If you’re using Databricks or Delta OSS, you need to ensure delta.enableChangeDataFeed is set at table creation. If you try to enable it on a massive, pre-existing table, you’re looking at a rewrite of that table's metadata, which can be a blocking operation. You also need to manage your delta.deletedFileRetentionDuration and delta.logRetentionDuration. If your job fails for three days and your retention is only two days, you’ve lost your pointer to the stream. You’ll be doing a full rebuild anyway.

Photo by Jacek Dylag on Unsplash

Failure modes and recovery

The failure mode for Full Overwrites is simple: the job takes too long and times out. Recovery is easy but painful—you just rerun it.

The failure mode for Watermarks is silent data corruption. If your control table update fails after the merge finishes but before the commit metadata is written, you might double-process or lose data on the next run. Tracking the state of your pipeline becomes a distributed systems problem that you probably aren't qualified to solve (neither am I).

CDF has a cleaner failure mode. Since it’s integrated with Spark Structured Streaming, you get checkpointing for free. If the job dies, it restarts from the exact offset in the transaction log. You don't have to guess where you left off. The failure mode isn't "data corruption"; it's "the job stopped." You restart it, it catches up, and you move on.

What I'd pick, and why

I’ve reached a point where I refuse to build a production pipeline that doesn’t use CDF. It is the only way to move from "managing data" to "managing streams."

If you’re sitting on a massive, legacy Silver layer, don't try to migrate it all at once. Start by enabling CDF on your new Bronze ingestions. Build a "Silver-Incremental" table alongside your old "Silver-Legacy" table. Once you prove the latency gains, point your downstream BI tools to the new table and drop the old one.

The caveats:
First, check your storage costs. CDF creates extra files (the change data). It increases your storage footprint by about 10–15%. In the world of cloud storage, that’s pennies compared to the hundreds of dollars an hour you’re burning on massive Spark clusters.

Second, be careful with schema evolution. If you add a column to your Bronze table, your Silver job needs to handle that. When using CDF, you’re often casting and transforming in a stream. If a schema change isn't backwards compatible, your streaming job will crash. You need to implement schema evolution (spark.databricks.delta.schema.autoMerge.enabled = true) on your write operations, but even then, test your Silver logic in a dev environment first.

Third, don't over-partition. A common mistake is partitioning by customer_id or transaction_id. If you have a high-cardinality column, your partitions will be too small (the "small file problem"). Stick to partitioning by date or region, and let Delta’s Z-Ordering handle the performance optimization.

Stop fighting the infrastructure. The tools to handle incremental data are already built into the Delta protocol. If you’re still doing full refreshes, you’re not an engineer—you’re a janitor cleaning up after a bad architecture. Stop the bleeding, enable the feed, and save your sleep.

Cover photo by Arash on Unsplash.

The Orchestration Fallacy: Why Your Medallion Architecture Doesn't Need a Swiss Army Knife

Aniket Abhishek Soni — Fri, 28 Aug 2026 13:00:36 +0000

The most persistent myth in data engineering is that you need a dedicated, heavy-duty orchestrator to manage a medallion architecture. If I hear one more architect argue that we need a complex DAG to move data from Bronze to Silver, I’m going to assume they’ve never had to debug a zombie process at 3:00 AM on a Sunday.

You don't need a complex orchestration layer. You need a reliable state machine that triggers your jobs when the data lands. Most of the complexity we build into our pipelines—retries, backfills, complex branching—is a direct result of choosing the wrong tool for the wrong scale.

You are likely staring at a familiar problem: you have a landing zone in S3 or ADLS, a Databricks workspace, and a business requirement that says the Silver table needs to be updated within fifteen minutes of the Bronze ingestion. You’re trying to decide whether to pay for the managed overhead of a workflow engine or just hack together some cloud-native glue.

The contenders

First, there’s AWS Step Functions. It’s the ultimate "I don't want to manage a server" play. You define your pipeline in Amazon States Language (ASL), essentially a JSON-based state machine. It integrates with everything in AWS, but it treats your data pipeline like a series of API calls.

Then, there’s Apache Airflow. It’s the industry standard for a reason: it’s Python. If you can write a script, you can write a DAG. But it’s a high-maintenance beast. You’re managing an environment—whether it’s MWAA, Cloud Composer, or a self-hosted Kubernetes cluster—and you’re fighting the "executor" war.

Finally, there’s Databricks Workflows. This is the "keep it in the house" option. It’s built into the platform where your compute already lives. It’s not just a job scheduler; it’s an integrated execution engine that knows the difference between a cluster startup failure and a Delta Lake transaction conflict.

Photo by Kellie Shepherd Moeller on Unsplash

Cost and the hidden tax of complexity

People often look at the price tag of a single execution and ignore the operational tax.

Step Functions looks cheap until you reach scale. At $25 per million state transitions, a complex pipeline with thousands of tasks will bleed your budget dry. More importantly, the cost of debugging a complex ASL JSON file is astronomical. When a TaskFailed event fires, you’re parsing CloudWatch logs and trying to reconstruct state from a visual graph that looks like a bowl of spaghetti.

Airflow is a hidden cost center. If you’re running it on a managed service like MWAA, you’re paying for the base instance price, which is rarely under $200-$300 a month before you even run a single job. If you’re self-hosting on EKS, factor in the engineering time. I’ve seen teams spend 20% of their "data engineering" time just keeping the Airflow scheduler and web server alive. That’s a massive salary tax to pay for the privilege of writing PythonOperator stubs.

Databricks Workflows is arguably the cheapest for medallion pipelines because of its integration with the compute layer. You aren't paying for a separate orchestration cluster. You’re using the existing Databricks compute, and the job orchestration is effectively free or negligible. You stop paying for the orchestrator the moment the job finishes. In a medallion architecture, where you’re often running jobs in sequence, this tight coupling is a feature, not a bug.

Failure modes and the "retry" trap

Let’s talk about real-world failure. In healthcare pipelines, "fail fast" is a requirement, not a suggestion.

In Airflow, the biggest failure mode is the "scheduler heartbeat." When your DAGs get too numerous or complex, the scheduler lags. You wake up to a dashboard showing nothing running, while your actual data is missing its SLA. You’re left digging through airflow-scheduler.log to find out why the heartbeats timed out. It’s a distributed system problem that you now own.

Step Functions has a "max duration" limit of one year, but it struggles with long-running, stateful processes. If your transformation job takes four hours and the underlying Lambda or Batch job hangs, Step Functions will keep the execution open, eating costs and potentially causing downstream concurrency limit issues. Retrying a failed step is easy, but retrying a failed data state (like a partial Delta write) is a nightmare you have to code manually.

Databricks Workflows has a massive advantage here: it understands the Delta Lake state. If a job fails because of a ConcurrentAppendException or a cluster start timeout, the platform handles the retries internally. It’s aware of the underlying compute, so if a node goes down, it doesn't just re-run the task; it re-provisions the cluster. It’s the only one of the three that doesn't treat your data pipeline like an abstract black box.

Photo by Francesca Tosolini on Unsplash

The operations burden and the "Python" illusion

Everyone loves Airflow because they love Python. It’s a trap.

When you write an Airflow DAG, you aren't writing data engineering code; you’re writing infrastructure code. You’re managing dependencies, requirements.txt files, and venv isolation. If you want to upgrade your Spark version, you might have to migrate your entire Airflow environment. It’s a classic case of the abstraction leaking.

Step Functions requires you to become an expert in Amazon States Language. It’s a declarative nightmare. You’ll find yourself writing nested Choice states just to handle a simple conditional logic that would have been a single if statement in any other language. You’re trading Python flexibility for AWS-specific lock-in.

Databricks Workflows allows you to define your pipeline in code—using Databricks Asset Bundles (DABs)—but it doesn't force you to manage the infrastructure. You define your task dependencies in YAML or Python, and the platform handles the deployment and the execution. It’s the only one that feels like a modern developer experience rather than a 2015-era sysadmin chore.

What I'd pick, and why

If you are building a medallion pipeline today, stop looking at Airflow. Unless you have a massive, heterogeneous environment where you need to orchestrate non-Databricks tasks (like triggering a legacy mainframe job or an on-prem file transfer), Airflow is overkill. It’s a platform for general-purpose workflow management, not a specialized tool for Delta Lake pipelines.

Step Functions is for when you are deep in the AWS ecosystem and need to trigger event-driven microservices. It’s not for heavy-duty ETL. If your pipeline involves moving gigabytes or terabytes through a Silver-to-Gold transformation, Step Functions will eventually become a bottleneck for your sanity.

I pick Databricks Workflows every single time for medallion architectures.

The caveat? You are locking yourself into the Databricks ecosystem. If your CTO decides to pivot to Snowflake or BigQuery tomorrow, you’re rewriting your orchestration layer. But let’s be honest: you’re already locked into Delta Lake. If you’re going to be locked in, you might as well use the tools that make your life easier.

Databricks Workflows with Databricks Asset Bundles (DABs) is the current gold standard. It gives you the CI/CD pipeline, the version control, and the observability you need without the "scheduler maintenance" headache of Airflow or the JSON-hell of Step Functions.

Use the right tool for the job. You’re a data engineer, not an Airflow cluster administrator. Spend your time fixing your data quality issues instead of babysitting a scheduler.

Cover photo by Tyler on Unsplash.

Snowpark vs. Spark: The 'Free' Compute Trap and Why You’re Probably Wrong

Aniket Abhishek Soni — Wed, 26 Aug 2026 13:00:12 +0000

Six months ago, our data pipeline was a sprawling Frankenstein of EMR clusters running Apache Spark 3.3.1, held together by custom Terraform modules and a prayer. We spent 20 hours a week just babysitting memory allocation, tuning spark.executor.memoryOverhead, and praying that a rogue join wouldn't trigger an OOM (Out of Memory) error at 3:00 AM on a Sunday.

Today, we run the same logic in Snowpark. Our infra team hasn't touched a node configuration in months. But—and this is a big, expensive but—our monthly Snowflake consumption bill for the dev environment alone would make a CFO weep.

You are currently deciding whether to keep your Spark jobs in a dedicated cluster or migrate them into the Snowflake ecosystem. Don't let the marketing decks fool you. This isn't just a syntax switch; it’s a trade-off between "operational misery" and "financial opacity."

The contenders

On one side, we have Apache Spark, specifically the 3.5.x line. It’s the industry standard for a reason: it’s battle-tested, highly tunable, and completely agnostic to where it runs. You can run it on EMR, Databricks, or a collection of dusty laptops under your desk. It’s a distributed computing framework that demands you understand the physics of your data.

On the other, we have Snowpark (Python API). It’s not a framework in the traditional sense; it’s an abstraction layer that translates your DataFrame operations into SQL, which then runs on Snowflake’s proprietary compute engine. It looks like Spark, it smells like Spark, but under the hood, it’s just Snowflake doing what Snowflake does best: pushing compute to where the data lives.

Photo by Heather Newsom on Unsplash

The cost of convenience

Spark is cheap until you factor in the "human" cost. If you’re running an EMR cluster, you aren't just paying for EC2 instances. You’re paying for the IAM roles, the VPC peering, the metadata store (Glue Catalog), and the inevitable hours an engineer spends debugging a java.lang.OutOfMemoryError.

However, Snowpark’s pricing model is dangerous for those who view it as a drop-in replacement. When you call session.table("large_table").join(...) in Snowpark, you are at the mercy of Snowflake’s Warehouse sizing. If your transformation requires a massive spill to disk, Snowflake will spin up a larger warehouse to handle the memory pressure. That isn't just an extra 5 minutes of compute; it’s a jump from an X-Small to a Large warehouse, effectively quadrupling your hourly cost instantly.

In my experience, a workload that costs $50/day on EMR can easily spike to $150/day on Snowpark if you aren't vigilant about warehouse_size settings. The benefit? You stop caring about garbage collection tuning. The downside? You start sweating every time a developer hits "Run" on a notebook without checking the join cardinality.

Operational burden and failure modes

Spark failure modes are predictable but annoying. When a Spark job dies, you get a beautiful, cryptic wall of Java stack traces. You’ll be hunting through the Spark UI, looking at executor logs, and trying to figure out why your spark.sql.shuffle.partitions was set too low for the data skew. You are the architect of your own failure.

Snowpark failure modes are different. When Snowpark breaks, it usually fails at the compilation phase. Because it’s translating to SQL, you’ll occasionally hit a "SQL compilation error" that is completely detached from the Python code you wrote.

Furthermore, let’s talk about libraries. In Spark, if you need a specific version of scikit-learn or a custom C-extension, you build a Docker image, push it to ECR, and point your job there. It’s tedious, but it’s deterministic. In Snowpark, you are constrained by what Snowflake allows in their Anaconda channel. If a library isn't there or requires a specific C-dependency not supported in the restricted Snowpark environment, you are stuck. You can upload custom packages to a stage, but performance takes a hit, and it feels like a hack.

Photo by 404 on Unsplash

Performance parity

If you are doing simple ETL—filtering, renaming, basic aggregations—Snowpark is often faster than Spark. Why? Because you’re avoiding the overhead of moving data between S3 and the compute nodes. The data is already sitting in Snowflake’s micro-partitions.

But for heavy, iterative machine learning or complex graph processing, Spark wins. Spark’s ability to cache RDDs and keep data in memory across stages is far superior to Snowpark’s approach, which is fundamentally tied to the execution plan of the underlying SQL. If your logic requires massive shuffles or complex cross-joins, Snowpark can struggle because it’s effectively limited by the size of the temporary storage assigned to the warehouse.

I’ve seen jobs that took 40 minutes on Spark finish in 10 on Snowpark, but I’ve also seen jobs that ran fine on Spark fail with a "Query too complex" error on Snowpark because the generated SQL became a recursive monster that Snowflake’s query optimizer couldn't untangle.

What I'd pick, and why

If you have a dedicated platform team and your data volume is in the multi-petabyte range, stay on Spark. You need the granular control over memory management and the ability to optimize shuffles. You need to be able to run on spot instances and use Graviton processors to keep costs sane.

If you are a lean team, or if your data already lives in Snowflake and you’re tired of the "S3-to-Spark-to-S3" tax, choose Snowpark. The developer velocity is undeniable. Being able to write Python that compiles to highly optimized SQL is a superpower. You will save hundreds of hours in infrastructure maintenance, and for many companies, that time is worth more than the premium you’ll pay on the Snowflake compute bill.

My caveat? Use Snowpark only if you have strict guardrails. Implement mandatory warehouse tagging and automated alerts for warehouse resizing. Treat your Snowpark code like production application code, not like a script. If you treat it like a sandbox, you’ll wake up to a bill that will make your CEO ask why we’re paying $400 for a simple customer aggregation.

The era of manual cluster management is dying, but the era of "set and forget" compute is just a different kind of trap. Choose your poison carefully.

Cover photo by Tyler on Unsplash.

How I standardized PySpark deployments with multi-stage Docker builds

Aniket Abhishek Soni — Mon, 24 Aug 2026 18:22:31 +0000

Why I chose this topic: I’ve spent too many 2:00 AM outages debugging "it works on my machine" failures caused by mismatched Python versions or missing shared libraries in PySpark executors. This pattern is the only way I’ve found to force consistency across the entire SDLC.

You ship the job. It passes CI. Then it hits the cluster, and the executor pods crash-loop with an ImportError because your local pandas version is 2.2.0, but the base image on your EMR cluster is pinning an ancient build of 1.3.5. You spend three hours SSHing into nodes or digging through CloudWatch logs, only to realize the environment variables are different, the LD_LIBRARY_PATH is missing, or someone updated a private package in Artifactory without telling you.

It’s a miserable loop. You’re managing infrastructure drift rather than writing data pipelines. We treat our application code like a first-class citizen, but we treat our execution environment like a neglected basement.

The real problem

The problem isn't your code; it’s the disconnect between the build-time environment and the runtime environment. Most PySpark deployments rely on "bootstrap scripts" or "init actions" to install dependencies on the fly. This is a recipe for disaster. Every node in your cluster tries to pip install simultaneously, resulting in network throttling, race conditions, or partial installs that fail midway through a 4-hour job.

If you aren't shipping a single, immutable container image that contains your OS, your Python runtime, your dependencies, and your job code, you aren't doing reproducible data engineering. You’re just gambling.

Photo by Bernd 📷 Dittrich on Unsplash

Step 1: Defining the build stage

We need to keep the final image slim. We don’t need compilers, C++ headers, or git credentials in the production image. We use a multi-stage build to isolate the "messy" build tools from the "clean" runtime.

I start with an official Python slim image. Why slim? Because alpine with PySpark is a nightmare of musl vs glibc incompatibilities that will eventually break your C-extensions like pyarrow.

# Stage 1: Build dependencies
FROM python:3.10-slim AS builder

RUN apt-get update && apt-get install -y --no-install-recommends \
    build-essential \
    libpq-dev \
    gcc \
    && rm -rf /var/lib/apt/lists/*

WORKDIR /app
COPY requirements.txt .

# Install to a local folder so we can copy it easily
RUN pip install --no-cache-dir --user -r requirements.txt

Step 2: Constructing the runtime stage

Now we move to the runtime stage. We copy only the installed packages from the builder stage. I prefer to keep the Python site-packages in a predictable location and ensure the PYTHONPATH is set correctly. Note that I am not including the job code here yet; I like to keep the environment static and the code dynamic (or baked in, if your CD pipeline allows).

# Stage 2: Final runtime
FROM python:3.10-slim

# Install runtime-only system dependencies (e.g., libpq for psycopg2)
RUN apt-get update && apt-get install -y --no-install-recommends \
    libpq5 \
    && rm -rf /var/lib/apt/lists/*

# Copy installed packages from builder
COPY --from=builder /root/.local /root/.local
ENV PATH=/root/.local/bin:$PATH
ENV PYTHONPATH=/root/.local/lib/python3.10/site-packages:$PYTHONPATH

# Set up non-root user for security
RUN useradd -m sparkuser
USER sparkuser
WORKDIR /home/sparkuser/app

# Copy your source code
COPY --chown=sparkuser:sparkuser ./src ./src

Step 3: Integrating with Spark configuration

Once you have your image pushed to your container registry (ECR, GCR, or ACR), you need to tell Spark to actually use it. The configuration keys are non-negotiable. If you aren't running on K8s, the logic is similar for YARN, but Kubernetes is where this pattern truly shines.

When submitting your job via spark-submit or a K8s manifest, point directly to your image.

# Example spark-submit command
spark-submit \
  --master k8s://https://<k8s-api-server>:6443 \
  --deploy-mode cluster \
  --conf spark.kubernetes.container.image=your-registry/repo/pyspark-job:latest \
  --conf spark.kubernetes.container.image.pullPolicy=Always \
  --conf spark.executorEnv.PYTHONPATH=/home/sparkuser/app/src \
  local:///home/sparkuser/app/src/main.py

One subtle trap: spark.kubernetes.container.image.pullPolicy. If you tag images as latest (which you shouldn't do in production, but we all do in dev), set this to Always. If you use immutable tags like v1.2.3, set it to IfNotPresent.

Photo by BoliviaInteligente on Unsplash

Lessons learned from production

The C-Extension Trap: If your job relies on pyarrow or pandas, verify that your Docker build is using the same glibc version as the base Spark image. If you mix and match, you will get obscure segmentation fault errors that appear only on 1 out of every 50 executors.
Size Matters: If your image is over 2GB, your pod startup time will skyrocket. If you’re pulling 2GB per executor on a 100-node cluster, you are effectively performing a self-inflicted DDoS attack on your container registry. Keep it lean, or use a local pull-through cache.
The User Problem: Spark containers often default to root. If your corporate security policy requires non-root users, you must explicitly set USER sparkuser in the Dockerfile and ensure your K8s SecurityContext doesn't conflict with that user ID.
Dependency Locking: Always use pip-compile or poetry to generate a requirements.txt with hashes. A simple pip install without version pinning is a ticking time bomb. I’ve seen production jobs break because a sub-dependency released a "patch" that broke the Spark context initialization.

Conclusion

Standardizing on multi-stage builds isn't just about "best practices." It's about reclaiming your time. By defining the environment in Docker, you make the environment part of the code review process. If a developer needs a new library, they modify the Dockerfile or requirements.txt, which triggers a CI build that tests that change before it ever touches production. No more surprises, no more bootstrap scripts, and no more guessing why the job failed at 2:00 AM.

Try it: Take your most unstable PySpark job, write a multi-stage Dockerfile for it, and deploy it to a staging environment. Compare the startup time and the frequency of "environment-related" failures against your current deployment method. You won’t go back.

Tags: #docker #pyspark #dataengineering #devops

Cover photo by Jonas Smith on Unsplash.

Stop Choosing Between Delta Lake and Iceberg: How UniForm Ends the Format Wars

Aniket Abhishek Soni — Sat, 22 Aug 2026 04:52:36 +0000

Two years ago, I spent six weeks migrating a 400TB churn-prediction dataset from Delta Lake 2.4 to Apache Iceberg 1.3 just because our new executive hire insisted on using Trino for ad-hoc SQL. We broke three upstream Spark jobs, corrupted a partition manifest, and spent a weekend manually editing JSON metadata files in S3.

Today, that same pipeline is running on Delta 4.2 with UniForm enabled. My Trino users query the Iceberg-compatible metadata, my Spark jobs use the native Delta logs, and I haven't touched a manifest file in eight months. The "format war" is officially a waste of your time.

Why I chose this topic: I’m tired of seeing senior engineers treat table formats like sports teams. We are infrastructure providers, not fanatics, and the technical debt of format migration is a tax your balance sheet shouldn't be paying in 2026.

Stop acting like your choice of table format is a foundational architectural decision. It isn't. It’s a storage implementation detail that should be abstracted away from your compute engines. If you are currently sitting in a meeting debating whether to "standardize on Iceberg" while your primary ETL workloads are locked into Delta, you are prioritizing dogma over delivery.

Why the common approach falls short

The "pick a side" strategy ignores the reality of modern polyglot data stacks. In the financial services sector, I see teams force-feeding Iceberg into systems that don't need it because they want "openness," only to find that their Spark-based streaming ingestions struggle with Iceberg's commit overhead. Conversely, I see teams stick to Delta and effectively orphan themselves from the best-of-breed query engines that prioritize Iceberg’s mature REST catalog support.

The common failure mode is the "dual-write" pattern. Engineering teams try to maintain two copies of the same data—one as a Delta table, one as an Iceberg table. This is a disaster waiting to happen. You get drift. You get one table updated at 02:00 UTC and the other failing at 02:05 UTC because of a transient networking blip. Then your dashboard shows different numbers for the same metric, and suddenly you’re explaining to a VP of Finance why the company’s revenue shifted by $40k overnight.

Photo by Frames For Your Heart on Unsplash

The mechanics of the unified layer

UniForm (Universal Format) effectively makes the choice of format a logical abstraction rather than a physical one. By enabling delta.universalFormat.enabledFilesystems and delta.universalFormat.icebergCompatVersion in your table properties, you are telling the Delta log writer to perform the heavy lifting of generating Iceberg metadata on the fly.

When you issue a MERGE INTO or a standard INSERT in Spark 4.0, the engine writes the Parquet data files and the Delta _delta_log JSON/Checkpoint files. With UniForm, it simultaneously generates the Iceberg manifest lists and snapshot files. You aren't duplicating data; you are duplicating the metadata.

ALTER TABLE prod.customer_transactions SET TBLPROPERTIES (
  'delta.universalFormat.enabledFilesystems' = 's3',
  'delta.universalFormat.icebergCompatVersion' = '2'
);

Once that property is set, any engine that understands Iceberg—Trino, Flink, StarRocks—points to the same S3 prefix. The Delta log is the source of truth, and the Iceberg metadata acts as a read-optimized projection. You don't need to perform an msck repair or a full table migration. You just need to ensure your compute engine supports the Iceberg version you’ve negotiated.

The reality of failure modes

Do not think this is a "set it and forget it" panacea. The primary risk isn't the data; it’s the consistency of the metadata. If you are running an older version of a query engine that doesn't fully support Iceberg V2 features (like row-level deletes or complex partitioning), it will choke on the UniForm-generated metadata.

I once saw a Flink job crash because it expected the Iceberg metadata to reflect a snapshot state that hadn't been fully flushed to the Delta log yet. We solved this by adjusting the delta.checkpointInterval and monitoring the latency of the Iceberg metadata generation. If your delta-to-iceberg-conversion lag exceeds your business SLA for fresh data, you haven't fixed the problem—you've just moved it into a background process.

You must treat the Iceberg metadata as a materialized view of your Delta table. If the view is stale, your query engines will be wrong. Monitor the file modification times in your metadata/ folder. If they aren't updating in lockstep with your main Delta logs, your conversion job is failing silently.

Photo by Daniel Lloyd Blunk-Fernández on Unsplash

The objections (and my answers)

"But what about vendor lock-in?"

This is the most common objection I hear. The argument is that by using Delta's UniForm, you are still "locked in" to the Delta protocol. My response is simple: define "locked in." If you can point Trino, Flink, and DuckDB at your data and query it without a specialized connector, you aren't locked in. The data is in Parquet. The metadata is standards-compliant Iceberg. If Delta disappeared tomorrow, you have a perfectly valid Iceberg table sitting in your bucket. The "lock-in" is purely in the writer engine, and let's be honest—you're going to keep using Spark or Flink anyway.

"Won't this increase storage costs?"

It does, slightly. Generating Iceberg metadata files takes up space. But let's look at the math. If you have 400TB of data, the metadata files account for maybe a few gigabytes. If you are worried about the storage cost of an extra 0.001% of your footprint, you have much bigger problems in your cloud billing department. The cost of manual migration, inconsistent data, and engineering time wasted on format parity is orders of magnitude higher than the storage cost of a few extra JSON and Avro files.

"Is this production-ready for mission-critical healthcare data?"

I’ve been running this for 14 months in a HIPAA-compliant environment. The key is to keep your Delta version current. Using delta-spark 4.2+ is non-negotiable. The stability of the UniForm implementation is tied directly to the version of the Delta kernel. Don't try to run this on legacy Databricks runtimes from 2022 and expect it to behave. If your platform team isn't willing to keep the stack updated, then yes, stay away from it. But if you’re maintaining your infrastructure, it’s arguably safer than maintaining two separate, competing table formats.

Conclusion

The era of choosing sides is dead. We are moving toward a world where the file format is an internal implementation detail of the storage layer, and the table format is just a view. By leveraging UniForm, you stop being a caretaker of format-specific metadata and start being an architect of data flow.

Stop the migrations. Stop the dual-writing. Stop the "Iceberg vs. Delta" religious wars. Enable the compatibility layers, point your engines at the unified metadata, and spend your time building features that actually generate value for your users. In 2026, the best engineer in the room isn't the one who knows every obscure nuance of the Iceberg manifest spec; it’s the one who makes the entire storage layer invisible to the rest of the company.

Cover photo by Tyler on Unsplash.

Stop Pretending BigQuery and Databricks Are Just 'SQL Warehouses

Aniket Abhishek Soni — Thu, 20 Aug 2026 15:19:15 +0000

Why I chose this topic: I’m tired of seeing architecture decisions made based on marketing whitepapers that ignore the reality of a 3 AM pipeline failure. Most engineers pick their platform based on hype; I’m writing this to force you to pick based on your actual ops budget.

You ship the job. It passes CI. Then, the P99 latency on your dashboard spikes from 400ms to 12 seconds, and the CFO is asking why the "serverless" bill just hit five figures for a simple join. You’re left staring at a query profile that tells you absolutely nothing useful.

The industry wants you to believe that BigQuery and Databricks SQL are interchangeable commodities. They aren't. Choosing between them is a choice between trading away your time for convenience (BigQuery) or trading away your sanity for control (Databricks).

Most of my peers argue that "it’s all just SQL, so pick the cheapest one." That is a dangerous simplification. In financial services, where I spend most of my time, the cost of a failed compliance report is higher than the cost of a slightly inefficient query. Treating these platforms as simple SQL interfaces ignores the underlying storage engines, the concurrency models, and the reality of how they handle partition evolution.

Why the common approach falls short

The "benchmark first" mentality is the primary reason projects go off the rails. You see a vendor-sponsored TPC-DS benchmark showing 2x speed on Databricks, so you migrate. Six months later, you realize you’re spending 40% of your engineering time tuning Z-ORDER clusters and managing VACUUM commands.

The common approach assumes that performance is purely a function of the engine. In practice, performance is a function of maintenance.

In BigQuery, you don't manage vacuuming. You don't manage cluster sizes. You don't manage the spark.sql.shuffle.partitions setting. If you’re coming from a world where you need to optimize your data layout to hit your SLAs, BigQuery feels like magic. But that magic has a price tag that is often opaque. When you run SELECT * on a petabyte-scale table without a WHERE clause in BigQuery, you aren't just slowing down the system; you’re literally burning cash. If you don't have rigid project-level quotas, you’re one rogue intern away from a six-figure invoice.

Photo by Igor Omilaev on Unsplash

The illusion of "Serverless" simplicity

I once worked on a migration where we moved a massive ETL pipeline from a custom Spark cluster to BigQuery. The "serverless" aspect was the selling point. For three weeks, it was great. Then, the INFORMATION_SCHEMA.JOBS_BY_PROJECT started showing a pattern: our complex, nested JSON parsing was hitting the 100GB per-query shuffle limit repeatedly.

In BigQuery, you hit a wall. When you hit a query limit, your only recourse is to rewrite the SQL into multiple stages or utilize BigQuery Scripting. You are at the mercy of Google’s internal query optimizer. You can’t "tune" the engine—you can only tune your own code.

-- BigQuery: You're stuck with their optimizer
-- If this hits the 100GB limit, you have to break it apart
SELECT
  user_id,
  ARRAY_AGG(STRUCT(event_type, timestamp)) as events
FROM `prod.events`
GROUP BY 1;

Contrast this with Databricks SQL. If a query is failing or underperforming, I can jump into the Spark UI. I can see the exact task skew. I can look at the physical plan, identify a broadcast join that’s failing, and force a hint.

-- Databricks: You have the levers
SELECT /*+ BROADCAST(dim_users) */
  fact.id,
  dim_users.name
FROM fact_table fact
JOIN dim_users ON fact.user_id = dim_users.id;

In Databricks, the "failure mode" is that you spend time fixing the query. In BigQuery, the "failure mode" is that the query just fails, and your hands are tied by the API.

The operational tax of the Lakehouse

Databricks SQL is fundamentally a lakehouse play. You are managing the underlying Delta Lake tables. This gives you immense power. You can travel back in time with VERSION AS OF, you can MERGE data with transactional integrity, and you can control how your files are laid out on S3 or ADLS.

But let’s be honest: VACUUM is a pain. If you forget to run it, your storage costs spiral because you’re keeping infinite history of every tiny update. If you don't OPTIMIZE your tables, your read performance degrades as your data evolves. This is "operational tax."

In healthcare, we handle PII and HIPAA-regulated data. Databricks allows me to control the storage layer access through Unity Catalog at a granular level. I can move data between regions or cloud providers without re-ingesting everything because it’s just Parquet/Delta. BigQuery is a walled garden. Once your data is in BigQuery Storage, you are using the BigQuery API. You are locked in. Moving out requires an egress bill that will make your finance department cry.

Photo by Oxa Roxa on Unsplash

The objections (and my answers)

The biggest pushback I get is: "But BigQuery is cheaper for ad-hoc analysts."

My answer? Only if you have perfect governance. BigQuery is a "spend-now, pay-later" platform. It’s excellent for teams that don't have a dedicated data engineer to manage infrastructure. If your organization is small, and you don't have the headcount to manage a Databricks workspace, BigQuery is the only logical choice. You pay the premium for the lack of operational overhead.

Another objection: "Databricks SQL is too expensive because of the compute clusters."

My answer: Databricks Serverless SQL warehouses are getting better, but yes, they cost more than a raw BigQuery query if they sit idle. However, the cost of an engineer’s time to fix a BigQuery performance bottleneck that can't be tuned—because the optimizer is a black box—is far higher than a 15% delta in compute costs. If you are at a scale where you are running massive, repetitive ETL jobs, you should not be paying for the "serverless" convenience of BigQuery. You should be paying for the "control" of Databricks.

Finally, people argue that BigQuery’s ML integration (BigQuery ML) is a game changer. It is, until you need a custom library that Google doesn't support. Then you’re back to exporting your data to a vertex AI pipeline, which defeats the purpose. Databricks handles the transition from SQL to Python/MLflow seamlessly because it’s the same underlying Spark environment.

Conclusion

If you are a lean startup or a mid-sized team with one or two data generalists, choose BigQuery. The lack of infrastructure management is worth the risk of an occasional surprise bill. You want to query, visualize, and get out.

If you are in an enterprise environment, especially in finance or healthcare, where data governance, auditability, and custom optimization are non-negotiable, choose Databricks. You need the ability to reach under the hood when things inevitably break.

Don't choose based on a benchmark. Choose based on how much time you want to spend being a database administrator versus a data engineer. I choose the control, even if it means I have to run a VACUUM command at 3 AM once in a while. At least I can fix it.

Cover photo by Marcin Jozwiak on Unsplash.

How Unity Catalog’s Attribute-Based Access Control Saved My Audit Trail

Aniket Abhishek Soni — Tue, 18 Aug 2026 11:31:42 +0000

Why I chose this topic: In financial services, "Role-Based Access Control" (RBAC) is a ticking time bomb that inevitably explodes once your headcount hits double digits. I’m writing this because I spent three weeks cleaning up a PII leak caused by a developer who "just needed" a role that had too much power, and I never want to do that again.

The PagerDuty alert hit at 3:14 AM on a Tuesday: HIGH PRIORITY: Data Governance Violation - Unauthorized PII Exposure.

My logs showed that a junior analyst running a notebook in a sandbox environment had successfully queried the transactions_prod table. They shouldn’t have seen the customer_ssn or credit_card_number columns. But there they were, printed out in cleartext in the stdout of a cluster that had no business touching production data. The cost? A mandatory report to our DPO, a week of manual audit logs review, and a very awkward conversation with our internal compliance team.

What we saw

The initial panic was centered on the GRANT statements. We checked sys.access in the Unity Catalog (UC) metastore. Everything looked correct. The analyst’s user group, analyst_sandbox_role, did not have SELECT on the transactions_prod table.

We chased shadows for four hours. We checked if the user had elevated their privileges via a service principal. We looked for rogue GRANT commands in the audit logs. We even scrutinized the cluster configuration, wondering if someone had bypassed the spark.databricks.passthrough.enabled setting.

Nothing. The permissions were locked down. The user technically didn't have access. But they still had the data.

Photo by Josh Snader on Unsplash

Root cause

The culprit wasn't a bypass of the security layer; it was a "Role Explosion" bug combined with a classic "God-mode" service principal.

We had an automated pipeline that synced our Active Directory (AD) groups to Databricks. The analyst was part of a group called global_data_science. That group had been granted SELECT access to a view called transactions_masked. This view, however, was built on top of a table where column-level security had been misconfigured.

Specifically, the view relied on a hard-coded CASE statement to nullify PII. When a senior engineer updated the underlying table schema to add a new PII field, they forgot to update the SELECT clause in the view. The view effectively defaulted to "show all" for any new columns added to the base table. Because the global_data_science role inherited broad read permissions from a legacy environment, the analyst just queried the underlying table directly.

Our RBAC model was brittle. We had thousands of granular roles, and the logic was "who you are" rather than "what the data is." We were managing permissions like we were still using SQL Server 2008.

Photo by Morthy Jameson on Unsplash

The fix

We nuked the old role-based approach and moved to Attribute-Based Access Control (ABAC) using Unity Catalog’s TAG system and dynamic masking functions.

Instead of managing who gets to see what table, we defined a policy based on the data’s properties. We tagged the sensitive columns in our UC catalog:

ALTER TABLE transactions_prod ALTER COLUMN customer_ssn SET TAGS ('pii' = 'high');
ALTER TABLE transactions_prod ALTER COLUMN credit_card_number SET TAGS ('pii' = 'high');

Then, we wrote a standard User-Defined Function (UDF) that masks data based on the presence of these tags. The core of the fix was applying a MASK to the column, which is evaluated at query time:

CREATE OR REPLACE FUNCTION mask_pii(col STRING)
  RETURN CASE WHEN IS_ACCOUNT_GROUP_MEMBER('pii_authorized_team') THEN col
  ELSE '***-**-****' END;

ALTER TABLE transactions_prod ALTER COLUMN customer_ssn SET MASK mask_pii;

Now, the permission isn't attached to the user’s role; it’s attached to the data attribute itself. If you aren't in the pii_authorized_team, the UDF fires automatically. It doesn't matter what role the user has, what notebook they are using, or if they are on a sandbox or production cluster. The UC engine enforces the function at the metadata level.

What we changed so it never happens again

We stopped thinking about "permissions" and started thinking about "data contracts."

First, we implemented a mandatory tag-enforcement policy. We use the Databricks Terraform provider to ensure that any new table created in the prod catalog must have a sensitivity tag. If a developer tries to create a table without tagging columns as public, internal, or pii, the CI/CD pipeline fails the deployment. You cannot create a table in production without explicitly telling the system what kind of data lives inside it.

Second, we moved away from managing access via individual role membership. We use IS_ACCOUNT_GROUP_MEMBER inside our masking UDFs. This decouples the security policy from the underlying object-level permissions. Even if someone accidentally grants SELECT access to the transactions_prod table to an intern, the masking UDF will still render the PII as ***-**-****. The data is protected by its metadata, not by the user's role.

Third, we automated our audit process. We use the Databricks system.access logs to monitor for any queries that trigger the masking function. If an unauthorized user attempts to query a column tagged with pii='high', an alert is sent to Slack. We don't wait for a compliance audit to tell us we're bleeding data; we see the attempts in real-time.

The biggest lesson here is that in complex financial environments, RBAC is a maintenance nightmare that scales linearly with chaos. When you add a new team, you have to update a dozen roles. When you add a new column, you have to update a dozen views.

ABAC forces you to classify your data once. Once the data is classified, the security policy is global and immutable. The developer can’t "forget" to include a column in a mask because the mask is tied to the column attribute, not the SQL view definition.

I’d rather spend three days setting up a robust tagging strategy than three hours explaining to a regulator why the PII is sitting in a text file on a sandbox cluster. If your security model requires you to remember to do something, it’s already broken. Move to attributes, let the catalog do the heavy lifting, and get some sleep.

Cover photo by Albert Stoynov on Unsplash.

Stop Z-ordering your tables unless you love wasting money

Aniket Abhishek Soni — Sun, 16 Aug 2026 14:24:26 +0000

Data teams waste roughly 30% of their compute spend on recurring OPTIMIZE jobs that move data around just to keep queries from crawling. If you are still running ZORDER BY on every high-cardinality column in your schema, you are essentially paying a "legacy tax" to the cloud provider for the privilege of manually managing data distribution.

Why I chose this topic: I spent three weeks troubleshooting a 4-hour OPTIMIZE job in a healthcare pipeline that was failing due to OOM errors, only to realize the distribution was static while our query patterns had drifted. We need to stop treating data layout like a manual maintenance chore and start treating it like a dynamic resource.

The common industry advice—that Z-ordering is the gold standard for multi-dimensional filtering—is dangerously outdated.

Why the common approach falls short

The "Z-order everything" dogma assumes two things that are rarely true in modern production environments: that your query patterns are static, and that you have the time to babysit your partitioning strategy.

When you run OPTIMIZE table_name ZORDER BY (user_id, event_date), you are baking a specific search pattern into the physical layout of your Parquet files. If a business analyst decides to start filtering by region_id or device_type next month, your Z-order becomes a liability. Your query engine will perform a full table scan because the data is physically organized for a query that no one is running anymore.

Furthermore, Z-ordering is a global operation. To maintain the Z-curve, you have to rewrite the entire partition. If you have 50TB of data, you aren't just "optimizing"; you are thrashing your storage layer. Every time you trigger that command, you are incurring a massive I/O penalty, inflating your cloud storage bill, and potentially locking up your tables for hours.

Photo by Logan Voss on Unsplash

The case for Liquid Clustering

Liquid Clustering, introduced in Databricks Runtime 13.3 LTS, is a fundamental shift in how we handle data layout. Unlike Z-ordering, which is a rigid, global operation, Liquid Clustering is a native, incremental approach to data organization.

When you define a table with CLUSTER BY (user_id, event_date), you aren't creating a static map. You are giving the Delta Lake engine permission to reorganize the data as it lands. It is a declarative, not imperative, approach.

In a recent production migration of a healthcare claims dataset, we moved from a weekly Z-order job to Liquid Clustering. The Z-order job cost us ~$450 in compute per run and took 210 minutes. With Liquid Clustering, the background maintenance is handled by the engine, keeping the data performant without the massive, monolithic "all-at-once" rewrite. We saw a 40% reduction in our monthly compute bill because we stopped rewriting files that didn't need touching.

The syntax is cleaner, but the real magic is under the hood. You can change your clustering columns at any time without needing to rewrite the entire history of the table.

-- The old way: The "hope it stays relevant" approach
OPTIMIZE claims_data ZORDER BY (patient_id, claim_date);

-- The new way: The "let the engine handle it" approach
ALTER TABLE claims_data CLUSTER BY (patient_id, claim_date);

Once you run that ALTER statement, new data written to the table is automatically clustered. Old data is reorganized lazily. You stop fighting the engine and start letting it work for you.

Measuring the success of your layout

If you aren't measuring your data layout efficiency, you’re just guessing. Most engineers look at query duration and call it a day. That’s a vanity metric.

To actually measure if your clustering is working, you need to look at Data Skipping Stats. You can check this by running DESCRIBE DETAIL table_name. Look at the minValues and maxValues in your file metadata. If you are filtering by user_id and the min/max ranges in your files are massive, your clustering is failing.

Beyond that, use the EXPLAIN command in Spark. If you see DataSkipping enabled but your scan_files count is still high, it means your files are not pruned effectively. I prefer to pull these metrics into a simple dashboard using the delta.history() command. If I see the number of files read consistently exceeding the number of files that should contain the target data, I know it’s time to re-evaluate the clustering keys.

Don't just watch the clock. Watch the I/O. If your bytes_read is significantly higher than the size of the result set, your physical layout is actively sabotaging your costs.

The objections (and my answers)

The pushback I hear most often is: "Liquid Clustering is Databricks-specific; I want to keep my data portable."

Fair enough. If you are running an open-source Delta Lake stack on your own managed Kubernetes cluster without the Databricks optimization layer, Liquid Clustering isn't available to you. But be honest—if you are building a production-grade data platform in 2024, are you really doing it to avoid vendor lock-in, or are you doing it because you haven't calculated the cost of maintaining your own infrastructure? The complexity of manual Z-ordering at scale is a "hidden cost" that far outweighs the cost of the platform.

The other objection is "lack of control." Engineers hate giving up the manual ZORDER knob because they feel like they lose precision. But here’s the truth: your manual "precision" is just an optimization for the query you wrote yesterday. The data lifecycle is dynamic. Your tables should be, too. If you think you know better than an engine that can analyze petabytes of access patterns in real-time, you’re likely overestimating your intuition.

Lastly, some argue that Liquid Clustering is "magic" and therefore unpredictable. I’ll take "predictably good enough" over "unpredictably perfect but expensive to maintain" any day of the week.

Conclusion

Stop treating your data layout as a permanent architectural decision. It’s a transient optimization that should evolve with your business requirements.

If you are dealing with tables that grow daily and query patterns that change monthly, Z-ordering is a dead end. It forces you into a cycle of expensive, global rewrites that provide diminishing returns. Liquid Clustering represents the shift toward infrastructure that actually manages itself.

Start by auditing your OPTIMIZE jobs. If you see a job that takes more than an hour to run and touches the same columns every single time, swap it for a CLUSTER BY configuration. You’ll save money, you’ll stop babysitting your pipelines, and your query performance will be more resilient to the inevitable changes in how your users interact with your data.

The era of manual data maintenance is over. Stop paying for it.

Cover photo by Ian Talmacs on Unsplash.

Your data pipeline isn't idempotent until you prove it breaks and recovers

Aniket Abhishek Soni — Fri, 14 Aug 2026 14:29:13 +0000

The "Exactly Once" myth is the most expensive fairy tale in data engineering. We tell ourselves that if we just buy a high-end orchestrator or use a specific streaming library, the system will magically handle retries without duplicating a single row. It won't. If you aren't designing for idempotency at the storage layer, you’re just waiting for a primary key violation to wake you up at 3 AM.

Why I chose this topic: I spent four years in healthcare fintech cleaning up "duplicate event" disasters caused by naive pipeline retries. I’m writing this because I’m tired of seeing engineers treat retry logic as an afterthought rather than a core requirement of every single data job.

It was 3:14 AM on a Tuesday. PagerDuty didn't just beep; it screamed. Our financial reconciliation pipeline, which processes millions of daily transactions, had failed. The dashboard showed a massive spike in 500 Internal Server Error responses from our downstream ledger API.

The Airflow UI was a sea of red. My first instinct—the one I’d been trained to do—was to hit "Clear" on the failed DAG tasks. I assumed the transient network blip had cleared, and hitting "Clear" would simply pick up where it left off. I hit the button. I went back to sleep. I was wrong.

What we saw

When I logged back in at 8:00 AM, the ledger was in total chaos. The reconciliation report showed a $12M discrepancy in our accounts payable. The logs were a mess of Duplicate Key exceptions.

We thought it was an API timeout issue. We spent three hours chasing the load balancer configuration, convinced that our max_retries setting of 3 was somehow causing a race condition in the API gateway. We checked the AWS X-Ray traces, we grepped the Nginx logs for upstream_response_time, and we even blamed the infrastructure team for a silent network partition.

Everything looked like a connectivity problem. Nothing looked like a logic problem. We were looking for a broken pipe, but the water was actually being pumped into the same bucket twice.

Photo by Henrique Ferreira on Unsplash

Root cause

The root cause was buried in a Python script that pushed processed batches to PostgreSQL. Our insert logic looked like this:

def load_data(batch):
    for record in batch:
        cursor.execute("INSERT INTO ledger_entries (tx_id, amount, status) VALUES (%s, %s, 'PENDING')", 
                       (record['id'], record['amount']))
    conn.commit()

The pipeline was configured with a retries: 3 and retry_delay: 300 in our Airflow task definition. When the network blip hit, the task failed mid-batch. But because the database connection didn't technically close immediately, some of those INSERT statements had already hit the wire and succeeded before the network interruption severed the connection.

When Airflow retried, it re-ran the entire task. It didn't know which records had already been inserted and which hadn't. It just blindly tried to insert the same tx_id values again. Our database had a PRIMARY KEY on tx_id, so the second attempt crashed. But on the third attempt, a different, partial set of records succeeded, creating a Swiss-cheese distribution of data in the production ledger.

We were relying on the "All or Nothing" promise of a transaction block, but our batching strategy was too large and our retry logic was too dumb.

Photo by Logan Voss on Unsplash

The fix

We stopped treating the entire batch as a single, fragile unit. Instead, we implemented a "Load-Stage-Merge" pattern using a staging table.

First, we changed the load process to move data into a temporary, un-logged staging table that shared the same schema as the production ledger.

-- Create temp table per execution
CREATE TEMP TABLE stage_ledger_entries (LIKE ledger_entries INCLUDING ALL) ON COMMIT DROP;

-- Bulk copy from S3 to stage
COPY stage_ledger_entries FROM 's3://bucket/data.csv' WITH (FORMAT csv);

-- Atomic UPSERT into production
INSERT INTO ledger_entries (tx_id, amount, status)
SELECT tx_id, amount, status FROM stage_ledger_entries
ON CONFLICT (tx_id) DO UPDATE SET
    status = EXCLUDED.status,
    amount = EXCLUDED.amount;

By using the ON CONFLICT clause (the UPSERT pattern), we made the operation inherently idempotent. Whether the job runs once, five times, or a hundred times, the final state of the ledger_entries table remains identical. If a task fails, we don't care how much data made it through before the crash. We just run the task again. The database handles the logic of ignoring duplicates or updating existing records.

What we changed so it never happens again

We stopped allowing "blind" retries. We now enforce a set of rules for every data pipeline we deploy.

First, we moved away from row-by-row inserts. Row-by-row is slow and makes partial failures almost impossible to untangle. We now use standard bulk loading tools like COPY (Postgres) or bcp (SQL Server), which are transactionally safer and significantly faster.

Second, we implemented "Versioned File Naming." Every output file in our S3 buckets is now suffixed with a hash of the content or a unique execution ID. If a pipeline runs, it writes to a specific path. If we retry, it writes to the same path, overwriting the previous partial attempt. This prevents the "junk data" problem where multiple failed runs litter your data lake with partial, corrupt files.

Third, we introduced an observability layer specifically for idempotency. We added a check in our CI/CD pipeline that scans for INSERT statements that lack an ON CONFLICT or a WHERE NOT EXISTS clause. If you’re writing an insert, you have to justify why it isn't idempotent. If you can’t, the build fails.

Finally, we adopted a "State-First" mindset. We treat our destination database as the source of truth for the job’s progress. Before any task starts, it queries the target table to see which tx_id values already exist. It then filters those out of the source batch. We basically turned the job into a self-pruning process.

You will have failures. The network will drop, the API will time out, and the power will flicker. Don’t build a system that needs human intervention to clean up the mess at 3 AM. Build a system that, when it wakes you up, allows you to say "just hit restart" and go back to sleep. If you can't restart your job without fear, you aren't doing engineering; you're doing crisis management.

Cover photo by Mark König on Unsplash.