DEV Community: Aniket Abhishek Soni

Don't Let Unity Catalog Migration Become Your Resume Generating Event

Aniket Abhishek Soni — Thu, 24 Sep 2026 14:29:20 +0000

Why I chose this topic: I spent three days last quarter fixing a "simple" metadata sync that locked 200 production jobs out of their own tables because of a misconfigured metastore credential. I’m writing this so you don’t have to drink the same lukewarm coffee at 3:00 AM that I did.

Two years ago, a junior engineer on my team ran a GRANT ALL script on the wrong Hive metastore schema. It took forty minutes for the monitoring alerts to fire, and by then, the ETL pipelines had already overwritten four months of regulatory reporting data with nulls. We spent a week restoring from snapshots and explaining to the CISO why our lineage tracking looked like a crime scene.

Migrating to Unity Catalog (UC) is the industry-standard fix for this chaos, but the migration itself is a minefield. You aren't just moving pointers; you are fundamentally changing how Spark identifies data. If you treat this like a simple "lift and shift," you’re going to wake up to a wall of TABLE_OR_VIEW_NOT_FOUND errors across your entire job fleet.

The real problem

The problem isn't the technology—it's the namespace. Hive metastore uses a two-level namespace: database.table. Unity Catalog introduces a three-level namespace: catalog.schema.table.

When you migrate, you aren't just changing a config; you are rewriting the identity of every single asset in your environment. Most teams try to do this by alias-shifting or partial migrations. Don't. You will end up with "split-brain" syndrome, where your legacy jobs are looking at the Hive metastore while your modern jobs are looking at UC, and your internal governance team is having a heart attack because they can’t find the lineage for 40% of the data.

Photo by Werzk Luuuuuuu on Unsplash

Step One: Inventory and namespace mapping

You cannot migrate what you cannot count. Do not start by moving tables. Start by dumping your existing Hive metastore into a CSV. You need a source-of-truth mapping file that links your legacy db.table to your new catalog.schema.table.

I use a simple PySpark script to iterate through the legacy catalog and generate a JSON manifest. If a table doesn't have a clear home in the new structure, delete it. If you have legacy junk, use this migration as the excuse to purge it.

# Extract current state to mapping file
databases = spark.catalog.listDatabases()
mapping = []

for db in databases:
    tables = spark.catalog.listTables(db.name)
    for t in tables:
        mapping.append({
            "legacy_name": f"{db.name}.{t.name}",
            "new_name": f"prod_catalog.{db.name}.{t.name}"
        })

import json
with open('migration_map.json', 'w') as f:
    json.dump(mapping, f)

Step Two: Testing the UC connectivity layer

Before you touch the production jobs, verify your cluster can actually talk to the UC-enabled workspace. Unity Catalog requires specific service principal permissions. If your cluster is still using Instance Profiles, you are going to hit an authentication wall.

Ensure your spark-conf includes the necessary authorization settings. If you’re using Databricks, verify that your cluster is running a runtime version that supports UC—anything below 11.3 LTS is a non-starter.

# Verify spark configuration for UC
spark.databricks.io.cache.enabled true
spark.sql.catalog.spark_catalog com.databricks.sql.managedcatalog.UnityCatalog
spark.databricks.acl.dfAcls.enabled true

If you don't set spark.sql.catalog.spark_catalog, your jobs will keep trying to default to the legacy Hive metastore, and you will spend hours debugging why your SELECT statements are returning empty results for tables you know exist.

Step Three: The phased transition with aliases

Don't do a "Big Bang" migration. Use a phased approach where you re-point the spark_catalog to UC, but keep a legacy catalog alias available for a fallback.

Modify your production job deployment pipeline to accept a CATALOG_NAME variable. If something goes sideways, you can flip the environment variable back to hive_metastore in the CI/CD pipeline and redeploy in seconds, rather than manually editing 200 jobs.

# CI/CD variable structure
jobs:
  etl_process:
    parameters:
      target_catalog: "prod_catalog"
    spark_conf:
      spark.sql.catalog.my_legacy_hive: "hive_metastore"

In your code, switch from SELECT * FROM db.table to SELECT * FROM {target_catalog}.db.table. This abstraction layer is the only thing that saved my team from a total outage during our migration.

Step Four: Validating the lineage and permissions

Once the jobs are running on UC, you need to verify that your IAM roles moved correctly. Hive metastore is notoriously lax; Unity Catalog is strict. A common failure mode is that your job has permission to read the data in S3/ADLS but lacks the USE CATALOG or USE SCHEMA permissions in UC.

Run this audit script immediately after your first successful job run. It checks if the principal actually has access to the underlying storage volume.

# Permission audit snippet
def check_access(catalog, schema, table):
    try:
        spark.sql(f"DESCRIBE TABLE {catalog}.{schema}.{table}").show()
        return True
    except Exception as e:
        print(f"FAILED ACCESS: {catalog}.{schema}.{table} - {e}")
        return False

Lessons learned from production

The "External Location" Trap: If you don't explicitly define your S3/ADLS storage paths as External Locations in UC, your jobs will fail with a cryptic PERMISSION_DENIED error, even if the underlying cloud IAM role is correct. You must grant CREATE EXTERNAL TABLE on the location to the metastore admin.
Views are not Tables: If you have complex Hive views, they will not migrate automatically. You have to recreate the DDL. If you try to copy the definition directly, you will hit issues with Hive-specific UDFs that don't exist in the UC namespace.
The Metadata Sync Lag: When you sync your legacy metastore to UC, there is a synchronization window. Do not run any writes during this window. I once had a job write data to the legacy path while the sync was moving the schema to UC, resulting in a dual-path write that corrupted the partition metadata.
Global Temp Views: If your jobs rely on GLOBAL_TEMP_VIEW, be aware that these are not automatically migrated. They are session-bound in the legacy metastore. You need to refactor these into temporary tables or persistent schema tables.

Conclusion

Migrating 200 production jobs is not a technical challenge; it is a discipline challenge. If you automate your mapping, abstract your catalog names, and test your permissions before you flip the switch, you can do this without downtime. If you try to hard-code your way through it, you’ll be the one explaining to management why the dashboard is blank on Monday morning.

Try it: Take your lowest-traffic, non-critical production job today. Refactor it to support a variable catalog parameter. If you can move that one job to Unity Catalog without breaking the output, you have a blueprint for the other 199.

Tags: #databricks #spark #engineering #migration

Cover photo by Tyler on Unsplash.

The Snowflake Iceberg Pivot: Why Your Data Warehouse Should Be a Storage Engine

Aniket Abhishek Soni — Tue, 22 Sep 2026 10:18:24 +0000

Ninety percent of the "performance" you pay for in Snowflake is actually just a tax on your inability to move your data. If you’re like most engineers I’ve worked with in fintech, you’ve convinced yourself that moving to an open table format like Iceberg means you have to go build a bespoke Spark/Trino stack on top of an S3 bucket and deal with the operational headache of catalog synchronization.

You are wrong. And that laziness is costing your company a fortune.

For years, we treated Snowflake as both the engine and the prison. We stored data in proprietary micro-partitions that were functionally invisible to every other tool in the stack. If you wanted to run a quick Python script to do some ML training, you had to clone the table, unload to CSV/Parquet, or pay for an expensive Snowflake compute cluster just to read a few gigabytes of rows.

Snowflake-managed Iceberg tables change the math. They allow you to maintain the convenience of a managed service while storing the actual data files—the Parquet files—in your own cloud storage. You aren't just saving on storage costs; you are effectively decoupling your state from your compute.

Why the common approach falls short

The "classic" Snowflake workflow is a black box. You CREATE TABLE and hope for the best. When you leave, you’re forced to perform an UNLOAD operation, which is essentially a massive export job that burns compute credits, messes with your time-travel metadata, and leaves you with a pile of flat files that lack the schema evolution capabilities your downstream apps need.

I’ve seen junior engineers try to "optimize" this by spinning up AWS Glue crawlers and manual Athena partitions. The failure mode is predictable: the Snowflake metadata and the Glue catalog drift apart within three weeks. You end up with a "source of truth" that isn't true, a bunch of broken dashboards, and a panicked Friday afternoon spent patching manifest files.

The common approach relies on the vendor’s proprietary binary format. If Snowflake goes down—or, more likely, if your bill hits a threshold where Finance starts asking questions—you are locked. You can't just point a Trino cluster at your S3 bucket and expect it to work. You need a migration strategy. With managed Iceberg tables, that migration strategy is just a standard SQL command.

Photo by Andris Gangis on Unsplash

The mechanics of the shift

Implementing this isn't a complex migration. It's a configuration change. You move from the default TABLE type to an ICEBERG table managed by Snowflake.

When you execute:

CREATE TABLE my_iceberg_table (
  id STRING,
  event_time TIMESTAMP,
  payload VARIANT
)
USING ICEBERG
CATALOG = 'SNOWFLAKE'
EXTERNAL_VOLUME = 'my_s3_external_volume'
BASE_LOCATION = 'my_iceberg_data';

You are essentially telling Snowflake: "I want you to handle the heavy lifting of metadata management, compaction, and schema evolution, but keep the actual Parquet files in my bucket."

The EXTERNAL_VOLUME is the key here. It defines the bridge between Snowflake and your S3/GCS bucket. Once this is set up, Snowflake acts as the "Engine," but the "Storage" lives in your VPC. If you decide to add a Trino cluster or a local DuckDB instance for quick testing, you just point them at the same S3 bucket. Because the table is using the Iceberg spec, your downstream engines understand the manifest files, the snapshot history, and the schema evolution without you needing to do a single EXPORT job.

Real-world failure modes and how to fix them

I’ve seen teams try to force this into every table they own. Don't do that. Managed Iceberg tables come with trade-offs.

First, the "Snowflake-managed" label means exactly that. If you manually delete a Parquet file from your S3 bucket, Snowflake’s catalog will get corrupted. You cannot treat your underlying S3 storage as a playground. You must use Snowflake’s ALTER TABLE ... REFRESH command to sync metadata if you’ve been messing with the files outside of the Snowflake engine.

Second, consider your write patterns. If you have a high-frequency streaming ingestion (think 50,000 inserts per second), Snowflake’s native format is still optimized for that. Iceberg introduces overhead because it has to generate new manifest files for every transaction. If you try to run massive, small-batch transactions against an Iceberg table, you will see performance degradation compared to standard Snowflake tables.

The strategy I use is "Tiered Persistence." My high-velocity staging tables remain in the proprietary format. Once the data reaches a state of "rest" (e.g., daily aggregates or cleaned facts), I move them into Iceberg managed tables for long-term storage and cross-engine accessibility.

Photo by David Trinks on Unsplash

The objections (and my answers)

The most common pushback I hear is, "But if I use Iceberg, I lose the proprietary Snowflake performance features like Search Optimization or Clustering Keys."

Yes, you do. And that's the point. If you need Search Optimization for a dataset, that dataset is likely part of your core product loop. You should pay the premium for that. But for 80% of the data sitting in your warehouse—the cold data, the audit logs, the historical facts—you aren't actually using those features. You’re paying for them because you didn't have a choice. Now you do.

Another objection: "It introduces complexity in IAM roles and bucket policies."

True, setting up the STORAGE INTEGRATION and the EXTERNAL VOLUME is more work than just running CREATE TABLE. You need to manage bucket policies, trust relationships, and VPC endpoints. But welcome to being a senior engineer. If you’re complaining about setting up an IAM role once to save your company 40% on their storage bill and gain absolute portability, you’re in the wrong seat.

Finally, some fear that Snowflake will "deprioritize" Iceberg performance to push users back to native tables. My read on the market is the opposite. Snowflake is terrified of the "Databricks effect." They know they have to play nice with open standards or they become the new Oracle. Supporting Iceberg as a first-class citizen is their way of keeping you from leaving entirely.

Conclusion

The goal of a senior engineer isn't to be a fanboy of a specific platform; it's to build a resilient system. Using Snowflake-managed Iceberg tables gives you the best of both worlds: the operational ease of a SaaS warehouse and the architectural sovereignty of open storage.

Stop paying for storage hostage-taking. Start moving your production tables to Iceberg, configure your external volumes, and give your organization the option to pivot when the economics stop making sense. You get to keep your tooling, your query history, and your dashboards, but you stop being an indentured servant to a single vendor's storage format.

In this business, control is the only currency that matters. If you can’t move your data without a massive, multi-week project, you don't own your data. Fix that today.

Cover photo by Alex Pudov on Unsplash.

Stop trying to make Airflow work for Medallion pipelines

Aniket Abhishek Soni — Mon, 21 Sep 2026 01:21:03 +0000

The "universal orchestrator" is a myth that keeps data engineers awake at 3:00 AM chasing zombie DAGs and orphaned Spark clusters. You’ve been told that if you just pick the right tool—usually the one with the most GitHub stars—your medallion architecture will magically materialize into a self-healing, performant pipeline.

I’ve spent six years cleaning up the aftermath of this delusion. I’ve seen healthcare pipelines stall because a Postgres metadata database hit its connection limit, and financial reconciliation jobs fail because a Lambda function timed out during a state transition. You don't need a "universal" tool; you need the right tool for the specific failure mode you’re willing to debug. Here is the field guide to keeping your bronze, silver, and gold layers moving without losing your mind.

1. Airflow is for people who love debugging Python environments

If your organization has a massive infrastructure team dedicated solely to keeping the Airflow scheduler alive, fine. Use it. But for the rest of us, Airflow is a dependency hell machine. Between pip conflicts in your worker nodes and the inherent fragility of the scheduler’s heartbeat, you are spending 40% of your time managing the orchestrator instead of the data.

In a medallion pipeline, you want atomicity. Airflow doesn’t give you that; it gives you a task-based graph that fails halfway through a Bronze-to-Silver merge, leaving you to write custom cleanup logic that inevitably fails as well.

# The classic Airflow "oops" - dependency bloat
from airflow.operators.python import PythonOperator
# Oh, your task needs pandas 1.5 but the DAG next door needs 2.0? 
# Good luck with your virtualenv hell.

Photo by Manuel Luikenga on Unsplash

2. Step Functions are the ultimate "Set and Forget" for AWS shops

If your Medallion architecture lives in S3 and interacts with Glue or EMR, AWS Step Functions are the only orchestrator that doesn't feel like a side project. You aren't managing a server; you're managing a state machine. The beauty here is Wait for Callback. When I trigger a long-running Spark job, the state machine enters a paused state until the job reports back via an API call.

Failure mode: You will eventually hit the execution history limit. If you have a pipeline that iterates through 10,000 files, you will blow past the 25,000-event limit for a single execution. Chunk your data, or you’ll be staring at a cryptic ExecutionLimitExceeded error in the middle of a Friday deployment.

{
  "Type": "Task",
  "Resource": "arn:aws:states:::elasticmapreduce:addStep.sync",
  "Parameters": {
    "ClusterId.$": "$.ClusterId",
    "Step": { ... }
  },
  "Retry": [{
    "ErrorEquals": ["States.TaskFailed"],
    "IntervalSeconds": 30,
    "MaxAttempts": 3
  }]
}

3. Databricks Workflows is the only "Native" choice

If you are running a Medallion pipeline on Delta Lake, stop using an external orchestrator. Databricks Workflows (the Jobs API) is built for this. It handles the cluster lifecycle, the notebook/JAR execution, and—most importantly—the underlying Delta commits.

When you use an external tool like Airflow, you are disconnected from the cluster's health. When you use Databricks Workflows, the orchestration is aware of the cluster status. If a node goes down, the job retries efficiently. You avoid the "orphan cluster" problem where your orchestrator thinks a job succeeded, but the cluster died during the final VACUUM command.

4. Don't build "God DAGs"

The biggest mistake I see in medallion architectures is the monolithic pipeline. A single DAG that runs Bronze, Silver, and Gold in one long serial chain is a ticking time bomb. If your Gold transformation fails, you have to restart the whole damn thing, effectively re-processing Bronze and Silver again.

Use decoupled triggers. Bronze completes, it emits a FileArrived event (or a Databricks Job completion signal), and that triggers Silver. Use a simple event-driven model. If Silver fails, you fix the logic and trigger Silver again. You don't touch the Bronze layer.

5. The Metadata Store is your source of truth, not the Orchestrator

Never use your orchestrator’s database to track your data quality. I’ve seen people write custom Airflow variables to track last_processed_timestamp. This is a disaster waiting to happen when you need to backfill or re-run a day of data.

Instead, keep your state in the data itself. Use a _metadata table in your Delta lake.

-- Pattern: Watermarking in the Medallion layer
SELECT * FROM bronze.events 
WHERE ingestion_timestamp > (
  SELECT MAX(processed_until) FROM silver.watermark_table
);

By keeping the state in the table, the orchestrator becomes "dumb." It just says "Run the job," and the job asks the metadata table "Where did we leave off?" This makes your pipeline orchestrator-agnostic. You could rip out Airflow and replace it with a Cron job, and your data wouldn't care.

Photo by Fahmi Anwar on Unsplash

6. Observability is not "Success/Failure"

Stop measuring success by the green checkmark in the UI. A job can finish "successfully" while loading empty data or corrupting a partition. Your orchestrator needs to gate-keep based on Data Quality (DQ) thresholds.

In Databricks, use dbt or Great Expectations as a task in your workflow. If the DQ check fails, the job fails. Don't let the pipeline proceed to the Gold layer if the Silver layer has null IDs.

# Databricks Workflows JSON config
tasks:
  - task_key: "dq_check"
    notebook_task:
      notebook_path: "/Tests/Silver_Validation"
  - task_key: "gold_load"
    depends_on:
      - task_key: "dq_check" # This is your primary circuit breaker

Conclusion

The orchestrator you choose matters less than how you decouple your logic. If you are deeply invested in Databricks, use their native Workflows—it’s the path of least resistance. If you are doing multi-cloud, multi-service orchestration, use Step Functions for their durability. If you are in a massive Python-heavy org that already has an Airflow platform team, stay there, but keep your pipelines granular and stateless.

The real question isn't "Which tool is best?" but rather: how much of your pipeline logic is stuck in your orchestrator, and how fast can you delete it when the tool inevitably goes out of fashion? Are you building a data platform, or are you just building a very expensive Airflow configuration file?

Cover photo by K. Mitch Hodge on Unsplash.

How Databricks Serverless Compute Cost My Team $14k in One Weekend

Aniket Abhishek Soni — Mon, 21 Sep 2026 01:20:58 +0000

It’s Sunday, 2:14 AM. The PagerDuty alert hits my phone with that specific, jarring frequency that makes your stomach drop before you’ve even opened your eyes. My Databricks billing alert wasn’t a standard "usage threshold reached" notification; it was the "you’ve hit 80% of your monthly cloud spend in 48 hours" panic text.

I sat up, opened the Databricks console, and stared at the Billing page. Our sql_warehouse_prod_v2 was burning DBU (Databricks Units) like it was a crypto-mining operation. We had shipped a new pipeline on Friday, it passed CI, the integration tests looked green, and the data landed on time. But while we were sleeping, the bill was growing faster than our debt in college.

What we saw

The symptom was simple: our bill went vertical. The dashboard showed a steady, flat line for the past three months, followed by a spike that looked like the edge of a cliff.

My first assumption was a runaway loop in a Python job. I checked the cluster logs for dbr 13.3 LTS. Nothing. Then I checked the spark_query_history. Nothing out of the ordinary—just the standard daily ingestion patterns.

The false lead was the "Auto-stop" setting. I looked at the UI for the Serverless SQL Warehouse and saw Auto-stop: 10 minutes. In my head, this was bulletproof. If the warehouse isn't doing anything, it shuts down. The math seemed solid: 10 minutes of idle time is negligible.

But I was looking at the wrong metric. I was looking at the cluster state, not the session state. The warehouse wasn't idling; it was being kept alive by a ghost.

Photo by Kisetsu Co on Unsplash

Root cause

The culprit was a hidden interaction between our BI tool (a standard Tableau integration) and the Databricks Serverless SQL Warehouse. We had migrated to Serverless because we wanted the "instant-on" experience.

In a traditional cluster, the warehouse would have eventually hit a resource contention limit or a timeout. But Serverless is designed to stay available. We had a dashboard connection string configured with Catalog and Schema settings that were hitting a system.information_schema query on a heartbeat interval.

Because we used a service principal with broad CAN USE permissions, the connection remained active. The Databricks Serverless SQL Warehouse interpreted these heartbeat pings as "active queries."

Crucially, the Auto-stop setting only triggers when the warehouse is truly idle. Because the heartbeat hit the SQL warehouse every 8 minutes, the 10-minute timer reset itself into infinity. The warehouse never hit its idle threshold. It wasn't "stuck" in a loop; it was being held hostage by a silent, low-latency heartbeat that didn't even show up in our main performance monitoring because it was sub-millisecond.

We were paying for a "Large" size warehouse—which runs at a significantly higher DBU rate—to serve a heartbeat ping that could have been handled by a "Starter" size or, better yet, a cached metadata call.

Photo by m. on Unsplash

The fix

The immediate fix was to kill the connection from the BI tool side and force the warehouse to scale down. I manually set the Auto-stop to 1 minute to ensure it would die instantly once the connection was severed.

Then, we had to redefine the warehouse configuration. We abandoned the single large warehouse for BI and split the workloads. We created a "Serverless-Small" for the heartbeat-heavy dashboarding and kept the "Large" warehouse strictly for ad-hoc analyst queries and heavy ELT transformations.

We also updated our connection string in the BI tool to point to a specific Unity Catalog schema that didn't require the broad information_schema scanning that was triggering our phantom queries.

Finally, I implemented a Tag policy. Databricks allows you to add custom_tags to your SQL warehouses. I added CostCenter: Finance and Owner: DataEng. While this didn't stop the spending, it allowed me to isolate the DBU consumption at the warehouse level in the billing export CSVs within minutes, rather than waiting for the bill to aggregate.

What we changed so it never happens again

We stopped trusting the "Serverless is magic" marketing. Serverless means you don't manage the nodes, but you absolutely have to manage the session lifecycle.

First, we implemented a strict "Warehouse Sizing" policy. No production warehouse is allowed to be larger than "Medium" unless it has an explicit, documented exemption in the Terraform repo. If you want a "Large" or "X-Large" warehouse, you have to open a PR, and the CI pipeline runs a cost-estimate check using the Databricks Billing API to flag the daily run-rate.

Second, we moved away from generic service principals for BI tools. We now use scoped service principals with READ ONLY access to specific schemas. This prevents the BI tool from querying the wider information_schema or system catalogs that trigger those hidden, expensive backend processes.

Third, we set up a "Budget Alarm" using a Lambda function that polls the Databricks billing data every 6 hours. If the daily burn rate exceeds a 20% variance from the 7-day rolling average, it fires a high-priority alert into our Slack #ops-alerts channel.

Lastly, I learned to never, ever set an Auto-stop to anything longer than 5 minutes for non-critical workloads. In the world of Serverless, that 5-minute window is the difference between a productive team and a very uncomfortable conversation with your CFO.

You ship the code, the tests pass, and the data is correct. That’s the easy part. The hard part is ensuring that the infrastructure—the invisible, elastic, "serverless" part—doesn't treat your credit card like a bottomless well. Keep your warehouses small, your permissions tight, and your heartbeats monitored.

Cover photo by Kevin Ache on Unsplash.

Is Your Data Lake Actually A Landfill?

Aniket Abhishek Soni — Wed, 16 Sep 2026 10:25:13 +0000

Why I chose this topic: I spent three weeks debugging a "simple" read query that took four minutes to return ten rows because the metadata layer had ballooned to 200,000 snapshots. I’m writing this so you don’t have to explain to your CTO why the data platform is hemorrhaging AWS credits.

You ship the job. It passes CI. Your dbt tests are green, and the data lands in S3 right on time. Then, three weeks later, your Spark jobs start failing with java.lang.OutOfMemoryError during metadata initialization, and your query latency looks like a jagged mountain range.

You aren’t experiencing a "data growth" problem. You are experiencing a "neglect" problem. Apache Iceberg is elegant, but it isn’t magic. If you treat it like a traditional Hive table—set and forget—you are effectively building a digital landfill.

Every time you commit a transaction, Iceberg creates a new snapshot. Every time you overwrite a partition, you leave behind orphaned files in your object storage. Left unchecked, the metadata layer becomes so heavy that your catalog service will choke, and your cloud provider will send you a bill that makes you wish you’d stayed in the finance sector.

The real problem

The problem isn't the data volume itself; it’s the fragmentation and the metadata overhead. When you perform frequent streaming writes or micro-batching, you create hundreds of tiny files. These files are the death of performance. Your query engine’s planner has to do more work just to resolve the table state than it does to actually scan the data.

Furthermore, Iceberg doesn't automatically delete the data you tell it to "remove." It marks it as expired in the metadata. If you don't prune those snapshots and physically delete the files, you are paying for storage that no longer exists in your current view of reality. It’s a classic "zombie file" scenario.

Photo by Anton Acosta on Unsplash

Step: Compacting the small file disaster

Small files are the primary reason your S3 LIST calls take forever. Use Spark’s rewriteDataFiles procedure. Don't run this on every single commit unless you want to crash your cluster. Schedule it.

I prefer the bin-pack strategy for general purpose, but use sort if you have high-cardinality filters in your WHERE clauses.

CALL catalog.system.rewrite_data_files(
  table => 'db.my_table',
  strategy => 'sort',
  sort_order => 'event_timestamp DESC',
  options => map(
    'min-input-files', '5',
    'max-concurrent-file-group-rewrites', '10'
  )
)

If you’re running this on EMR or Databricks, monitor the rewrite-data-files job memory. If it fails with OOM, increase your executor memory; don't just reduce the number of files. You want fewer, larger files (aim for 128MB to 512MB), not more, smaller ones.

Step: Expiring snapshots with a death clock

Metadata accumulates linearly. If you are doing 50 writes a day, you have 18,000+ snapshots a year. Your catalog (Glue, Nessie, or Postgres) will eventually hit a latency wall trying to load the table state.

You need a strict retention policy. I typically default to a 7-day retention period. If you need to "time travel" further back than a week, you’re doing audit logs wrong—keep those in a separate, immutable table.

CALL catalog.system.expire_snapshots(
  table => 'db.my_table',
  older_than => timestamp '2023-10-20 00:00:00.000',
  retain_last => 5
)

Note the retain_last => 5. This ensures that even if you accidentally drop the table or run an aggressive expiry, you always have a safety net of the most recent commits. Never set this to zero unless you are decommissioning the table.

Step: Cleaning up the orphans

expire_snapshots removes the reference, but the physical files might still exist in S3 if you don't run the cleanup process. If you have "orphaned" files—files that exist in the storage layer but aren't in the metadata—you’re burning money.

Use remove_orphan_files. Be careful with the older_than parameter. If you run this while a long-running read query is still scanning the table, you might delete files that the reader is currently trying to access, leading to a FileNotFoundException.

CALL catalog.system.remove_orphan_files(
  table => 'db.my_table',
  older_than => current_timestamp - interval '3' days
)

I always leave a 72-hour buffer here. It’s safer to pay for three days of ghost storage than to have a production job fail because a partition was mid-read when the janitor job kicked in.

Photo by Annie Spratt on Unsplash

Lessons learned from production

Glue Catalog Throttling: If you are using AWS Glue as your Iceberg catalog, frequent maintenance triggers will hit the Glue API limits. Increase your Glue throughput quotas before you scale your maintenance jobs, or expect constant 400/500 errors.
The write.format.default trap: If you are migrating from Parquet to Iceberg, ensure your maintenance jobs aren't silently converting back to Parquet via session defaults. Always verify the output format in your DESCRIBE DETAIL output.
Maintenance shouldn't be "on-write": Everyone wants to set write.metadata.delete-after-commit.enabled = true. Don't. It makes your ingestion jobs brittle. Decouple maintenance into a separate, scheduled Airflow DAG.
Monitoring is non-negotiable: If your maintenance job fails, it doesn't alert you by default unless you hook it into your logging stack. Pipe the Spark logs from these maintenance procedures into Datadog/CloudWatch. A silent failure here is a ticking time bomb for your storage costs.
Know your partition evolution: If you change your partitioning scheme, your compaction strategy must change. Compacting across old and new partition specs can be computationally expensive. Read the docs on rewrite_position_delete_files if you’re doing heavy deletions or updates.

Conclusion

Iceberg is a powerful tool, but it is not a "set-and-forget" database. It is a complex distributed file system management task disguised as a SQL table. If you don't treat your maintenance jobs with the same rigor as your ingestion pipelines, you’re not an engineer—you’re just a temporary occupant waiting for the system to collapse under its own weight.

Try it: Run SELECT count(*) FROM "db.my_table.snapshots" in your query engine today. If the number is in the thousands, stop what you are doing, write a cleanup script, and schedule it immediately. Your cloud bill will thank you.

Cover photo by Bodega on Unsplash.

Why are you still debugging Spark on your host machine?

Aniket Abhishek Soni — Mon, 14 Sep 2026 22:54:06 +0000

Last October, a "minor" dependency bump in a PySpark job cost us four hours of downtime and roughly $12,000 in cloud compute credits. A developer had tested a new UDF locally using an older version of delta-spark than what we ran on our EMR cluster. Locally, the serialization worked fine. In production, the different Scala/Java versions in the underlying runtime caused a java.io.NotSerializableException that only manifested when the data volume hit a specific shuffle threshold.

We spent hours tailing logs and staring at obscure stack traces while the pipeline backed up. I swore then that if a dev’s laptop didn't look exactly like the cluster, they weren't allowed to ship.

You are currently deciding between two paths: staying in the "my-machine-is-special" hell of managing local Java/Scala/Python versions, or biting the bullet to containerize your local development environment. You think the latter is overkill. You think it's too much configuration overhead. You're wrong.

The contenders

You have two real options for local Spark/Delta Lake development.

The first is the "Host-Native" approach: You install openjdk@11, python 3.9, spark 3.3.2, and delta-spark 2.2.0 directly on your macOS or Ubuntu machine. You use pyenv and sdkman to try and mimic the cluster. It feels fast, but it’s a lie.

The second is the "Containerized Replica" approach: You build a Dockerfile that mirrors your base image (likely amazoncorretto:11 or a specific Spark-provided image) and mount your code into a container. You use docker-compose to spin up a local MinIO instance to act as your S3-compatible storage for Delta tables. You are essentially carrying a mini-cluster in a container.

Photo by Karl Paul Baldacchino on Unsplash

The burden of parity

In the Host-Native camp, the ops burden is invisible until it isn't. You spend 30 minutes every few weeks "syncing" your local versions. You will eventually run into a mismatch between your local pyspark package and the spark-submit environment. When your local PySpark uses a different py4j version than the driver, you get weird, non-deterministic failures.

With the Containerized approach, the burden is front-loaded. You spend two hours writing a Dockerfile once. You define the SPARK_HOME, the HADOOP_CONF_DIR, and the AWS_ACCESS_KEY_ID for your local MinIO. Now, when you run docker-compose up, you know that if it runs on your machine, it runs in EMR or Databricks. The ops burden shifts from "fumbling with paths" to "writing one clean Dockerfile."

Failure modes and debugging

If you’re running locally on your host, you are prone to the "Global Namespace" problem. You’ve likely got a dozen versions of Java installed. Maybe a global ~/.ivy2 cache is corrupted. When a Spark job fails, you never know if it’s your code, a library mismatch, or a local environment setting.

In a container, you have a clean slate. If the job fails, you can exec into the container and inspect the /opt/spark/work directory, check the environment variables with env, and confirm the exact CLASSPATH. If you need to debug a Delta Lake write failure, you can inspect the _delta_log files directly in your mounted local folder. You aren't guessing; you're observing.

Photo by Štefan Štefančík on Unsplash

The cost of velocity

People argue that Docker slows down the inner loop of development. They’re usually doing it wrong. Yes, building a 2GB container image takes time. But you don't rebuild the image every time you change a line of code.

You write a docker-compose.yml that mounts your source code directory as a volume. You run your Spark job in the container. When you hit save in your IDE, the change is reflected inside the container instantly. You get the portability of the production environment with the speed of local execution. The "cost" is effectively zero after the initial setup. Compare that to the cost of one failed production deploy because a local environment was "fast and easy."

What I'd pick, and why

I’d pick the containerized approach every single time.

My recommendation: Use a multi-stage Dockerfile. In the first stage, install your build dependencies. In the final stage, use a slim JRE (like eclipse-temurin:11-jre-focal) to keep the image size manageable. Use docker-compose to link your service to a MinIO container.

Set your DELTA_SPARK_VERSION and SPARK_VERSION as build arguments (ARG) in your Dockerfile. When the platform team updates the production cluster, you change two lines in your .env file, run docker-compose build --no-cache, and you’re synced.

The caveat: This is not for beginners. If you don't understand how spark.driver.host or Hadoop filesystem configurations work, Docker will be a black box that frustrates you. You need to understand the network bridge between your host and the container. You need to understand how to map ports so your local Spark UI (usually port 4040) is actually accessible from your browser.

However, if you're working on production financial or healthcare data, "it works on my machine" is a fireable offense. Containerize it, or keep paying the bill when your pipeline dies at 3 AM.

Tags: #spark #docker #data #engineering

Cover photo by Domaintechnik on Unsplash.

Is your Data Lakehouse actually a HIPAA liability?

Aniket Abhishek Soni — Sat, 12 Sep 2026 05:36:55 +0000

Ninety-two percent of healthcare data breaches occur because of misconfigurations in the storage layer, yet most engineers I interview still think "encryption at rest" is the end of the conversation. If you think your S3 bucket policy is enough to satisfy an auditor, you’re about to have a very bad quarter.

In the world of HIPAA-compliant data engineering, the "security by obscurity" mindset is not just lazy; it’s a legal hazard. We treat PHI (Protected Health Information) like a radioactive isotope, yet we insist on dumping it into the same raw landing zones as our clickstream data and log files.

Stop doing that.

Why the common approach falls short

The industry-standard failure mode is the "God-mode service account." You know the one—the generic data-eng-prod role that has read/write access to s3://company-datalake/raw/. When a junior engineer accidentally writes a Spark job that spills unmasked patient names into a development environment, the audit trail shows the service account did it, but it provides zero granularity on who triggered the job or why.

Most teams try to fix this by masking data at the BI layer (Looker, Tableau). That’s a mistake. If your data lakehouse allows a data scientist to run a SELECT * on the raw table and get back unmasked PHI because they have read permissions on the bucket, you have failed the "Minimum Necessary" standard of HIPAA. You are trusting your downstream application to be the gatekeeper. That is not a security architecture; that is a prayer.

Photo by Laura Rivera on Unsplash

Hard-partitioning at the ingestion boundary

If you want to survive an audit, you need to stop thinking about data as a single bucket. You need to treat your landing zone as a triage center.

I implement a "Zero-PHI-Landing" policy. When data arrives from a source system (like a FHIR API or an EMR dump), it hits an ingestion Lambda or a Fargate task that immediately inspects the schema. If a field is tagged as PHI, it gets routed through a deterministic masking or tokenization service before it ever touches the persistent storage layer.

Here is what that looks like in a real Spark implementation. Don’t use generic udfs; use Column-level security via Delta Lake’s GRANT syntax.

-- The only way to handle PHI is to restrict the column itself
-- Do not rely on view-based security, which is easily bypassed by anyone with table access
CREATE TABLE patient_records (
    id STRING,
    name STRING,
    dob DATE,
    ssn STRING MASKED WITH (FUNCTION 'mask_ssn')
);

By binding the masking logic to the schema definition, you move the security responsibility from the developer’s intent to the storage engine’s configuration. If the developer doesn't have the UNMASK privilege—which should be restricted to a single, audit-logged service principal—they simply cannot see the data.

Immutable audit trails and the identity problem

The biggest gap I see in production pipelines is the lack of identity propagation. When you run a Spark job on EMR or Databricks, the cluster usually runs as a single IAM role. If you have 50 engineers with access to that cluster, you have zero accountability.

You need to implement fine-grained access control (FGAC) that maps the user identity to the storage layer. If you are on Databricks, stop using shared clusters for production. Use "Single User" mode or Unity Catalog with Attribute-Based Access Control (ABAC).

Tag your data. If a table contains PHI, tag it PII=True in your metadata catalog. Then, write a policy that enforces:

Access to PII=True resources requires MFA.
Every SELECT statement on a PII=True table must be logged to an immutable S3 bucket with Object Lock enabled.

Here is a snippet of a Terraform policy for your metadata store that actually keeps auditors happy:

resource "aws_iam_policy" "phi_access_policy" {
  name = "phi-access-restriction"
  policy = jsonencode({
    Version = "2012-10-17"
    Statement = [
      {
        Action   = ["lakeformation:GetDataAccess"]
        Effect   = "Deny"
        Resource = "*"
        Condition = {
          StringNotLike = {
            "aws:PrincipalTag/Role": ["data-steward", "compliance-officer"]
          }
        }
      }
    ]
  })
}

This prevents accidental exposure. Even if an engineer manages to gain access to the raw files, the Lake Formation layer will intercept the call and deny it because their IAM tag doesn't match the required security clearance.

Photo by Sasun Bughdaryan on Unsplash

The objections (and my answers)

"But this kills developer velocity."

Yes, it does. That is the point. HIPAA compliance is not supposed to be "fast." If you prioritize the speed of a data scientist over the privacy of a patient, you are in the wrong industry. You can build a sandbox environment with synthetic, anonymized data for experimentation. If they need real PHI, they file a ticket, the data steward reviews the request, and the access is granted for a limited duration.

"We use encryption at rest, isn't that enough?"

Encryption at rest protects you if the physical hard drive is stolen from a data center. It does absolutely nothing when someone with legitimate internal network access writes a SELECT * and exports the patient database to a CSV. Encryption is the floor, not the ceiling. If you are relying solely on KMS keys, you are missing the entire point of the HIPAA Privacy Rule.

"The cloud provider handles this for us."

The cloud provider handles the infrastructure security. They are not responsible for your data governance. If you grant s3:* to your production role, AWS will happily help you leak 10 million patient records. The "Shared Responsibility Model" is not a suggestion; it’s the legal boundary where your liability begins and the provider's ends.

Conclusion

Building a HIPAA-compliant data lakehouse isn't about buying the right tool; it's about enforcing a strict hierarchy of access. Move the logic as close to the storage as possible, use deterministic masking that is tied to the schema, and stop sharing credentials.

If your pipeline doesn't break when a user tries to access data they shouldn't, you haven't built a security boundary—you’ve built a data leak waiting to happen. The audit isn't a formality; it’s a stress test. If your current architecture allows a developer to bypass your masking logic with a simple DROP VIEW command, you’re not compliant. You’re just lucky.

Cover photo by Tyler on Unsplash.

Why our multi-million row join cost $4,000 in one hour

Aniket Abhishek Soni — Fri, 11 Sep 2026 13:47:27 +0000

You ship the job. It passes CI. The data quality checks return green. Then, at 3:14 AM, the PagerDuty alert fires. Your GCP billing dashboard is currently reporting a "spend anomaly," and your Databricks SQL Warehouse is throwing an OUT_OF_MEMORY error that is currently cascading into a service-wide outage.

I’ve been here. Twice. Once on BigQuery, once on Databricks. They aren't the same beast, and treating them as interchangeable "SQL engines" is exactly how you end up explaining a $4,000 hourly burn rate to a VP who doesn't care about your partition pruning strategy.

What we saw

It started with a simple request: join our 4TB events table with a 500GB user_metadata table. In our staging environment, which used a 10% sample of the data, the query finished in 42 seconds. In production, it didn't finish. It hung for 45 minutes, then the BigQuery slot usage spiked to 10,000+ slots, and the Databricks SQL Warehouse (a 2XL Serverless instance) simply evaporated, dumping a java.lang.OutOfMemoryError: Java heap space into the logs.

The false lead was the index. My junior dev thought we missed a partition key on user_id. We spent two hours adding a CLUSTER BY on the BigQuery side and a Z-Order on the Databricks side. It changed absolutely nothing. The execution plan remained a disaster. The symptoms—massive latency and OOM errors—weren't about data locality. They were about the engine trying to perform a broadcast join on a table that was too damn big to fit in a single worker's memory.

Photo by Salah Ait Mokhtar on Unsplash

Root cause

In BigQuery, the culprit is almost always "Shuffling." When BigQuery can't perform a map-side join, it triggers a massive shuffle phase. I saw the job details: Bytes shuffled: 12.4 TB. BigQuery charges for data processed, but when you hit a join that forces a massive shuffle, you aren't just paying for the read; you're paying for the compute slots spinning in circles trying to sort and move data across the network fabric.

On Databricks, the mechanism is different but equally lethal. We were using a Databricks SQL Warehouse with AUTO join optimization. The Catalyst optimizer looked at our 500GB metadata table and incorrectly estimated it could fit in the memory of a single node for a broadcast join. It couldn't. The executor tried to pull the entire 500GB into memory, hit the heap limit, and triggered a hard OOM. The spark.sql.autoBroadcastJoinThreshold default is 10MB, but our specific configuration had been bumped to 1GB by a previous team trying to "speed up" smaller queries. It was a ticking time bomb.

Photo by Gorilla ROI Data Connector on Unsplash

The fix

We stopped trying to outsmart the optimizers and forced the physical plan.

For BigQuery, we implemented a sub-query re-write. Instead of a standard JOIN, we used a JOIN with a WHERE clause that explicitly limited the join keys to a pre-filtered subset of the data. We also switched from SELECT * to specific columns—a boring, manual fix that reduced our shuffle volume from 12TB down to 800GB. We also had to switch the job to INTERACTIVE priority; the BATCH queue was holding onto slots for too long, causing a resource deadlock.

For Databricks, we had to disable the broadcast join for that specific query. We added the hint /*+ MERGE(events, user_metadata) */ to the SQL. This forced the engine to use a Shuffle Sort-Merge Join, which is much slower than a broadcast join but significantly more stable. It spilled to disk instead of exploding the heap. The query took 12 minutes instead of failing, but it finished. I'll take a 12-minute success over a $4,000 failed job any day.

What we changed so it never happens again

First, we implemented a "Query Size Budget." We now have a post-commit hook in our CI pipeline that runs EXPLAIN on every SQL file. If the estimated bytes processed in BigQuery exceed 1TB, or if the Databricks plan shows a BroadcastExchange on a table larger than 500MB, the PR is automatically blocked. You don't get to deploy code that could bankrupt the department.

Second, we stopped using AUTO settings for warehouse sizing. We now use specific warehouse tags for specific workloads. If a job is heavy on joins, it runs on a warehouse with enable_serverless_compute = true but with min_serverless_worker_count set to a higher floor. We treat the warehouse like a dedicated resource, not a magic bucket.

Third, we enforced a culture of looking at the "Query Profile." In BigQuery, that means checking the "Wait time" and "Shuffle stage" metrics in the UI. In Databricks, it means looking at the "Metrics" tab in the SQL query history to see how much data was spilled to disk. If you see "Spilled to disk," you haven't failed; you've just found the physical limit of your cluster.

Finally, I stopped trusting the "Big Data" marketing. BigQuery and Databricks are incredible tools, but they assume you know how a distributed join works. If you treat them like a MySQL instance on a laptop, they will treat your credit card like an unlimited ATM. Don't look at the execution time. Look at the data movement. Data movement is the tax you pay for poor query design. Keep the data local, keep the shuffles minimal, and for the love of god, keep an eye on your broadcast thresholds.

Cover photo by Albert Stoynov on Unsplash.

How I stopped fearing the 3 AM pager by forcing idempotency everywhere

Aniket Abhishek Soni — Tue, 08 Sep 2026 06:48:56 +0000

If your pipeline isn't idempotent, it isn't production-ready; it’s just a fragile script waiting to ruin your weekend. Most engineers treat "idempotency" as an academic concept for distributed systems papers, but in the trenches of fintech and healthcare, it’s the difference between a minor blip and a regulatory filing. If you can’t run your job five times in a row with the exact same input and get the exact same state, you aren't doing data engineering—you're doing data gambling.

I’ve spent six years cleaning up the messes left by "append-only" thinking. I’ve seen millions of dollars in duplicate ACH transactions and patient records corrupted by "just one more retry" logic. This guide covers the patterns I use to make sure that when the scheduler kicks off at 3 AM, I can sleep through the alarm because the system knows how to fix itself.

1. Stop relying on "Append" mode

The biggest sin in data engineering is assuming that your destination table is a clean slate. When a job fails halfway through, you don't want a partial load sitting in your production warehouse. Never use INSERT INTO blindly. If you are using BigQuery, Snowflake, or Databricks, use MERGE or overwrite-on-partition.

If you are using SQL-based ELT, write your transformations to stage data in a transient table before swapping it into production. Never push directly to the target.

-- The wrong way: INSERT INTO target_table SELECT * FROM staging
-- The right way: Use an atomic swap or a MERGE statement
MERGE INTO production.transactions AS T
USING staging.transactions AS S
ON T.transaction_id = S.transaction_id
WHEN MATCHED THEN UPDATE SET T.amount = S.amount, T.status = S.status
WHEN NOT MATCHED THEN INSERT (transaction_id, amount, status) VALUES (S.transaction_id, S.amount, S.status);

Photo by 🇻🇪 Jose G. Ortega Castro 🇲🇽 on Unsplash

2. Partitioning is your safety net

If your pipeline runs daily, your data must be partitioned by that day. If you are loading data without a partition key (or worse, appending to a single massive table), you are doomed. By using partition overwrites, you turn a complex "delete and re-insert" logic into a simple atomic operation.

In Spark, this is trivial. If your job fails, you just re-run the whole partition. If you overwrite the partition, you guarantee the state is consistent.

# Spark example for overwriting a specific partition
df.write \
  .mode("overwrite") \
  .partitionBy("event_date") \
  .option("partitionOverwriteMode", "dynamic") \
  .saveAsTable("prod.daily_metrics")

3. Deterministic execution IDs

When debugging a failure at 3 AM, you need to know exactly which run produced which record. I add a job_run_id and an ingestion_timestamp to every row. This allows me to query the state of the world at any specific point in time.

More importantly, if you use a surrogate key, generate it using a hash of the natural keys rather than an auto-incrementing integer. If you re-run the job, the hash stays the same, ensuring that the row is updated rather than duplicated.

import hashlib

def generate_id(data_dict):
    # Deterministic ID based on business logic
    seed = f"{data_dict['user_id']}-{data_dict['event_date']}"
    return hashlib.md5(seed.encode()).hexdigest()

4. The "No Side Effects" rule

A pipeline job should only do one thing: move data from A to B. If your data pipeline is also sending Slack alerts, updating a cache, or calling an external API, you have a problem. Side effects are not idempotent.

If you must trigger an external action, do it downstream using a "state machine" pattern. Write the data first, then have a separate observer process look for the completion marker and trigger the side effect. If the observer fails, you can re-run it safely because it only checks the existence of the final record.

5. Embrace the "Delete-Before-Insert" pattern

Sometimes MERGE is too slow or complex for your specific warehouse engine. In those cases, don't over-engineer. Use a transaction block to delete the data for your target time range before inserting the new batch.

In Postgres or Snowflake, wrap this in a transaction. If the insert fails, the delete rolls back. The system is back to its original state, ready for another attempt.

BEGIN;
DELETE FROM daily_reports WHERE report_date = '2023-10-27';
INSERT INTO daily_reports SELECT * FROM staging_reports WHERE report_date = '2023-10-27';
COMMIT;

6. Idempotent API consumers

If your pipeline fetches data from an API, don't just dump the raw response. If you are hitting a REST endpoint, use the ETag or Last-Modified headers to decide if you even need to pull the data.

For the data processing side, use a local cache (like Redis) to store the IDs of records already processed in the current window. If the pipeline dies and restarts, check the cache before hitting the sink. It saves costs and prevents duplicate processing.

7. Configuration as Code, not Magic

I’ve seen too many pipelines fail because a developer manually updated a variable in a UI dashboard. Your pipeline configuration—the source paths, the target tables, the look-back windows—should be in a version-controlled config file (YAML).

If you need to re-run a job, you shouldn't have to guess the parameters. You should be able to check out the repo at the commit hash from 3 AM, look at the config, and understand exactly what the job was trying to do.

# config.yaml
job_name: daily_user_sync
lookback_days: 1
destination: prod.users
partition_key: event_date
enable_merge: true

Photo by Yuta Koike on Unsplash

8. Monitor for drift, not just failure

A pipeline can "succeed" but still produce wrong data. Idempotency helps you fix things, but you need to know when to fix them. I implement "data quality contracts" using something like Great Expectations or even simple SQL queries that run immediately after the load.

If the row count for the current partition is 0, or if the sum of revenue is negative (in a domain where that's impossible), raise an alert. Because your pipeline is idempotent, the fix is just to trigger the re-run.

Conclusion

Making a pipeline idempotent isn't about writing more code; it's about removing the "statefulness" that makes systems unpredictable. By forcing atomic writes, deterministic keys, and transaction-safe operations, you stop managing failures and start managing outcomes. You stop being a firefighter and start being an architect.

When the 3 AM alert goes off, will you be scrambling to delete duplicate rows from a production table, or will you be confident enough to just hit "retry" and go back to sleep?

Cover photo by Tyler on Unsplash.

Are You Shipping a Data Warehouse or a Malware Delivery Vehicle?

Aniket Abhishek Soni — Tue, 08 Sep 2026 06:48:35 +0000

Ninety-eight percent of the production container images I audit in financial services contain at least one critical vulnerability, and nearly half of those vulnerabilities have a fix available that the engineering team simply hasn't bothered to apply.

It matters because when you’re pulling down a python:3.11-buster image, you aren't just getting an interpreter. You’re getting a Debian distribution, a shell, a package manager, and enough attack surface to keep a red team busy for a month. In a regulated environment, that’s not just tech debt; that’s a liability that will get you a stern email from compliance during your next SOC2 audit.

Why I chose this topic: I spent three weeks last quarter cleaning up a Log4j-style mess that only existed because a legacy data job was pulling a bloated, unpatched base image. I’m writing this because I’m tired of seeing production clusters running bloated images that act as a buffet for bad actors.

You’re currently facing a binary choice: continue to ship heavy, "convenient" images that make debugging easy but security impossible, or embrace the friction of minimal, hardened artifacts that keep you out of the headlines.

The contenders

Most data engineers in my circles land on one of three paths when containerizing their PySpark or Pandas workloads.

First, there’s the "Standard Distro" approach. This is FROM python:3.11-slim or FROM ubuntu:22.04. It’s familiar, it has apt, and you can pip install anything without breaking a sweat.

Second, we have the "Distroless" camp. This is Google’s gcr.io/distroless/python3. It contains absolutely nothing but your app and its runtime dependencies. No shell, no package manager, no local tools.

Third, there is the "Alpine/Musl" route. This is FROM python:3.11-alpine. It’s tiny, but it swaps the standard glibc for musl, which is a recipe for disaster if your data science libraries rely on C-extensions.

Photo by CHUTTERSNAP on Unsplash

The hidden cost of "easy" images

If you’re using python:3.11-slim, you’re deploying roughly 800MB to 1.2GB of junk. In a high-frequency data pipeline, that translates to real money.

Let’s look at the math. If you spin up 500 Spark executor pods a day, and each image pull takes 45 seconds because your image is 1GB, you are wasting over 6 hours of cumulative pod startup time daily. That’s compute cost, but it’s also latency. When a node fails and Kubernetes tries to reschedule, that bloat is the difference between a sub-second recovery and a cascading failure.

The "Standard Distro" path is a trap. You think you’re saving time, but you’re actually paying a "security tax" every time a CVE scan hits. When apt-get upgrade flags a critical vulnerability in libssl, you have to rebuild, re-push, and redeploy. With a 1GB image, your registry storage costs balloon, and your CI pipeline slows to a crawl as you push massive layers across the network.

The reliability tax of musl vs. glibc

The Alpine approach looks tempting because it’s usually under 100MB. But here is the concrete failure mode: pandas or numpy or pyarrow will eventually crash on Alpine. Why? Because they are compiled against glibc, and Alpine uses musl libc.

I’ve watched junior engineers spend three days debugging a Segmentation fault that only happens in production. It’s because pyarrow tried to call a memory allocation function that behaves differently in musl. You end up recompiling the world from source just to get your data job to run. If you think you’re saving time by using Alpine, you’re just front-loading the pain into the debugging phase. It’s a false economy.

Photo by Ivan N on Unsplash

The "Distroless" security reality check

"Distroless" is the gold standard for security, but it’s a nightmare for the developer experience. If a job fails in production, you cannot kubectl exec into the pod to check a config file or run a quick ping to test connectivity. The shell simply isn't there.

In a regulated environment, this is a feature, not a bug. If an attacker gains entry to your pod, they can’t run nmap or curl to pivot to your internal database. But you have to be ready to support your team when they scream that they can't debug their code. You need robust observability—OpenTelemetry traces, structured logs, and remote debugging tools—or your engineers will revert to FROM ubuntu just so they can sleep at night.

What I'd pick, and why

If I’m building a production data platform today, I’m using Wolfi (the un-distro) or a strictly controlled debian-slim base that I own.

Here is my hierarchy of needs:

Multi-stage builds are non-negotiable. I build in a heavy image with all the compilers and headers, then copy only the artifact (the virtual environment) into a lean runtime image. If you are doing pip install in your final production image, you’ve already failed.
SBOMs are the new baseline. I use syft to generate an SBOM at build time and grype to scan it. If the scan returns a Critical or High CVE that has a fix, the build fails. Period. No exceptions.
The "Distroless" ideal, with a twist. I use a "debug" sidecar pattern. My production container has no shell. If I need to inspect it, I attach a temporary sidecar container to the pod that has all the diagnostic tools I need. It’s the best of both worlds: a hardened production environment and the ability to debug when things go sideways.

My recommendation: Stop treating your container images as "virtual machines." They are artifacts of a specific process. If you find yourself wanting to install vim or git inside your container, you are building the wrong thing. Push that logic into your CI/CD pipeline or your infrastructure configuration.

The goal isn't just a small image. The goal is an immutable, verifiable, and minimal set of instructions for your data to execute. In finance and healthcare, we don't get to be "lazy" with the supply chain. Every extra package you leave in that container is a liability you’re signing your name to. Clean it up, lock it down, and stop shipping your problems to production.

Cover photo by Zoshua Colah on Unsplash.

Migrating Petabyte-Scale Parquet to Iceberg Without Dropping a Single Row

Aniket Abhishek Soni — Fri, 04 Sep 2026 11:09:57 +0000

Roughly 70% of companies that migrate their data lakes to Iceberg end up with "zombie" datasets: half-migrated, out-of-sync, and consuming double the storage costs. They treat the migration like a switch-flip, and that is exactly how you cause a P0 incident.

Why I chose this topic: I’ve been burned by "big bang" migrations in healthcare environments where downtime is measured in lost compliance certifications. After three failed attempts at manual synchronization, I settled on a shadow-table strategy that treats the migration as a background process, not a deployment window.

Most engineers interact with the Hive metastore daily. They treat it like a source of truth, but it’s actually a glorified index of files that has no idea if a Parquet file is corrupted, moved, or partially deleted. We rely on it to tell us where our data lives, yet we treat it as an immutable oracle. When you move to Iceberg, you aren’t just changing a table format; you’re replacing a loose collection of files with a transactional state machine.

How it actually works

You don't migrate by moving data. You migrate by shadowing the write path.

The strategy is simple: keep your existing Parquet pipeline as the "Primary," and introduce a secondary "Shadow" sink that writes to an Iceberg table simultaneously.

First, you need a dual-write mechanism. If you’re using Spark, don't try to manage this in your application logic. Use a Kafka Connect sink or a structured streaming job that reads from your source (e.g., Kinesis or Kafka) and commits to both targets.

In your Spark job, keep your Parquet sink as-is. Add the Iceberg sink:

// The primary Parquet path
df.write.format("parquet").mode("append").save(parquetPath)

// The shadow Iceberg path
df.writeTo("iceberg_db.shadow_table")
  .tableProperty("write.format.default", "parquet")
  .tableProperty("format-version", "2")
  .append()

The magic happens in the metadata. Iceberg creates a metadata/ folder inside your table directory. This is where the snapshot IDs live. Even if your source data is identical, the two tables will eventually diverge if your schema evolution isn't handled perfectly.

To ensure parity, you need an automated reconciliation job. Run a daily Spark job that performs a MINUS or EXCEPT operation between the two datasets:

-- Reconciliation check
SELECT * FROM parquet_table EXCEPT SELECT * FROM iceberg_table;

If that query returns rows, your shadow write failed or your schema translation is buggy. Do not proceed to the switch-over phase until that query returns zero rows for seven consecutive days.

Photo by Nicolas Arnold on Unsplash

The tradeoffs nobody mentions

This strategy is not free.

First, the storage cost. You are effectively doubling your storage footprint for the duration of the migration. In a healthcare context, this means keeping two copies of PII-encrypted data. Ensure your cloud lifecycle policies are set to expire the Parquet files only after you have fully decommissioned the old table.

Second, the "Ghost Latency." If your Spark job is writing to two different destinations, the overall job duration will be capped by the slower of the two. Iceberg’s commit process—which involves checking for conflicts in the catalog—can add 5-15 seconds to your job. If you have sub-second SLA requirements for your pipelines, this might be a non-starter.

Third, the metadata overhead. If you are migrating a table with 100,000+ partitions, the initial metadata load in Iceberg can cause your Driver OOM (Out of Memory) errors. You’ll need to set spark.driver.memory significantly higher than you think, often into the 16GB-32GB range, just to handle the snapshot state management during the initial transition.

Photo by Celine Nadon on Unsplash

When to reach for it (and when not to)

Reach for the shadow-table strategy if you have a "live" table—one that is being queried by BI tools or downstream microservices 24/7. This is non-negotiable in financial services where a missing row in a ledger report is a regulatory disaster.

Don't reach for this if your data is static. If you have a table that only gets updated once a month, just perform an INSERT OVERWRITE into a new Iceberg table during a maintenance window. The shadow strategy is complex because it’s meant for high-velocity, high-concurrency environments. If you don't have the write volume to justify the complexity, you are just inviting more failure points into your architecture.

Also, avoid this if your Parquet files are highly fragmented (millions of tiny 1KB files). Migrating "bad" data into Iceberg just gives you a "clean" format for "dirty" data. Use the migration window as an opportunity to perform a compaction job before you point your production queries to the new Iceberg table.

Conclusion

Migrating to Iceberg is less about the technical transition and more about the psychological shift from "file-based data" to "transactional data." By shadowing your writes, you buy yourself the time to catch the edge cases—like subtle schema mismatches or timezone shifts in timestamp columns—that would otherwise blow up in your face on a Monday morning.

The goal isn't to be fast. The goal is to reach a state where you can point your production traffic to the new table, delete the old Parquet files, and have your stakeholders remain completely unaware that anything happened under the hood.

Tags: #data #engineering #iceberg #migration

Cover photo by Valentin Lacoste on Unsplash.

Don't put an LLM in charge of your production database

Aniket Abhishek Soni — Fri, 04 Sep 2026 11:09:23 +0000

Last June, a junior analyst pushed a "helpful" GenAI assistant to our internal Tableau-connected lakehouse. Within forty minutes, the agent generated a SELECT * across a 40-terabyte partitioned table joined against a cross-region S3 bucket. The query didn't just fail; it locked the Databricks SQL Warehouse, blew our monthly compute budget in a single afternoon, and triggered a PagerDuty incident that ruined my kid’s birthday dinner. That query cost us $4,200 in DBU burn and an hour of downtime for our actual business stakeholders.

Why I chose this topic: I’m tired of seeing engineers treat Text-to-SQL as a magic wand rather than a dangerous, non-deterministic interface. I wrote this because production-grade data governance requires moving past "prompt engineering" into hard, infrastructure-level constraints.

The decision you are facing isn't whether to use LLMs for data—it's whether you want to build an expensive, unreliable hallucination engine or a system that actually respects your data perimeter. You are currently choosing between three architectural patterns: the "Raw Agent" (blind trust), the "Semantic Layer" (the guardrail approach), and the "Frozen Schema" (the brute-force approach).

The contenders

The "Raw Agent" is what happens when you just point a LangChain SQLDatabaseChain at your Unity Catalog metastore. It’s the "move fast and break things" approach, except you're breaking your company's P&L.

The "Semantic Layer" uses an intermediate abstraction—think dbt Semantic Layer or a specialized metric store—that acts as a firewall between the LLM and the raw SQL. The LLM talks to the model, not the table.

The "Frozen Schema" is the nuclear option. You don't give the LLM the entire catalog. You give it a strictly curated, subsetted DDL definition of exactly three tables, with no write permissions, and a LIMIT clause hard-coded into the underlying execution proxy.

Photo by Zulfugar Karimov on Unsplash

Latency and the cost of non-determinism

In production, latency is a feature, not a bug. If your Text-to-SQL agent takes 15 seconds to parse, plan, and execute, your business users will go back to asking the data team for CSVs.

The Raw Agent is a nightmare here. If you provide a schema with 500 tables, the token count for the system prompt alone will set you back significant latency on every single request. Using gpt-4o, you’re looking at 2-3 seconds of TTFT (Time To First Token) just to generate a JOIN that will likely fail because it missed a join key on a non-indexed column.

The Semantic Layer wins on efficiency. By exposing only core metrics—revenue_by_region, churn_rate_monthly—the LLM has a search space of 20 variables instead of 20,000 columns. You can cache these responses in Redis for common queries. With a 300ms retrieval time from cache, you’re looking at a sub-second user experience that feels like a real product.

Failure modes and the "Hallucination Gap"

The biggest failure mode in Text-to-SQL is the "Syntactically Correct, Semantically Wrong" query. The LLM might write a perfect JOIN statement that executes without error, but calculates gross_margin by subtracting shipping_costs from revenue when your actual business logic requires (revenue - cost_of_goods_sold) - shipping_costs.

Raw Agents fail silently. They give the user the wrong number, and the user makes a million-dollar decision based on it. There is no error message because the SQL is valid.

The Semantic Layer forces the LLM to use pre-defined logic. You move the source of truth from the LLM’s "knowledge" into your dbt project. When the LLM asks for "margin," it hits a view that already contains the logic. If the LLM tries to add a column that doesn't exist, the query parser throws an AnalysisException before the data is even touched.

Photo by Logan Voss on Unsplash

Infrastructure and operational burden

Governance is where most of these projects die. If you aren't using Row-Level Security (RLS) and Column-Level Security (CLS), you are one prompt away from a data breach.

With the Raw Agent, you have to implement RLS on the warehouse side, but you’re still exposing your entire schema. You need to keep your INFORMATION_SCHEMA restricted. If you use Databricks, you’re looking at complex GRANT hierarchies. It’s brittle. If you add a new column for PII, you have to remember to hide it from the LLM’s system prompt, or you're leaking sensitive data.

The Semantic Layer shifts the burden to the platform team. You maintain the semantic layer, and the agent acts as a client of that layer. This is the only way to scale. You don't grant the LLM a connection to the raw tables; you grant it a service account connection to the semantic interface. The service account has SELECT access only to the views you've blessed.

What I'd pick, and why

I’d pick the Semantic Layer every single time. If you’re building in a regulated industry—healthcare or finance—you don't have a choice.

Here is my non-negotiable stack:

The Guardrail: A dedicated Python intermediary (FastAPI) that sits between the LLM and the SQL engine.
The Constraint: Strict LIMIT enforcement on every query. If the LLM doesn't include LIMIT 100, the FastAPI layer injects it automatically before sending it to the warehouse.
The Identity: Never use a generic "data" service account. Use short-lived tokens mapped to the user requesting the data. If the user doesn't have access to the patients table, the agent can't generate a query that hits it.

The caveat? It’s harder to build. You have to write the code that maps natural language intent to specific semantic metrics. It’s not "plug and play." But in production, "plug and play" just means "plug in and pray."

Don't let an LLM write raw SQL against your production lakehouse. Build a layer, wrap it in strict permissions, and force the agent to play by your rules. Your PagerDuty rotation will thank you.

Cover photo by Tyler on Unsplash.