DEV Community: Aniket Abhishek Soni

Your data pipeline isn't idempotent until you prove it breaks and recovers

Aniket Abhishek Soni — Fri, 14 Aug 2026 14:29:13 +0000

The "Exactly Once" myth is the most expensive fairy tale in data engineering. We tell ourselves that if we just buy a high-end orchestrator or use a specific streaming library, the system will magically handle retries without duplicating a single row. It won't. If you aren't designing for idempotency at the storage layer, you’re just waiting for a primary key violation to wake you up at 3 AM.

Why I chose this topic: I spent four years in healthcare fintech cleaning up "duplicate event" disasters caused by naive pipeline retries. I’m writing this because I’m tired of seeing engineers treat retry logic as an afterthought rather than a core requirement of every single data job.

It was 3:14 AM on a Tuesday. PagerDuty didn't just beep; it screamed. Our financial reconciliation pipeline, which processes millions of daily transactions, had failed. The dashboard showed a massive spike in 500 Internal Server Error responses from our downstream ledger API.

The Airflow UI was a sea of red. My first instinct—the one I’d been trained to do—was to hit "Clear" on the failed DAG tasks. I assumed the transient network blip had cleared, and hitting "Clear" would simply pick up where it left off. I hit the button. I went back to sleep. I was wrong.

What we saw

When I logged back in at 8:00 AM, the ledger was in total chaos. The reconciliation report showed a $12M discrepancy in our accounts payable. The logs were a mess of Duplicate Key exceptions.

We thought it was an API timeout issue. We spent three hours chasing the load balancer configuration, convinced that our max_retries setting of 3 was somehow causing a race condition in the API gateway. We checked the AWS X-Ray traces, we grepped the Nginx logs for upstream_response_time, and we even blamed the infrastructure team for a silent network partition.

Everything looked like a connectivity problem. Nothing looked like a logic problem. We were looking for a broken pipe, but the water was actually being pumped into the same bucket twice.

Photo by Henrique Ferreira on Unsplash

Root cause

The root cause was buried in a Python script that pushed processed batches to PostgreSQL. Our insert logic looked like this:

def load_data(batch):
    for record in batch:
        cursor.execute("INSERT INTO ledger_entries (tx_id, amount, status) VALUES (%s, %s, 'PENDING')", 
                       (record['id'], record['amount']))
    conn.commit()

The pipeline was configured with a retries: 3 and retry_delay: 300 in our Airflow task definition. When the network blip hit, the task failed mid-batch. But because the database connection didn't technically close immediately, some of those INSERT statements had already hit the wire and succeeded before the network interruption severed the connection.

When Airflow retried, it re-ran the entire task. It didn't know which records had already been inserted and which hadn't. It just blindly tried to insert the same tx_id values again. Our database had a PRIMARY KEY on tx_id, so the second attempt crashed. But on the third attempt, a different, partial set of records succeeded, creating a Swiss-cheese distribution of data in the production ledger.

We were relying on the "All or Nothing" promise of a transaction block, but our batching strategy was too large and our retry logic was too dumb.

Photo by Logan Voss on Unsplash

The fix

We stopped treating the entire batch as a single, fragile unit. Instead, we implemented a "Load-Stage-Merge" pattern using a staging table.

First, we changed the load process to move data into a temporary, un-logged staging table that shared the same schema as the production ledger.

-- Create temp table per execution
CREATE TEMP TABLE stage_ledger_entries (LIKE ledger_entries INCLUDING ALL) ON COMMIT DROP;

-- Bulk copy from S3 to stage
COPY stage_ledger_entries FROM 's3://bucket/data.csv' WITH (FORMAT csv);

-- Atomic UPSERT into production
INSERT INTO ledger_entries (tx_id, amount, status)
SELECT tx_id, amount, status FROM stage_ledger_entries
ON CONFLICT (tx_id) DO UPDATE SET
    status = EXCLUDED.status,
    amount = EXCLUDED.amount;

By using the ON CONFLICT clause (the UPSERT pattern), we made the operation inherently idempotent. Whether the job runs once, five times, or a hundred times, the final state of the ledger_entries table remains identical. If a task fails, we don't care how much data made it through before the crash. We just run the task again. The database handles the logic of ignoring duplicates or updating existing records.

What we changed so it never happens again

We stopped allowing "blind" retries. We now enforce a set of rules for every data pipeline we deploy.

First, we moved away from row-by-row inserts. Row-by-row is slow and makes partial failures almost impossible to untangle. We now use standard bulk loading tools like COPY (Postgres) or bcp (SQL Server), which are transactionally safer and significantly faster.

Second, we implemented "Versioned File Naming." Every output file in our S3 buckets is now suffixed with a hash of the content or a unique execution ID. If a pipeline runs, it writes to a specific path. If we retry, it writes to the same path, overwriting the previous partial attempt. This prevents the "junk data" problem where multiple failed runs litter your data lake with partial, corrupt files.

Third, we introduced an observability layer specifically for idempotency. We added a check in our CI/CD pipeline that scans for INSERT statements that lack an ON CONFLICT or a WHERE NOT EXISTS clause. If you’re writing an insert, you have to justify why it isn't idempotent. If you can’t, the build fails.

Finally, we adopted a "State-First" mindset. We treat our destination database as the source of truth for the job’s progress. Before any task starts, it queries the target table to see which tx_id values already exist. It then filters those out of the source batch. We basically turned the job into a self-pruning process.

You will have failures. The network will drop, the API will time out, and the power will flicker. Don’t build a system that needs human intervention to clean up the mess at 3 AM. Build a system that, when it wakes you up, allows you to say "just hit restart" and go back to sleep. If you can't restart your job without fear, you aren't doing engineering; you're doing crisis management.

Cover photo by Mark König on Unsplash.

Stop Letting LLMs Write Raw SQL Against Your Production Lakehouse

Aniket Abhishek Soni — Wed, 12 Aug 2026 16:12:56 +0000

You ship the job. It passes CI. The integration tests green-light the agent. Then, at 2:00 AM on a Tuesday, a junior analyst asks the bot, "Show me all transactions for the last year," and your Databricks cluster spins up a 400-node task because the LLM generated a Cartesian join on an unpartitioned 50TB fact table.

Your boss isn't asking about the "power of GenAI" anymore. They’re asking why the cloud bill spiked by four figures in three hours.

I’ve spent six years in financial services and healthcare. I’ve seen what happens when you treat an LLM like a junior DBA who doesn't know the schema and definitely doesn't know when to use a LIMIT clause. If you want to put Text-to-SQL in production, stop building demos and start building guardrails. Here is how you stop the bleeding.

1. The Deny-List is Your First Line of Defense

Never let an LLM touch your raw catalog. If you give an agent access to information_schema, it will eventually try to DROP or TRUNCATE something because it hallucinated a table name that sounded like a temp file. Use a hard-coded schema allow-list. If it isn't in the config, the agent doesn't know it exists.

# Use a strict Pydantic model for your allowed tables
class AgentSchema(BaseModel):
    allowed_tables: List[str] = ["fact_transactions", "dim_customers"]
    forbidden_columns: Dict[str, List[str]] = {
        "fact_transactions": ["ssn", "credit_card_number"]
    }

# Inject this into your system prompt
system_prompt = f"""
You are a read-only SQL assistant.
You have access to: {", ".join(AgentSchema().allowed_tables)}.
NEVER query columns: {AgentSchema().forbidden_columns}.
"""

Photo by Pierre Bamin on Unsplash

2. Force the LIMIT Clause at the Engine Level

LLMs love to perform "select star" queries. In a governed lakehouse, that is a death sentence. You cannot trust the LLM to remember to add LIMIT 100. If you don't enforce this, your compute costs will be the least of your worries—the driver node OOMs will be. Don't rely on the LLM to behave; rely on the SQL proxy.

-- Wrap your LLM-generated SQL in a CTE or a view
SELECT * FROM (
    /* LLM OUTPUT HERE */
    SELECT * FROM fact_transactions WHERE region = 'EMEA'
) LIMIT 100;

Better yet, use Spark session configs to cap the row return if you’re using a notebook-based agent. Set spark.sql.execution.maxOutputRows to 1000 and let the driver kill the query before it hits the network stack.

3. The "Semantic Router" Pattern

Don't let the LLM generate SQL for complex analytical queries. Use a router. If the user asks for "total revenue by region," the LLM should hit a pre-defined, optimized SQL template or a stored procedure, not a raw SELECT SUM(...). Raw SQL generation should be reserved for simple data discovery only. If the query requires a window function or a complex join, point the agent toward a curated view.

4. Column-Level Security is Non-Negotiable

In healthcare, if the LLM hallucinates and includes a WHERE clause on a patient_dob column that shouldn't be exposed, you’re looking at a HIPAA violation. Implement an interceptor that validates the generated AST (Abstract Syntax Tree) against a security policy before execution. Use sqlglot to parse the LLM's output.

import sqlglot
from sqlglot import exp

def validate_query(sql_statement):
    parsed = sqlglot.parse_one(sql_statement)
    for column in parsed.find_all(exp.Column):
        if column.name in ["ssn", "medical_record_id"]:
            raise SecurityException(f"Illegal column access: {column.name}")
    return True

If the LLM tries to touch PII, kill the process. Don't log it; block it.

5. Human-in-the-Loop for DDL and Mutations

If your agent is capable of running INSERT, UPDATE, or DELETE commands, you have already failed. A production-safe Text-to-SQL agent should be strictly SELECT only. If you absolutely must have the agent update records, implement a "Human-in-the-Loop" (HITL) step where the SQL is rendered in a UI for a human to click "Approve" before the spark.sql() command is ever triggered.

6. Token Budgeting as a Circuit Breaker

LLMs are verbose. They love to explain their SQL. In a production pipeline, this is just noise. Set a max_tokens limit on your response, but more importantly, measure the length of the generated SQL string. If your agent generates a 500-line query for a simple question, it’s likely looping or confused. Treat long queries as a failure state and trigger a fallback to a "I don't understand" response.

Photo by Ilnur on Unsplash

7. Audit Logging the Latent Space

You need to log the prompt, the raw generated SQL, and the query execution stats (rows scanned, bytes spilled). When a query takes 10 minutes to run, you need to know exactly what the LLM thought it was doing. Use a decorator to log these to your observability platform (Datadog, Honeycomb, or even a simple ELK stack).

def log_agent_execution(func):
    def wrapper(*args, **kwargs):
        start = time.time()
        result = func(*args, **kwargs)
        log.info(f"Query: {args[0]} | Duration: {time.time() - start}")
        return result
    return wrapper

Conclusion

Text-to-SQL is not magic; it’s just a dangerous way to compile natural language into high-cost compute operations. If you don't build these guardrails, you’re essentially giving your entire organization a blank check to empty your data warehouse budget.

Start by parsing the SQL before it runs, enforce row limits, and keep the agent away from your PII. The LLM is the engine, but the guardrails are the steering wheel. Without them, you’re just driving toward a wall at 100 miles per hour.

When was the last time your LLM agent generated a query that cost you more than $50 in compute time?

Tags: #sql #llm #data #security

Cover photo by Tyler on Unsplash.

LLM Observability is Broken: Why MLflow 3 is the Only Way Out

Aniket Abhishek Soni — Mon, 10 Aug 2026 12:31:47 +0000

Six months ago, debugging our RAG pipeline meant staring at a wall of unstructured CloudWatch logs, trying to figure out which chunk of a 50-page PDF caused the hallucination. It was a digital scavenger hunt where the clues disappeared as soon as the request finished. Today, I look at the MLflow UI, filter by a specific trace_id, and see the exact RAG retrieval context, the system prompt, and the vector similarity score side-by-side.

If you’re still using print() statements and manual CSV tracking for your GenAI prompts, you aren’t building production software; you’re building a ticking time bomb.

Why the common approach falls short

Most of my peers in financial services treat LLMs like traditional Scikit-Learn models. They log a few parameters, save a pickle file, and call it a day. That works for a Random Forest where the input is a static feature vector. It fails catastrophically for LLMs.

In a GenAI system, the "code" is the prompt, the "data" is the retrieved context, and the "model" is a black box that changes its output based on a slight shift in temperature or a system message tweak. When your customer support bot starts telling users it’s okay to bypass compliance, you don’t need a model weight update; you need a prompt versioning history and a way to re-run that specific inference against a regression test suite.

The "standard" way—logging inputs and outputs to a SQL database—is insufficient because it ignores the call stack. You need to know the latency of the embedding call versus the generation call. You need to see the chain of thought. If you aren't capturing the full trace, you're debugging blind.

Photo by Tobias Fischer on Unsplash

Versioning prompts, not just weights

I’ve seen senior engineers hardcode prompts into their Python logic. prompt = "You are a helpful assistant...". This is amateur hour. When the product team wants to tweak the tone of the bot, they have to wait for a full CI/CD cycle, re-testing, and redeployment.

With MLflow 3, we treat prompts as first-class artifacts. We store them in the MLflow Model Registry, versioned like code.

import mlflow

# In production, we fetch the prompt by alias
prompt_template = mlflow.models.get_model_uri(
    model_uri="models:/compliance-bot-prompt/production"
)

# Using mlflow.log_input to track the exact prompt version
with mlflow.start_run():
    mlflow.log_param("prompt_version", "v1.4.2")
    mlflow.langchain.log_model(lc_model, "model")

When I get a ticket saying the bot is acting weird, I check the mlflow.run metadata. I see that v1.4.1 was deployed at 10:00 AM, and the latency spikes began at 10:05 AM. I don't guess; I roll back the registry alias to v1.4.0 with one API call. It’s boring, reliable engineering.

Tracing is the new unit testing

Evaluation in GenAI is messy because there is no ground truth. Is the answer "correct"? Who knows. But we can measure the components of the answer.

We use MLflow Tracing to capture the entire request lifecycle. When we run an evaluation suite, we don't just check the final string; we check the RAG context. Did the retrieval engine pull the right document? If the vector database returned a 0.7 cosine similarity score, that’s a failure mode we can track.

I’ve set up a custom evaluator that runs automatically on every PR. It pulls a subset of "golden questions," runs them through the trace, and uses an LLM-as-a-judge (usually GPT-4o) to compare the current trace against a known good trace.

# A snippet of our custom evaluation logic
eval_results = mlflow.evaluate(
    model_uri="runs:/<run_id>/model",
    data=eval_df,
    targets="expected_output",
    evaluators="default",
    evaluator_config={
        "col_mapping": {"inputs": "prompt", "outputs": "response"}
    }
)

The key here is that mlflow.evaluate isn't just checking accuracy; it’s checking the integrity of the trace. If the trace shows the model didn't use the provided context, the PR fails. No human needs to review the output manually.

Photo by Nicholas Cappello on Unsplash

The objections (and my answers)

The pushback I usually get is: "MLflow adds too much overhead. Why not just use a dedicated LLM observability tool like LangSmith or Arize?"

My answer is simple: Integration and ownership.

In a healthcare or fintech environment, sending proprietary PII-heavy traces to a third-party SaaS vendor is a compliance nightmare. You have to go through months of security reviews for every new vendor. MLflow is open-source. We host it on our own Kubernetes cluster. We own the data, the security posture, and the uptime.

Another objection: "MLflow 3 is heavy. We just need a simple logging dashboard."

That’s what I said three years ago. Then we had an incident where we needed to audit every single prompt-response pair sent to a customer over a 48-hour period to comply with a regulatory request. If we were using a "simple" tool, we would have been screwed. MLflow’s backend is extensible. We use S3 for artifacts and a Postgres database for metadata. It scales because it relies on infrastructure we already manage.

Finally, the "it's too complex" argument. Yes, learning how to structure your code for tracing takes a day of reading documentation. But you spend that day once. You spend the rest of your career fixing bugs caused by "simple" logging that didn't tell you the whole story.

Conclusion

Stop building "GenAI apps" that are just glorified CLI scripts wrapped in a web framework. You are shipping production software, and that requires production-grade tooling.

MLflow 3 gives you the observability, versioning, and evaluation framework that turns "it feels like it works" into "I can prove it works." If you aren't tracing your prompts, you don't know what your model is doing. And in the world of financial services and healthcare, "not knowing" is a liability you cannot afford.

The tooling exists. Stop making excuses, stop printing logs to stdout, and start versioning your prompts like the rest of your production code. Your future self, debugging a P0 incident at 2:00 AM, will thank you.

Cover photo by Wood Hong on Unsplash.

Why your Feature Store is a $50k/month glorified Delta table

Aniket Abhishek Soni — Sat, 08 Aug 2026 14:14:08 +0000

03:14 AM. My pager didn't just vibrate; it felt like a tactical nuke went off in my nightstand. The latency alerts for our fraud detection model had spiked from 45ms to 14 seconds. By the time I logged in, our P99s were off the charts, and the transaction approval queue in our core banking gateway was backing up faster than a clogged sink.

We were six months into a "modern" ML stack overhaul. We had bought into the Feature Store hype—a premium, managed service that promised to bridge the gap between our offline training data and online serving. It was supposed to be the "source of truth." Instead, at 3:14 AM on a Tuesday, it was the graveyard of our user experience.

What we saw

The symptom was simple: the feature-lookup microservice was timing out. The logs were a sea of RequestTimeoutException errors. Naturally, we went down the wrong path first. We blamed the networking team, assuming a VPC peering issue between our Kubernetes cluster and the feature store’s API endpoint.

We checked the max_connections configuration in our gateway. It looked healthy. We looked at the feature store’s internal dashboard—the "Health" tab was glowing a cheerful, deceptive green.

Then we saw it. The GetFeatureValue requests weren't failing because the network was down. They were failing because the underlying database—a distributed key-value store optimized for high-throughput reads—was choking on a massive batch write job. The feature store provider had scheduled a "Point-in-Time Join" synchronization job that was effectively DDoSing its own serving layer.

Photo by ashinJPG on Unsplash

Root cause

The culprit was our FeatureDefinition YAML. We had configured a refresh_interval of 1 minute on a feature vector that spanned 400 million rows.

# The offending config
feature_set: user_transaction_history
refresh_interval: 60s
backfill_strategy: incremental
storage_engine: optimized_kv_v2

The feature store was trying to compute a rolling 30-day window of transaction aggregates for every active user in our system, every sixty seconds, and writing the result into the online store. Because of how the provider handled concurrency, the "incremental" backfill wasn't incremental at all; it was locking tables and causing row-level contention that blocked the read API.

The vendor’s documentation had a footnote on page 42—the one everyone skips—that mentioned optimized_kv_v2 didn't support non-blocking writes for large-scale aggregations. We had effectively built a distributed lock on our most critical path.

Photo by 1981 Digital on Unsplash

The fix

We needed to get the site back up immediately, so we bypassed the feature store entirely. We took the offline training data—which lived in a perfectly healthy Delta table in our S3 bucket—and mounted it as a read-only cache in our feature-lookup service.

I pushed a hotfix that changed the service discovery logic. Instead of calling the feature store’s API, the service queried a local sidecar container running a lightweight version of the Delta table.

# The hotfix: bypass the vendor and hit the lake
import delta_sharing
# We used the Delta Sharing protocol to query the table directly 
# from the S3 bucket, bypassing the feature store's API.
df = delta_sharing.load_as_pandas(table_url)
lookup_val = df.loc[user_id, 'rolling_txn_sum']

The latency dropped from 14 seconds to 80ms instantly. The load on our infrastructure vanished. The feature store remained in its degraded state, completely irrelevant to the actual business requirement of authorizing a transaction.

What we changed so it never happens again

We stopped using the feature store as a database for high-velocity lookups. It’s a painful lesson: feature stores are excellent at discovery and versioning, but they are often terrible at acting as your primary hot-path storage.

We moved to a "Lake-First" architecture. All features now live in Delta tables. We treat the Delta table as the source of truth, and we use a simple, low-latency Redis cache for the hot-path lookups. We update the Redis cache via a simple Spark streaming job that listens to our Kafka stream, bypassing the "feature store" abstraction entirely.

If you are just doing simple lookups, a Delta table with a proper partitioning strategy (e.g., PARTITIONED BY (user_id % 100)) combined with a fast cache is all you need. You don't need a $50k/month vendor to perform a SELECT * FROM features WHERE id = X.

The complexity of "Feature Stores" is almost always a tax on your engineering velocity. We keep the feature store now only for metadata management and lineage—the "catalog" of our features. We no longer let the vendor touch our production traffic.

In 2026, the best ML infra is the one that's boring. If you find yourself debugging a vendor’s proprietary write-contention lock at 3 AM, you’ve already lost. Use Delta tables for your heavy lifting, Redis for your speed, and keep the "store" as a catalog, not a runtime dependency. The complexity isn't worth the dashboard.

Cover photo by Tyler on Unsplash.

Encrypting at Rest is for Amateurs: Building HIPAA-Compliant Lakehouses with Column-Level Access Control

Aniket Abhishek Soni — Thu, 06 Aug 2026 15:16:06 +0000

Most data engineers building for healthcare are dead wrong: they think that if they enable AES-256 at the storage layer, they’ve satisfied HIPAA. They haven't. They’ve just satisfied a checkbox on a low-effort audit. If you aren't governing data access at the column level, you aren't protecting PHI; you’re just waiting for a breach notification letter to hit your desk.

You are currently standing at a crossroads. You either build a lakehouse where access is controlled via coarse-grained IAM policies, or you build one where security is baked into the storage format itself.

The contenders

On one side, we have the "Infrastructure-as-Perimeter" approach. This is the classic cloud-native pattern: S3 buckets or Azure ADLS containers guarded by IAM roles. You grant a service account access to a bucket, and they get everything in it. PHI handling is done via "data zoning"—moving sensitive data into a separate, highly restricted bucket.

On the other side, we have the "Table-Format-as-Perimeter" approach. This uses modern open-table formats like Apache Iceberg or Delta Lake, paired with a metadata-driven access control layer like Unity Catalog or Starburst/Trino. Here, access is defined by SQL policies (GRANT SELECT ON TABLE...) rather than filesystem paths.

Photo by max im on Unsplash

The cost of operational complexity

The "data zoning" approach—the bucket-per-sensitivity-level model—is a nightmare that scales linearly with your paranoia. I’ve managed pipelines where we had to maintain three separate copies of the same patient demographics data just to keep the "Research" team out of the "Clinical" bucket.

You end up paying for storage three times, and your egress costs go up because your Spark jobs are constantly shuffling data across bucket boundaries. More importantly, it kills your agility. Every time a data scientist needs one more column for a longitudinal study, you’re stuck updating Terraform scripts, running a terraform plan that takes six minutes, and praying you don't break a production IAM policy.

The table-format approach is more expensive in terms of licensing or specialized compute, but cheaper in terms of "human-in-the-loop." When I use Unity Catalog or similar frameworks, I can tag a column as PHI and apply a masking policy. If someone queries the column, they get ***-**-1234 instead of a real SSN. The cost is a slight overhead on every query, but I don't pay for triple-redundant storage, and I don't pay for the senior engineer’s time spent babysitting bucket policies.

Failure modes and audit footprints

When your security is tied to IAM roles, your failure mode is "over-permissioning." It’s the path of least resistance. A developer needs access to a table for a quick bug fix? You grant them the IAM role that covers the whole bucket. Six months later, that developer has left the company, but their role still exists, and they still have access to 400 gigabytes of PHI because nobody remembered to prune the permissions.

In the table-format approach, the failure mode is "metadata drift." If your catalog isn't perfectly synced with your underlying Iceberg manifests, you might experience a "ghost access" issue where a user can see a file but not the table metadata, or vice-versa.

However, the audit footprint is where the table format wins. In the IAM model, an auditor asks, "Who accessed this PHI?" and you have to parse through CloudTrail logs. Have you ever tried to map a GetBucket event to a specific patient record? It’s impossible. You get an IP address and an IAM ARN. In the table-format model, the audit log gives you the identity, the exact SQL statement executed, and the specific column mask applied. If I’m an auditor, I know exactly what you saw.

Photo by Anna Evans on Unsplash

The reality of data leakage

The biggest lie in healthcare data engineering is that you can "anonymize" data by stripping identifiers. You can’t. You have to treat everything as PHI until proven otherwise.

In the bucket-based approach, you eventually reach the "Oops, I accidentally moved raw PHI to the dev environment" stage. It happens. Someone runs a cp command from the wrong terminal, and suddenly you have production patient data in a non-compliant sandbox. Because the security is at the bucket level, that sandbox is now a massive compliance liability.

With granular column-level security, even if a dev accidentally pulls raw data into a downstream table, the access policies follow the metadata tags. If the downstream table is tagged as PHI, the masking policy still triggers. You don't have to rely on the discipline of the developer; you rely on the policy engine.

What I'd pick, and why

If you are building a lakehouse today, stop building "data zones." Move to an Iceberg-based architecture with a centralized metadata layer.

I’d pick Apache Iceberg on S3, governed by a solution like Unity Catalog (if you’re on Databricks) or Starburst/Trino (if you’re running a mix of EMR/Glue). The reason is simple: it allows you to decouple storage from policy.

The caveat—and this is a big one—is that you must treat your access control code as production-grade application code. Do not just use the UI to click "Grant Access." You need to treat your GRANT and MASK SQL statements as code, checked into Git, and reviewed by a security engineer.

If you aren't doing code reviews on your access policies, you are just replacing bucket-level IAM mess with SQL-level IAM mess. The risk profile shifts from "accidental exposure" to "logic error in a policy script." But at least with a policy script, you can write unit tests. You can’t write a unit test for an IAM bucket policy that checks if the policy prevents unauthorized access to a specific column.

Finally, keep your pii_masking_udf simple. Don't try to build a complex cryptographic hashing function inside your SQL layer unless you have the crypto-engineering expertise to manage the keys. Keep the logic predictable. A simple regex-based mask for a Social Security number is better than a complex, bespoke hashing algorithm that nobody understands and that breaks when you upgrade your Spark runtime from 3.3 to 3.5.

Your goal is not to be clever. Your goal is to pass the audit without losing your mind. Move the security to the data, not the server.

Cover photo by Lee Lawson on Unsplash.

Kill the Full Refresh: Implementing Delta Lake Change Data Feed for Silver Pipelines

Aniket Abhishek Soni — Tue, 04 Aug 2026 13:26:08 +0000

The "Bronze-to-Silver full-refresh" is the industry’s most expensive lie. Every time I see a pipeline re-processing ten terabytes of historical patient records just to capture the five hundred updates from the last hour, I see money literally burning in the cloud provider’s furnace.

Why I chose this topic: In my last three roles, I’ve had to fix "silver" layers that collapsed under their own weight because they relied on full-table overwrites. Change Data Feed (CDF) is the only way to build pipelines that actually scale, and I'm tired of seeing engineers treat it like it’s too "complex" to touch.

Your current pipeline probably looks like this: A massive batch job kicks off at 2 AM, scans the entire Bronze storage, executes a deduplication window function over the last 90 days of partition data, and writes a massive OVERWRITE to the Silver table. It works fine when your data is small. But once you hit the 10-terabyte mark, your Spark driver starts throwing OOM errors, and your finance department starts asking why the Databricks bill for the "ETL Sandbox" is higher than the marketing budget.

You’re not doing data engineering; you’re doing data laundering. You are taking raw data, washing it in a massive compute cycle, and outputting the same data you already had, just slightly cleaner. It is a loop of technical debt that compounds every time a new source system is ingested.

The real problem

The problem isn't your code; it's the paradigm of batch-oriented thinking. We treat tables like immutable blobs that must be replaced to be updated. When you run a full refresh, you are essentially telling the cloud provider, "Please ignore the fact that 99% of this data hasn't changed, and charge me for the privilege of re-calculating it."

Delta Lake’s Change Data Feed (CDF) changes the contract. Instead of asking "What is the state of the world?", you ask "What happened since the last time I checked?" By enabling CDF, Delta writes a sidecar log of row-level changes—inserts, updates, and deletes. You can consume these changes incrementally, transforming only the delta, and merging them into your downstream tables. This is how you go from an 8-hour batch job to a 5-minute incremental micro-batch.

Photo by Winston Chen on Unsplash

Step 1: Enable the change feed on your bronze tables

Before you can do anything, you need to turn the feature on. This isn't retroactive, so do it today. If you have a legacy table, you’ll need to set the property and then perform a one-time rewrite (the last full-refresh you'll ever do).

ALTER TABLE bronze_patient_data 
SET TBLPROPERTIES (delta.enableChangeDataFeed = true);

-- If the table is already massive, run this to populate the change log
-- effectively from this point forward.
INSERT OVERWRITE TABLE bronze_patient_data 
SELECT * FROM bronze_patient_data;

Once this property is set, Delta starts tracking the operations in the _change_data folder hidden within your table path. Note that this increases storage slightly—usually by about 5–10%—but the cost of that storage is pennies compared to the cost of your redundant compute.

Step 2: Read the changes using Spark structured streaming

Now, instead of reading the whole table, you point your reader at the change log. You don't need to manually calculate watermarks or manage state if you stick to the readChangeFeed API. This is where the magic happens. You treat the changes as a stream, applying your logic—data type casting, PII masking, and deduplication—only to the records that actually changed.

# Read the feed as a stream
df_changes = spark.readStream \
    .format("delta") \
    .option("readChangeFeed", "true") \
    .option("startingVersion", "150") \
    .table("bronze_patient_data")

# Perform your business logic here
# Note: Since this is an incremental stream, 
# your transformations must be idempotent.
silver_df = df_changes.select("id", "patient_name", "status", "_change_type")

# Write to your silver table
silver_df.writeStream \
    .format("delta") \
    .outputMode("append") \
    .option("checkpointLocation", "/mnt/delta/checkpoints/silver_patient") \
    .table("silver_patient_data")

The _change_type column is your best friend here. It will tell you if the row was an insert, update_preimage, update_postimage, or delete. You don't need to guess why a row appeared; the audit trail is built-in.

Step 3: Implement the upsert logic in silver

Your Silver layer shouldn't just be an append-only log; it should be a mirror of the latest state of your entities. Since you are reading changes, you need to merge those changes into your final Silver table. This is where MERGE INTO becomes the backbone of your architecture.

def upsert_to_silver(microBatchDF, batchId):
    microBatchDF.createOrReplaceTempView("updates")

    # Merge the micro-batch into the target table
    microBatchDF._jdf.sparkSession().sql("""
        MERGE INTO silver_patient_data target
        USING updates source
        ON target.id = source.id
        WHEN MATCHED AND source._change_type IN ('update_postimage', 'insert') THEN
            UPDATE SET *
        WHEN NOT MATCHED THEN
            INSERT *
        WHEN MATCHED AND source._change_type = 'delete' THEN
            DELETE
    """)

# Execute using foreachBatch
silver_df.writeStream \
    .foreachBatch(upsert_to_silver) \
    .option("checkpointLocation", "/checkpoints/silver_patient") \
    .start()

By using foreachBatch, you gain full control over the MERGE logic. You handle the update_postimage for existing IDs and handle the delete events cleanly. This keeps your Silver table perfectly in sync with the Bronze source without ever scanning the entire dataset.

Photo by August Phlieger on Unsplash

Lessons learned from production

I’ve seen this setup fail in predictable ways. Here is how you avoid the midnight pager duty:

The Checkpoint Trap: Never, ever delete your checkpoint directory. If you do, you lose the state of your stream. You will be forced to reprocess the entire table from version zero, which is exactly what we are trying to avoid. Keep your checkpoints in a highly available, versioned S3 bucket with lifecycle policies that prevent accidental deletion.
The Deletion Lag: If your source systems perform hard deletes and you don't have a reliable way to capture those deletes in the Bronze layer, CDF won't help you with the Silver layer. Ensure your upstream integration (e.g., Debezium, Fivetran) is configured to propagate DELETE events. If you only see INSERTs, your Silver layer will become a "hall of mirrors" for deleted records.
Schema Evolution: CDF is sensitive to schema changes. If you add a column to your Bronze table, your MERGE statement in the Silver writer might fail if you are using UPDATE SET *. Be explicit in your MERGE column mappings if you expect your Bronze schema to evolve frequently.
The "Too Many Files" Problem: Because you are streaming, you might end up with millions of tiny files in your Silver table. Run OPTIMIZE and VACUUM on a schedule (e.g., weekly), but do it outside of your streaming window. If you try to run OPTIMIZE while a stream is writing to the same partition, you will hit lock contention errors.

Conclusion

Change Data Feed isn't a "nice to have" feature for high-scale platforms; it's a survival requirement. Once you stop treating your data lake like a collection of static files and start treating it like a stream of events, your infra costs will drop, and your pipeline reliability will skyrocket. It requires more upfront design than a simple INSERT OVERWRITE, but that's the difference between a junior engineer and a senior architect.

Try it: Enable delta.enableChangeDataFeed on your smallest, most annoying production table today. Use spark.read.format("delta").option("readChangeFeed", "true") in a local notebook to see exactly what those updates look like. Once you see the audit log, you’ll never go back to full-table refreshes again.

Tags: #datalake #spark #delta #engineering

Cover photo by Maria Ionova on Unsplash.

Is Exactly-Once Processing in Spark Actually Real?

Aniket Abhishek Soni — Sun, 02 Aug 2026 19:08:50 +0000

The "Exactly-Once" guarantee in Spark Structured Streaming is the most dangerous myth in modern data engineering. We are taught to believe that if we just wrap our Kafka source and our Parquet sink in the right configuration, our data counts will magically remain perfect despite network partitions, executor crashes, or node evictions.

I’ve spent the last six years cleaning up the mess left by engineers who took that claim at face value. Exactly-once is not a mathematical certainty inherent to your code; it is a delicate, fragile contract between your storage layer, your checkpoint directory, and your retry logic. If any of those three pillars is misconfigured, "exactly-once" quickly devolves into "at-least-once" or, worse, "data-loss-forever."

How it actually works

At the heart of the guarantee is the Write-Ahead Log (WAL) mechanism managed via the checkpointLocation. When you start a Spark Structured Streaming job, it initializes a state store—usually RocksDB or an in-memory map—and records the progress of every micro-batch in the checkpoint directory.

The magic happens during the commit protocol. When a micro-batch finishes, Spark writes a metadata file to the _spark_metadata folder in your sink. This file tells the driver: "These are the files I just wrote, and here is the offset I just processed."

If an executor dies mid-batch, Spark doesn't try to magically recover the partial writes. Instead, it relies on the fact that your sink is idempotent. If you are writing to S3, Spark uses a temporary staging directory. It writes the files there first, then performs an atomic rename or a move operation to the final destination only after the batch succeeds.

If you are using Kafka as a sink, Spark relies on the transaction coordinator. It starts a Kafka transaction, writes the records, and then commits the transaction. If the Spark driver crashes before the commit, the transaction expires, Kafka aborts the writes, and the Spark re-run simply attempts to write that same batch again. The consumer sees no duplicates because it is configured with isolation.level = read_committed.

The code looks deceptively simple:

val query = df.writeStream
  .format("parquet")
  .option("checkpointLocation", "s3a://my-bucket/checkpoint/job-name/")
  .start("s3a://my-bucket/output-path/")

But notice what is missing: error handling. The "exactly-once" logic is baked into the checkpointLocation. If that directory is deleted, corrupted, or moved to a different S3 bucket, your exactly-once guarantee evaporates instantly.

Photo by Carrie Borden on Unsplash

The tradeoffs nobody mentions

The primary cost of exactly-once is latency. You are constantly flushing state to durable storage. Every micro-batch has to wait for the metadata log to be updated in S3 or HDFS. In a high-throughput environment, the fsync overhead on your metadata storage will eventually become your bottleneck.

Then there is the issue of schema evolution. If you change your schema and your sink is Parquet, you might break the append operation. If the task fails to write a new file due to a schema mismatch, Spark will retry. If your retry logic isn't perfectly tuned, you end up with infinite retry loops that crash the driver, forcing you to manually clean the checkpoint directory—which almost always results in duplicate data or missed records.

Another massive pain point is state store growth. If you are doing windowed aggregations with withWatermark, Spark keeps that state in the checkpointLocation forever unless you trigger manual compaction or set the watermark correctly. I’ve seen production jobs grind to a halt because the RocksDB state store grew to several terabytes. When the driver tried to read that metadata on restart, it timed out, and the job entered a boot-loop.

Furthermore, exactly-once only applies if the entire pipeline is idempotent. If your Spark job calls an external REST API for enrichment, Spark has no way of knowing if that API call succeeded during a failed micro-batch. If the job retries, it hits the API again. Suddenly, you’ve hit your rate limit or, worse, you’ve charged a customer twice for the same event. Exactly-once in Spark is a local guarantee; it does not extend to the outside world.

Photo by Claudio Guglieri on Unsplash

When to reach for it (and when not to)

You reach for exactly-once when your business logic involves financial accounting, inventory reconciliation, or any case where a missing or duplicate record triggers an audit flag. If you are building a real-time dashboard or a recommendation engine, you probably don't need the overhead.

If you are dealing with external systems that are not transactional, stop pretending that Spark is doing the heavy lifting for you. You need to implement your own idempotency keys. Write your output to a staging table keyed by (event_id, batch_id), then use a MERGE INTO statement with a WHERE clause to ensure you never overwrite or duplicate.

Never rely on the default behavior if your sink is anything other than Parquet or Delta Lake. If you’re writing to a legacy database using JDBC, Spark’s exactly-once guarantee is practically non-existent because JDBC drivers rarely support the two-phase commit protocol required to roll back an insert if the Spark task crashes after the write but before the metadata update.

If your data volume is massive, consider trading exactly-once for at-least-once. If you can handle duplicates via a downstream deduplication layer (like a SELECT DISTINCT or a row-number window function in your warehouse), you will save yourself a massive amount of stress. You can increase the micro-batch interval, reduce the frequency of checkpoint writes, and drastically improve your cluster’s stability.

Conclusion

Exactly-once in Spark Structured Streaming is a tool, not a religion. It is a highly optimized, state-aware commit protocol that works perfectly as long as your storage is fast, your checkpoint directory is immutable, and your sinks are transactional.

The moment you step outside those bounds—by hitting external APIs, using non-transactional databases, or mismanaging your checkpoint metadata—the guarantee dies. Stop chasing the perfect pipeline. Instead, focus on building idempotent sinks and robust observability. If you can’t verify your data at the sink, it doesn’t matter how many "exactly-once" boxes you checked in the Spark UI.

Tags: #spark #streaming #data #engineering

Cover photo by Albert Stoynov on Unsplash.

Why I stopped trusting your JSON schemas and started enforcing data contracts in CI

Aniket Abhishek Soni — Fri, 31 Jul 2026 21:10:38 +0000

Schema registries are a waste of time if they don't break your build.

If you are still relying on a Confluent Schema Registry to "warn" your downstream consumers about a breaking change, you have already lost the war. You are treating data integrity as a runtime request for permission rather than a pre-compilation requirement. In six years of building data platforms for healthcare and fintech, I’ve seen enough production outages caused by "harmless" field deprecations to know that human communication is not a data strategy.

If it isn't in the CI pipeline, it doesn't exist.

Why the common approach falls short

The industry standard is currently "passive enforcement." You publish an Avro schema to a registry, and maybe, if you’ve configured your Kafka producers correctly, you get a compatibility check. But what about the downstream SQL transformation layer? What about the dbt models that expect a user_id to be a STRING but suddenly receive a BIGINT because someone in billing decided to clean up their database?

Passive registries are decoupled from the lifecycle of the actual code. A producer can change their schema, wait for the registry to return a 200 OK, and deploy their service. The consumer—often a different team entirely—finds out when their Pydantic models explode at 3:00 AM because the total_amount field is now a nested object instead of a float.

We treat data like a stream of bytes that might change, rather than an API that must be versioned. If your data pipeline doesn't treat a schema drift as a failed unit test, you aren't doing data engineering; you’re doing data archaeology.

Photo by okeykat on Unsplash

Moving contracts into the repository

My team moved away from centralized registry-only workflows toward a "Contract-as-Code" model. We define our schemas in YAML or Protobuf within the producer's repository, but we also maintain a contracts/ directory in a shared monorepo that our CI pipelines reference.

When a producer team wants to make a change, they don't just update their code. They open a PR that includes a change to the contract file. Our CI pipeline runs a tool like data-contract-cli (or a custom wrapper around jsonschema and dbt-checkpoint) to validate that the new schema doesn't break the existing downstream expectations.

# example-contract.yaml
dataContract: 1.0.0
servers:
  kafka:
    type: kafka
    bootstrapServers: broker.prod.svc
models:
  transactions:
    fields:
      transaction_id: { type: string, required: true }
      amount: { type: decimal, required: true }
      currency: { type: string, required: true }

When a developer tries to remove the currency field, the CI job executes a diff against the current production-deployed contract. If the diff indicates a breaking change—like removing a required field—the build fails. The developer is stopped before they even hit the deploy button. They are forced to either negotiate the change with the consumers or version the topic to transactions_v2.

Building the gatekeeper in CI

The magic happens in the .github/workflows/contract-check.yaml. You need to treat this job with the same severity as a terraform plan or a pytest suite. If the contract check fails, the PR cannot be merged.

jobs:
  validate-contract:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4
      - name: Install contract tool
        run: npm install -g @data-contract/cli
      - name: Validate schema compatibility
        run: |
          datacontract test ./contracts/transactions.yaml \
            --source ./schema/transactions.json \
            --compatibility backward

By setting --compatibility backward, we ensure that any new deployment is strictly additive. If a developer accidentally changes a type, the CI pipeline throws an exit code 1. This isn't just a linter; this is an automated negotiation.

In fintech, we dealt with PCI-DSS compliance. If a producer changed a field name that inadvertently exposed PII, our contract checks—which included regex patterns for field names—would catch the leak. By enforcing these rules at the PR level, we moved security and data quality from a reactive "ops" problem to a proactive "dev" problem.

Photo by Brecht Corbeel on Unsplash

The objections (and my answers)

"This will slow down our velocity," is the most common pushback I hear. Peer engineers argue that they should be able to iterate on their data structures as fast as they iterate on their services.

My answer: Velocity without reliability is just moving fast toward a cliff. When you break a consumer in production, you aren't just "moving fast"—you’re spending three days in an incident response war room fixing the downstream dependency chain. That is a massive tax on your velocity. A five-minute wait for a CI check is an investment, not a speed bump.

"What if we need to make a breaking change?" they ask. Then you version your stream. If you are changing the meaning of your data, you are fundamentally creating a new product. If you change a field type, you are breaking the contract. Create a new topic, publish the new schema, and run them in parallel for a migration window. Yes, it takes more work. That’s because data is a persistent asset, not a transient cache. If you aren't willing to manage the lifecycle of your data, you shouldn't be allowed to expose it to other teams.

"This is too much overhead for small teams." If you are a team of two, you might think you don't need this. You’re wrong. You’re just smaller, which means the cost of a catastrophic data failure is more likely to kill your startup. Automate the contract check early. It costs almost nothing to set up when you have three services. It’s nearly impossible to retrofit when you have three hundred.

Conclusion

We have to stop treating data producers as kings who can change the shape of the world whenever they feel like it. Data is the foundation of every financial ledger and healthcare record we manage. Treating it like an afterthought is negligence.

By moving your data contracts into your repository and enforcing them with hard failures in CI, you move the friction to where it belongs: the moment the change is proposed, not the moment it hits production. You don't need a fancy data mesh or a million-dollar governance platform. You need a YAML file, a basic CLI tool, and the backbone to make your CI pipeline block bad code.

Stop trusting the registry. Start breaking the build. Your downstream consumers will thank you—or at least, they won't be calling you at 3:00 AM.

Cover photo by Fons Heijnsbroek on Unsplash.

Stop copying your data for Vertex AI pipelines

Aniket Abhishek Soni — Thu, 30 Jul 2026 00:43:07 +0000

80% of data scientists spend their time cleaning data, but 90% of data engineers spend their time babysitting ETL pipelines that move the exact same data from a Lakehouse into a training-specific bucket.

We are literally burning cloud credits to store the same Parquet files in two different places because we’re too lazy to build a connector. If you are still running a "data export" step before your Vertex AI pipeline, you are paying a "laziness tax" that kills both your storage budget and your data lineage.

Why I chose this topic: In my time handling HIPAA-regulated datasets, I’ve seen teams lose six figures annually in egress costs and storage bloat simply because they didn't trust their Lakehouse to serve a model directly. I’m writing this because I’m tired of seeing "copy-to-GCS" steps in production pipelines that have no business being there.

We all use BigQuery or Dataproc daily, but most of us treat them like glorified file cabinets rather than compute engines. We treat the Lakehouse as a passive sink. That’s a mistake. The real power of the modern stack is that the feature engineering layer should be an extension of your query engine, not a separate shuttle service.

How it actually works

The goal is to stop moving data and start querying it in place using the Vertex AI SDK for Python. Instead of a pipeline that pulls data into a staging bucket, you should be using BigQuery-backed feature sets or direct integration with your Lakehouse (Databricks or BigQuery) using Vertex AI Feature Store.

Here is the mechanics of how we do this without a single gsutil cp command.

You define your training input as a BigQuery URI rather than a GCS path. When you initialize your CustomJob or PipelineJob, you point the input_data_config directly to your source table.

from google.cloud import aiplatform

job = aiplatform.PipelineJob(
    display_name="feature-eng-pipeline",
    template_path="pipeline.json",
    parameter_values={
        "bq_input_uri": "bq://my-project.feature_dataset.user_features_v1",
        "model_output_uri": "gs://my-bucket/models/"
    }
)

Inside the pipeline component, you shouldn't be reading CSVs. You should be using the google-cloud-bigquery client to create a temporary view or a materialized table that represents your training slice. If your feature engineering involves complex window functions or time-series joins, run those as a BigQueryToDataset operation.

The magic happens when you use BigQuery as the direct source for your Vertex AI TrainingPipeline. By setting the training_task_inputs to point at a BQ table, you allow Vertex to manage the read operation. If you’re using BigQuery ML (BQML), you aren't even leaving the warehouse. You’re shipping the compute to the data, which is the only way to scale in a financial services environment where data gravity is a real, tangible problem.

Photo by A Chosen Soul on Unsplash

The tradeoffs nobody mentions

Let’s be honest: avoiding data duplication isn't free.

First, you lose the "snapshot" guarantee of a flat file. If your source table in BigQuery changes while your training job is running, you get non-deterministic results. You are essentially training on a moving target. To fix this, you have to implement proper partition pruning or point-in-time snapshots in your DDL. If you don't use FOR SYSTEM_TIME AS OF in your BigQuery queries, you will eventually have a production model that fails to reproduce because the underlying data drifted during the training run.

Second, the BigQuery read API has throughput limits. If you have a massive dataset (hundreds of terabytes) and you try to pull it into a custom container running on Vertex AI, you will hit the 100MB/s per project limit for the BigQuery Storage Read API. You’ll be scratching your head wondering why your pipeline is taking five hours to initialize when the query itself took ten seconds. You’ll need to explicitly manage your concurrency settings.

Third, debugging becomes a headache. When you have an error in a CSV on GCS, you can just cat the file. When you have an error in a BQ-backed pipeline, you’re dealing with IAM permissions, service account scopes, and project-level quotas. If your Vertex AI service account doesn't have bigquery.jobs.create and bigquery.datasets.get on the source dataset, the job will fail silently or hang in a state of perpetual "pending."

Photo by Growtika on Unsplash

When to reach for it (and when not to)

Reach for in-place feature engineering when your data is already in BigQuery and you are running iterative experiments. If you are doing fast-paced model iteration, you cannot afford to wait for a 20-minute ETL job to shuffle files to GCS every time you want to add a feature.

Do NOT reach for it if you are dealing with unstructured data. If your Vertex AI pipeline is processing images, audio, or raw binary blobs, keep those in GCS. The overhead of storing binary data in a database as BLOBs is a disaster for performance and cost. Keep your structured features in the Lakehouse and your raw unstructured data in GCS, then use a BigQuery join or an index file to link them during the training loop.

Also, avoid this pattern if your team lacks strong SQL skills. If your data scientists are "Python-native" and refuse to touch SQL, you will end up with broken, unoptimized queries that scan the entire table instead of using partitions. You will bankrupt your cloud account in a week. If you can’t write a partition-aware query, stick to your GCS file copies.

Conclusion

The "data duplication" tax is an artifact of a time when compute and storage were tightly coupled. Today, with BigQuery and Vertex AI, they are logically separated but physically close. By treating your Lakehouse as the primary source of truth for your training pipelines, you reduce your attack surface for data drift, lower your storage costs, and simplify your lineage.

Stop treating your cloud storage like a temporary trash bin for ETL jobs. Start treating your Lakehouse like the compute engine it was designed to be. It’s harder to set up, and the IAM policies will make you pull your hair out for an afternoon, but the result is a clean, scalable pipeline that doesn't duplicate the world every time you want to train a model.

Tags: gcp, vertexai, data, engineering

Cover photo by Tyler on Unsplash.

Stop Betting the Farm on Glue: Why 2026 Spark Runtimes Demand Nuance

Aniket Abhishek Soni — Mon, 27 Jul 2026 22:54:07 +0000

In 2021, my "data platform" was a custom-built script suite managing EMR clusters with a 45-minute spin-up time and an S3 bucket that looked like a digital landfill. If a partition skewed, the cluster died, I got a PagerDuty alert at 3:00 AM, and I spent two hours manually resizing instances.

Today, I’m running thousands of jobs across a mixed-mode architecture. I don't touch cluster sizing for 90% of my pipeline. But the tradeoff? I’m now fighting "black box" performance tuning and bill shock from services that hide the underlying mechanics. The industry tells you these runtimes are plug-and-play. They aren't.

Why I chose this topic: I’m tired of reading "comparison tables" that treat these platforms as interchangeable commodities. In financial services, a 10% variance in executor memory allocation isn't just a cost center—it's a compliance failure or a missed SLA.

We all use Spark, but we treat the underlying runtime like a magic spell. We submit a job, pray to the resource manager, and check the Spark UI if it crashes. Most engineers don't understand that Glue, Databricks, and EMR Serverless are fundamentally different ways of abstracting the same open-source engine.

How it actually works

AWS Glue is essentially a managed Spark environment built on top of a highly abstracted YARN-like layer. When you run a Glue job, you aren't just running Spark; you are running it inside an AWS-managed container that forces specific configurations on you. Glue’s "Worker Type" (G.1X, G.2X, G.8X) is the core abstraction. A G.1X worker gives you 16GB of RAM, but only 4GB is actually usable for your executors after the overhead of the Glue agent and the OS.

Databricks is a different beast entirely. It uses Photon, a vectorized query engine written in C++. If you’re running standard open-source Spark on Glue, you’re hitting the JVM. If you’re on Databricks, you’re hitting Photon. This is why a query that takes 10 minutes on Glue can take 90 seconds on Databricks. It’s not just the "managed" part; it’s that they’ve essentially rewritten the execution layer to bypass Java’s memory management issues.

EMR Serverless is the middle ground. It is closer to "vanilla" Spark than Glue but without the cluster management headache. It uses a "pre-initialized" capacity model. When you define an Application in EMR Serverless, you are defining a set of worker nodes that are kept warm. The magic here is the spark-submit parity. If you have an existing on-prem Spark job, you can port it to EMR Serverless with minimal changes. You can’t say the same for Glue, where you’re often fighting the glueContext and the specific way it handles Dynamic Frames.

Photo by jonakoh _ on Unsplash

The tradeoffs nobody mentions

Let’s talk about the Glue job-bookmark feature. It’s a lifesaver for incremental loads, but it’s a nightmare when you have to reprocess data. Because the state is stored in the Glue metadata service, "resetting" a bookmark is an opaque process that often fails if your schema evolution is complex. I’ve spent more hours debugging Glue metadata locks than I care to admit.

Then there is the Databricks "Cloud Provider Tax." You pay for the DBU (Databricks Unit) on top of the underlying EC2 instance cost. In a high-volume environment, that DBU cost will dwarf your AWS bill. If your team isn't using Unity Catalog to optimize governance, you’re paying for a Ferrari and driving it to the grocery store. Also, the cold start time for Databricks Serverless is still a crapshoot. If your pipeline triggers on-demand, you’re looking at a 30-to-60-second latency before a single row of data is processed.

EMR Serverless is the most "honest" service, but it’s also the most unforgiving. The logging is… verbose. When a job fails in EMR Serverless, you’re often digging through CloudWatch logs that look like a cat walked across a keyboard. Because it lacks the high-level management UI of Databricks or the simplified Glue visual interface, your team needs to be comfortable with spark-submit arguments like --conf spark.executor.memoryOverhead. If your engineers don’t understand how to tune the JVM heap versus off-heap memory, EMR Serverless will just burn your budget with OOM (Out Of Memory) errors that provide zero actionable feedback.

Photo by Akshat Sharma on Unsplash

When to reach for it (and when not to)

Glue is your go-to for "I have a bunch of small, infrequent, ETL jobs." If you’re doing data movement from S3 to Redshift/Snowflake and you don't want to manage a single line of infrastructure code, Glue is fine. Do not use Glue for heavy machine learning model training or massive, multi-terabyte joins. The G.8X workers are expensive, and the lack of fine-grained control over the Spark shuffle service will lead to "shuffle block fetch failure" errors that will haunt your weekends.

Reach for Databricks when you are building a full-scale Data Lakehouse. If your engineering team is large and you need a unified interface for data scientists and data engineers, the DBU cost is justified by the collaborative tooling. Use Databricks when your performance requirements are non-negotiable—specifically, when you have complex SQL queries on massive datasets that Spark’s catalyst optimizer struggles to handle alone. Do not use Databricks if you are a small startup with a limited budget; the overhead of managing workspaces, clusters, and DBU consumption will kill your runway.

Use EMR Serverless when you have a well-defined, containerized Spark pipeline that you want to move to the cloud without re-architecting your logic. It is the best choice for "lift and shift" or for organizations that want to maintain a pure open-source Spark stack without the "vendor lock-in" of Databricks-specific APIs. Avoid EMR Serverless if your team is junior or lacks deep Spark internals experience; it is not a "managed" service in the sense that it solves your bad code—it just runs it faster and more reliably.

Conclusion

The "best" runtime is the one that minimizes your team’s cognitive load, not the one with the highest benchmark score. In 2026, the market has commoditized the hardware; the value is now in the abstraction.

If you’re building for scale and performance, pay the Databricks tax. If you’re building for ease of use and low maintenance, use Glue. If you’re building for portability and fine-grained control, EMR Serverless is your champion. Just stop pretending they’re the same, and stop choosing them based on a marketing slide deck. Pick your poison, tune your memory overhead, and for the love of everything, stop letting jobs run in the default namespace.

Cover photo by John Barkiple on Unsplash.

Why I Chose Snowflake Dynamic Tables Over Databricks DLT (And When I Regret It)

Aniket Abhishek Soni — Sat, 25 Jul 2026 22:01:38 +0000

It was 3:14 AM on a Tuesday when the PagerDuty alert fired. Our DLT pipeline, processing HIPAA-sensitive patient telemetry, had stalled. A library conflict in a custom Python UDF—introduced via a transient cluster restart—caused a silent data drift in our gold-layer aggregations. By the time we caught the stale dashboards, we’d pushed downstream reports to three hospital systems with flawed occupancy metrics. The cost wasn’t just the $4,000 in wasted DBU consumption during the retry storm; it was the two weeks of manual reconciliation to regain the trust of our compliance team.

Why I chose this topic: I’ve spent the last six years cleaning up broken pipeline architectures in regulated environments. I’m tired of reading marketing fluff that ignores the reality of debugging distributed state; here is how these tools actually behave when the pressure is on.

You are currently deciding how to build your next data platform. You have two primary paths: letting Snowflake handle the state management of your transformations via Dynamic Tables, or letting Databricks manage your DAGs via Delta Live Tables (DLT). Both promise to replace the "if-this-then-that" imperative spaghetti of Airflow with declarative SQL. Both lie about how easy that is.

The contenders

Snowflake Dynamic Tables (DTs) are a logical extension of the Snowflake storage engine. You define a TARGET_LAG (e.g., '1 minute'), and the underlying Snowflake scheduler handles the incremental materialization. It’s essentially a materialized view that doesn’t require you to manually write the refresh logic.

Databricks DLT is an abstraction layer on top of Spark. It manages the infrastructure (clusters), the dependencies, and the data quality (Expectations). It’s a full-fledged orchestration engine hidden inside a SQL/Python framework.

Photo by Logan Voss on Unsplash

The cost of abstracting the scheduler

Snowflake’s DT pricing is straightforward but dangerous. You pay for the compute used during the refresh. If your TARGET_LAG is set to 1 minute, Snowflake’s scheduler is constantly spinning up or using a warehouse to check for changes. If your underlying base table is constantly being ingested into (e.g., streaming Kafka data), that warehouse is running 24/7. I’ve seen junior devs set TARGET_LAG = '1 minute' on a table with 50 terabytes of data, effectively creating a permanent, expensive compute sink.

Databricks DLT pricing is more granular but significantly more complex. You have the DLT compute cost (DBUs) plus the underlying cluster cost. Because DLT uses Spark, it’s remarkably efficient at processing massive, partitioned shuffles—things that would make a Snowflake warehouse cry. However, the "Enhanced Autoscaling" feature is a trap. I’ve seen it spin up an r6id.4xlarge cluster to process a tiny backfill, then hang around for the 15-minute "down-scaling delay," burning money while doing absolutely nothing.

Failure modes and observability

When a Snowflake Dynamic Table fails, you check DYNAMIC_TABLE_REFRESH_HISTORY. The error codes are generally readable: TABLE_NOT_FOUND or COMPUTE_RESOURCE_LIMIT. It’s essentially a black box. If the refresh fails, Snowflake retries based on its own internal logic. You have very little control over the retry backoff or the sequence of operations. It’s "set it and forget it," until it breaks, at which point you are at the mercy of Snowflake Support.

DLT failure modes are far more "interesting." Because DLT is Spark, you get the full, ugly stack trace. You’ll see AnalysisException, Py4JJavaError, and the classic ExecutorLostFailure. The good news is that DLT's EXPECTATIONS are a game-changer. You can define CONSTRAINT patient_id_not_null EXPECT (patient_id IS NOT NULL) ON VIOLATION DROP ROW. This is declarative data quality that actually works. If you try to do this in Snowflake, you’re writing CASE WHEN statements in your select list like it’s 2012.

Photo by Craftsman Concrete Floors on Unsplash

Operational overhead and infrastructure

Snowflake wins on pure platform integration. If your data is already in Snowflake, you don't provision anything. You don't manage cluster sizes. You don't worry about node instance types. You enable the DYNAMIC_TABLE_USER role, grant the permissions, and write your CREATE DYNAMIC TABLE. It is the "I don't have a dedicated DevOps team" solution.

Databricks DLT requires you to be a Spark operator. You need to understand spark.sql.shuffle.partitions, memory overhead, and how to configure your pipelines.yml to ensure your clusters aren't thrashing. If you have a team that knows how to tune a Spark session, DLT is a Ferrari. If you don’t, it’s a Ferrari with the wheels welded on crooked.

What I'd pick, and why

If I’m building a greenfield project for a startup or a mid-sized data team where the primary goal is velocity and standard transformation (JOINs, simple aggregations), I am picking Snowflake Dynamic Tables. The maintenance burden is zero. You can put your entire pipeline in a single schema.sql file and version control it. The lack of granular control is a feature, not a bug, because it prevents your engineers from over-engineering the shuffle partitions.

However, if I am working in healthcare or finance where complex window functions, high-volume streaming, and rigorous data quality constraints are non-negotiable, I am picking Databricks DLT.

The caveat: You must accept that DLT requires a "Platform Engineer" on staff. You cannot treat DLT as a managed service that requires zero intervention. You will spend time monitoring cluster health, tuning storage-to-compute ratios, and debugging the occasional Spark driver OOM (Out of Memory) error.

If you choose Snowflake and your data volume hits the petabyte scale, you will eventually hit the wall of "warehouse compute limits" and move to DLT anyway. Choose Snowflake for the simplicity, but keep your DLT migration plan in a drawer. You’ll need it sooner than you think.

Cover photo by Liam Briese on Unsplash.

Stop Choosing Between Delta and Iceberg: UniForm is the Pragmatic Exit

Aniket Abhishek Soni — Thu, 23 Jul 2026 23:36:28 +0000

It was 3:00 AM on a Tuesday when the PagerDuty alert for our primary billing pipeline hit. We had a job failing on a Spark 3.5 cluster because our upstream vendor decided to switch their export format to Iceberg, while our entire analytical stack was locked into Delta Lake 3.0. The migration cost us six hours of downtime and roughly $45,000 in SLA penalties. The issue wasn’t the data quality; it was a religious war between two metadata layers that refused to speak the same language.

We treat table formats like sports teams, but they’re just protocols. You don’t need to pledge allegiance to Databricks or the Apache Software Foundation. You need your data to be readable by the specific query engine that actually does the job, whether that’s Trino, DuckDB, or Spark.

The illusion of engine-agnosticism

Most engineers assume that because they use an open format, they are portable. That’s a lie. If you write a Delta table, you are tethered to the Delta Standalone reader or a Spark implementation that supports the Delta protocol. If you use Iceberg, you’re at the mercy of the Iceberg Catalog and its specific manifest file structures.

The thing we rarely look at is the metadata root. In Delta, it’s a directory of JSON logs (_delta_log/). In Iceberg, it’s a snapshot-based tree structure starting from a metadata file that points to manifest lists. They are fundamentally different ways of tracking state, yet they both aim to solve the same problem: atomic ACID transactions on top of a pile of Parquet files.

Photo by Paolo Chiabrando on Unsplash

How it actually works

UniForm (Universal Format) is the bridge. Specifically, Delta Lake UniForm allows you to write in Delta format while the engine automatically generates the Iceberg metadata in the background.

When you enable delta.universalFormat.enabledIceberg in your table properties, you aren't just tagging metadata; you are triggering an asynchronous background process that translates the Delta log into Iceberg manifests.

Here is what that looks like in a Spark session:

ALTER TABLE my_production_table SET TBLPROPERTIES (
  'delta.universalFormat.enabledIceberg' = 'true',
  'delta.iceberg.catalogName' = 'my_hive_catalog'
);

Behind the scenes, Delta is essentially running a translation layer. Every time a commit happens in the _delta_log/, a background Spark job (or the writer itself, depending on configuration) maps those file additions and removals into the Iceberg snapshot format.

For the end user, this is magic. You point your Trino or Starburst cluster at the same S3 prefix you use for Spark, and Trino sees a perfectly valid Iceberg table. You aren't duplicating the data files—the Parquet files remain identical. You are only duplicating the metadata pointers.

The tradeoffs nobody mentions

If this sounds too good to be true, it’s because it involves operational "hidden" costs.

First, the background translation job is not free. If you are doing high-frequency streaming writes (e.g., every 30 seconds), the overhead of keeping the Iceberg metadata in sync can cause write latency spikes. I’ve seen commit latencies jump from 200ms to over 2 seconds because the cluster had to lock and update both the Delta log and the Iceberg snapshot history.

Second, version skew is a real failure mode. If your Delta version is 3.2, but the Iceberg translation logic is lagging behind in the current library version, you might end up in a state where the table is readable, but "time travel" queries fail. I once debugged a case where SELECT * FROM table AS OF VERSION AS OF '2026-05-01' worked in Spark but returned a TableNotSupported error in Trino because the Iceberg manifest was missing the specific partition evolution metadata that Delta had handled natively.

Finally, you are doubling your storage metadata footprint. In a multi-petabyte environment, the _delta_log and the metadata/ directory for Iceberg will grow to millions of files. If your object store has high latency on LIST operations, your catalog discovery will eventually become the bottleneck, not the data retrieval itself.

Photo by Pankaj Patel on Unsplash

When to reach for it (and when not to)

Use UniForm if your organization has a split-brain architecture. If you have a legacy Spark-based heavy processing pipeline but your analysts insist on using Trino or Snowflake for ad-hoc exploration, this is the only way to avoid the maintenance nightmare of double-writing data.

Do not use UniForm if your primary goal is "future proofing" without a clear current need. If your entire stack is already Spark-native, UniForm is just adding complexity and potential points of failure. Stick to pure Delta. If your stack is fully integrated with Iceberg-native tools like Tabular or Nessie, don’t introduce Delta just to "bridge" things.

The decision comes down to your query engine requirements. If your data must survive a lift-and-shift from a proprietary Databricks environment to a self-managed Trino cluster, UniForm is a lifesaver. If you are a startup with a single query engine, you’re just paying for extra compute cycles to translate metadata that nobody is reading.

Conclusion

We are moving into an era where format-lock is becoming a legacy burden. By using Delta as your primary writer and UniForm to project Iceberg metadata, you get the robust ecosystem support of Delta and the engine interoperability of Iceberg.

Stop worrying about which company’s "standard" wins the market. Focus on the metadata translation layer that keeps your data accessible. The goal isn't to pick a side; the goal is to ensure that when your primary query engine goes down, you can pivot to another one without having to rewrite your entire data lake. In 2026, the only real technical debt is a siloed format.

Tags: #data #engineering #delta #iceberg

Cover photo by Taylor Vick on Unsplash.