Stop Pretending BigQuery and Databricks Are Just 'SQL Warehouses

Why I chose this topic: I’m tired of seeing architecture decisions made based on marketing whitepapers that ignore the reality of a 3 AM pipeline failure. Most engineers pick their platform based on hype; I’m writing this to force you to pick based on your actual ops budget.

You ship the job. It passes CI. Then, the P99 latency on your dashboard spikes from 400ms to 12 seconds, and the CFO is asking why the "serverless" bill just hit five figures for a simple join. You’re left staring at a query profile that tells you absolutely nothing useful.

The industry wants you to believe that BigQuery and Databricks SQL are interchangeable commodities. They aren't. Choosing between them is a choice between trading away your time for convenience (BigQuery) or trading away your sanity for control (Databricks).

Most of my peers argue that "it’s all just SQL, so pick the cheapest one." That is a dangerous simplification. In financial services, where I spend most of my time, the cost of a failed compliance report is higher than the cost of a slightly inefficient query. Treating these platforms as simple SQL interfaces ignores the underlying storage engines, the concurrency models, and the reality of how they handle partition evolution.

Why the common approach falls short

The "benchmark first" mentality is the primary reason projects go off the rails. You see a vendor-sponsored TPC-DS benchmark showing 2x speed on Databricks, so you migrate. Six months later, you realize you’re spending 40% of your engineering time tuning Z-ORDER clusters and managing VACUUM commands.

The common approach assumes that performance is purely a function of the engine. In practice, performance is a function of maintenance.

In BigQuery, you don't manage vacuuming. You don't manage cluster sizes. You don't manage the spark.sql.shuffle.partitions setting. If you’re coming from a world where you need to optimize your data layout to hit your SLAs, BigQuery feels like magic. But that magic has a price tag that is often opaque. When you run SELECT * on a petabyte-scale table without a WHERE clause in BigQuery, you aren't just slowing down the system; you’re literally burning cash. If you don't have rigid project-level quotas, you’re one rogue intern away from a six-figure invoice.

Photo by Igor Omilaev on Unsplash

The illusion of "Serverless" simplicity

I once worked on a migration where we moved a massive ETL pipeline from a custom Spark cluster to BigQuery. The "serverless" aspect was the selling point. For three weeks, it was great. Then, the INFORMATION_SCHEMA.JOBS_BY_PROJECT started showing a pattern: our complex, nested JSON parsing was hitting the 100GB per-query shuffle limit repeatedly.

In BigQuery, you hit a wall. When you hit a query limit, your only recourse is to rewrite the SQL into multiple stages or utilize BigQuery Scripting. You are at the mercy of Google’s internal query optimizer. You can’t "tune" the engine—you can only tune your own code.

-- BigQuery: You're stuck with their optimizer
-- If this hits the 100GB limit, you have to break it apart
SELECT
  user_id,
  ARRAY_AGG(STRUCT(event_type, timestamp)) as events
FROM `prod.events`
GROUP BY 1;

Contrast this with Databricks SQL. If a query is failing or underperforming, I can jump into the Spark UI. I can see the exact task skew. I can look at the physical plan, identify a broadcast join that’s failing, and force a hint.

-- Databricks: You have the levers
SELECT /*+ BROADCAST(dim_users) */
  fact.id,
  dim_users.name
FROM fact_table fact
JOIN dim_users ON fact.user_id = dim_users.id;

In Databricks, the "failure mode" is that you spend time fixing the query. In BigQuery, the "failure mode" is that the query just fails, and your hands are tied by the API.

The operational tax of the Lakehouse

Databricks SQL is fundamentally a lakehouse play. You are managing the underlying Delta Lake tables. This gives you immense power. You can travel back in time with VERSION AS OF, you can MERGE data with transactional integrity, and you can control how your files are laid out on S3 or ADLS.

But let’s be honest: VACUUM is a pain. If you forget to run it, your storage costs spiral because you’re keeping infinite history of every tiny update. If you don't OPTIMIZE your tables, your read performance degrades as your data evolves. This is "operational tax."

In healthcare, we handle PII and HIPAA-regulated data. Databricks allows me to control the storage layer access through Unity Catalog at a granular level. I can move data between regions or cloud providers without re-ingesting everything because it’s just Parquet/Delta. BigQuery is a walled garden. Once your data is in BigQuery Storage, you are using the BigQuery API. You are locked in. Moving out requires an egress bill that will make your finance department cry.

Photo by Oxa Roxa on Unsplash

The objections (and my answers)

The biggest pushback I get is: "But BigQuery is cheaper for ad-hoc analysts."

My answer? Only if you have perfect governance. BigQuery is a "spend-now, pay-later" platform. It’s excellent for teams that don't have a dedicated data engineer to manage infrastructure. If your organization is small, and you don't have the headcount to manage a Databricks workspace, BigQuery is the only logical choice. You pay the premium for the lack of operational overhead.

Another objection: "Databricks SQL is too expensive because of the compute clusters."

My answer: Databricks Serverless SQL warehouses are getting better, but yes, they cost more than a raw BigQuery query if they sit idle. However, the cost of an engineer’s time to fix a BigQuery performance bottleneck that can't be tuned—because the optimizer is a black box—is far higher than a 15% delta in compute costs. If you are at a scale where you are running massive, repetitive ETL jobs, you should not be paying for the "serverless" convenience of BigQuery. You should be paying for the "control" of Databricks.

Finally, people argue that BigQuery’s ML integration (BigQuery ML) is a game changer. It is, until you need a custom library that Google doesn't support. Then you’re back to exporting your data to a vertex AI pipeline, which defeats the purpose. Databricks handles the transition from SQL to Python/MLflow seamlessly because it’s the same underlying Spark environment.

Conclusion

If you are a lean startup or a mid-sized team with one or two data generalists, choose BigQuery. The lack of infrastructure management is worth the risk of an occasional surprise bill. You want to query, visualize, and get out.

If you are in an enterprise environment, especially in finance or healthcare, where data governance, auditability, and custom optimization are non-negotiable, choose Databricks. You need the ability to reach under the hood when things inevitably break.

Don't choose based on a benchmark. Choose based on how much time you want to spend being a database administrator versus a data engineer. I choose the control, even if it means I have to run a VACUUM command at 3 AM once in a while. At least I can fix it.

Cover photo by Marcin Jozwiak on Unsplash.