BigQuery overspend in practice looks different from what you'd expect
Most people imagine it as a single runaway query that processes petabytes of data and generates a massive one-time charge
What I actually see is quieter and more persistent
Regular scheduled queries, running daily or hourly, that were written for exploration and never optimized for production
A dashboard that refreshes every 15 minutes and runs a full table scan on a 200GB table every time it loads
An analytics pipeline that joins four large tables without partition filters because when it was written the tables were small and it didn't matter
Each individual query isn't catastrophic. Together, running constantly, they add up to thousands of dollars a month of data processing that could be done for hundreds
I worked with a team where the BigQuery bill was $5,200 a month. They had one analyst and four engineers. The data volume didn't justify that cost
We used INFORMATION_SCHEMA.JOBS to look at the top 20 queries by bytes processed over the last 30 days
Three queries were responsible for 71 percent of total data processed
One was a daily pipeline that selected all columns from a 400GB table. Rewriting it to select only the 8 columns the pipeline actually used and adding a partition filter reduced its data scan from 400GB to 3GB per run
One was the dashboard I mentioned. We materialized the results into a summary table that got refreshed twice a day instead of running the full scan on every dashboard load
One was a join query that could be rewritten to filter earlier, before the join, instead of after
Those three rewrites reduced the monthly BigQuery cost to $800
$4,400 a month from query optimization on three queries
The code changes took two days
The savings were immediate and permanent
Top comments (0)