When our project manager forwarded the May 2026 invoice, the number had tripled from where it was two months earlier. That was the monthly GCP bill for two regional deployments of the same platform. We had seen it climb through April, but this was the first time the daily cost chart showed no sign of flattening. The peak day had already hit 3x our normal baseline.
Nothing was broken. The converter was processing records and the dashboards were up. The system was doing exactly what it was built to do. Together with Mas Chiqo, our devops teammate, we pulled up the billing console and started figuring out whether the system actually needed to cost that much.
What we run
We operate a federated health data platform deployed in two districts in Indonesia. The platform collects data from twelve different source systems: electronic medical records from hospitals, nutrition tracking from community health workers, TB surveillance from the national system, immunization registries, midwife apps, and several others. All of that gets converted into FHIR R4 resources and stored in a central FHIR datastore per district.
Each district runs its own GCP project. The converter runs on Cloud Run alongside about 30 other microservices per project. BigQuery handles analytics and dashboard queries. Cloud Healthcare API is the FHIR datastore. Cloud Scheduler triggers the converter and about 50 scheduled queries across both projects. Then there is Secret Manager for credentials and checkpoints, Cloud SQL for application databases, a handful of Compute Engine VMs for data science and tooling, and Cloud Logging collecting output from everything.
At peak, we were processing over 500,000 records per day across 80 source tables. Two projects, two districts, running 24/7. That is a lot of moving parts generating cost, and before the spike, we had never looked carefully at which parts were generating how much.
Following the data to find the cost drivers
We did not start by guessing. We set up the billing export queries, pulled Cloud Run utilization data, and looked at INFORMATION_SCHEMA.JOBS_BY_PROJECT to understand where the money was actually going:
SELECT
DATE(creation_time) AS dt,
ROUND(SUM(total_bytes_billed) / POW(2, 40), 2) AS tib_billed
FROM `region-asia-southeast2.INFORMATION_SCHEMA.JOBS_BY_PROJECT`
WHERE creation_time >= TIMESTAMP_SUB(CURRENT_TIMESTAMP(), INTERVAL 30 DAY)
GROUP BY dt ORDER BY dt
This query gave us daily scan volume in TiB, which we could plot against the billing export to see exactly when costs started climbing and why. BigQuery was roughly 36% of the total bill. Cloud Run was another 21%. But the percentages alone did not explain the spike.
What the data showed was a compounding pattern. Several upstream data providers had pushed large historical backfills, which is a normal part of operating a health data platform. Migrations happen. New data sources come online. One upstream migration brought in 427,000 rows in one week (43x the normal volume). Another pushed 2.27 million rows in a single week.
The system handled it correctly. The converter processed the records, and the FHIR store grew. The main resource table went from 150 GiB to 201 GiB. But a deduplication scheduler was scanning the full FHIR store three times per hour. At 150 GiB, each scan cost very little. At 201 GiB and growing, the same scheduler became the single most expensive query in the system. The scan volume data on one project tells the story clearly:
| Period | Daily BQ scan |
|---|---|
| Week 1 | 2.36 TiB |
| Week 4 | 4.0 TiB |
| Week 5 | 7.37 TiB |
| Week 6 | 9.26 TiB (peak) |
| After scheduler pauses | 1.34 TiB |
| After all fixes | 0.47 TiB |
From 2.36 TiB to 9.26 TiB in five weeks, without anyone changing a single scheduler configuration. The data volume changed, and the cost followed. That same pattern was happening on the other project too, going from 1.1 TiB to 2.85 TiB daily, though the absolute numbers were smaller.
What the data told us to do
Once we had the billing and utilization data in front of us, we split the work across BigQuery, Cloud Run, Compute Engine, and the smaller services. Each fix started with a specific metric.
BigQuery: query analysis pointed to the converter polling pattern
We used INFORMATION_SCHEMA.JOBS_BY_PROJECT to rank queries by bytes billed. The top consumer was the converter's polling query, which checks for unprocessed records using an anti-join against a multi-million-row ledger table. Every poll for every table scanned the entire ledger.
The scan data told us a CLUSTER BY table_name on the ledger, combined with adding a table_name filter to the JOIN clause, would let BigQuery prune 90% of the scan. The change was small:
-- Before: scans entire ledger on every poll
LEFT JOIN processing_report AS B
ON A.uuid = B.uuid
-- After: BigQuery prunes to just this table's slice (~94% fewer bytes)
LEFT JOIN processing_report AS B
ON A.uuid = B.uuid AND B.table_name = 'source_table_name'
After deploying that change, per-query scan dropped from 531 MiB to 53 MiB.
We then looked at the pattern more carefully. The data showed that on a typical hourly run, most tables have zero new rows. We built a high-water mark routing layer: if a table has no backlog, use a cheap timestamp query instead of the full anti-join.
def get_unprocessed_data(table_name):
"""Routes to HWM or anti-join based on backlog check."""
if table_name not in _HWM_READY_TABLES:
backlog = _check_backlog(table_name)
if backlog == 0:
_HWM_READY_TABLES.add(table_name)
else:
return _get_unprocessed_antijoin(table_name)
return _get_unprocessed_hwm(table_name)
The HWM path runs a simple WHERE timestamp > last_processed instead of joining millions of rows. This brought the converter's BigQuery cost down by over 95%.
BigQuery: evaluating Enterprise Edition after reducing query volume
Both projects were running BigQuery Enterprise Edition with autoscaling slots. Enterprise Edition pricing is based on slot usage, and when scan volume was high, the autoscaler was spinning up slots constantly. This was the single largest line item on the bill.
After deploying the clustering fix, the high-water mark routing, and pausing the expensive schedulers, our actual BigQuery compute usage dropped by over 90%. We looked at the numbers and asked whether Enterprise Edition still made sense. The autoscaling slots were now mostly idle, but we were still paying for the baseline. We deactivated Enterprise Edition on both projects and fell back to on-demand pricing.
The result was immediate. The projected BigQuery cost on on-demand is about 4% of what we were paying with Enterprise Edition. The query volume reductions we had already made meant that on-demand pricing was now far cheaper than even the minimum Enterprise Edition commitment. This single change, deactivating a pricing model that no longer matched our usage, was the largest cost reduction in the entire effort. There is a risk: if scan volume spikes again (another large backfill, or re-enabling the dedup scheduler), on-demand could become more expensive than Enterprise Edition was. We are monitoring on-demand costs weekly to catch that before it happens.
Scheduled query audit: frequency, dead joins, and dead code
We listed every scheduled query and its run frequency, then compared it against when the output data was actually consumed. The gaps were wide.
One dedup scheduler ran a UNION ALL full-scan of every FHIR resource table three times per hour. One scheduler, hundreds of dollars per month. We checked its downstream consumers and found it could safely be paused while we work on a better dedup approach.
We found schedulers with a dev- prefix running in the production project. One ran every 7 minutes, another ran every hour. They did not produce errors, so they never surfaced in monitoring. They only showed up when we audited the scheduler list against actual usage.
Export queries ran every 12 hours but were consumed once a day. Dashboard refresh queries ran every 5 hours but updated a table used in a daily report. We matched frequency to actual consumption patterns across both projects and paused or reduced 26+ schedulers.
Rewriting scheduled query SQL: the dead JOIN that ate 96% of slot-time
Frequency was not the only problem. Some scheduled queries were expensive because the SQL itself was inefficient.
Two scheduled queries that build a maternal health dashboard table started timing out at 6 hours. The SQL had not changed. The data had. Both queries contained a LEFT JOIN against a clinical orders table. The joined alias was referenced only in the ON clause, never in any SELECT or WHERE. It did nothing except multiply rows. The orders table stores one row per ordered lab test, so each visit record (full screening panel) fanned out 113 times. With 362,000 visit records, the intermediate result was 41 million rows before SELECT DISTINCT collapsed them back down.
This was harmless when the orders table was small. Then bulk conversion of two data sources landed 8 million new rows in one week. The fan-out blew past the 6-hour query timeout. When we queried INFORMATION_SCHEMA.JOBS_BY_PROJECT for slot usage, these two queries had consumed 96.1% of all project slot-time over the previous three days (62.6% and 33.5% respectively), leaving 3.9% for the 35,582 other queries.
The fix was one line: comment out the dead JOIN. Run time dropped from 6 hours (timeout) to 25 minutes. We also rewrote the partitioned update query to read from a 52 MB computed dashboard table instead of scanning 1 TB of raw FHIR tables directly. That cut its monthly cost from about USD 130 to USD 3.
The 4-billion-row table that should have been 12,000
One table had 4,040,397,903 rows. The actual data was 12,033 records.
A scheduled query had been running with WRITE_APPEND since February, duplicating every row on each run. The table grew to 764 GiB. This showed up when we sorted scheduled query outputs by storage size. We rebuilt the table from source, switching to WRITE_TRUNCATE. If we had not looked at storage sizes, this table would still be growing.
Cloud Run: utilization metrics vs. allocated resources
We pulled 30-day p99 CPU and memory utilization for every Cloud Run service across both projects. The gap between allocated and actual was large. One service had 4 CPU and 12 GiB allocated, with peak utilization at 12.3% CPU and 1.0% memory. Several services had similar profiles: provisioned for a load that never came, or provisioned for a peak that happened once and never returned.
Based on the utilization data, we downsized 44 services in one session, deleted 4 with zero traffic in 30 days, and switched 17 idle always-on services to request-based billing. The Cloud Run revision history shows it all happened in about 13 minutes. One project's daily compute cost dropped by 57% the next day.
We also right-sized the converter itself from 4 CPU / 16 GiB to 2 CPU / 4 GiB, and reduced the trigger frequency from every 30 minutes to hourly. During the backfill peak, the higher spec made sense. The converter was processing 571,000 records per day across 80 tables, running 20 worker threads with batches of 25 records. After the backfill cleared, utilization data showed the converter sitting well below capacity. We lowered workers and batch sizes to match the new steady state: 2 table workers, batches of 5, query limit down from 100,000 to 15,000.
Secret Manager: storage billing we were not tracking
We also went through every line item in the billing export, including services we assumed were cheap. Secret Manager was not cheap. Our patient deduplication service stores its checkpoint as a secret version, creating a new version on every run. Since late 2025, old versions were never destroyed. We found over 30,000 accumulated versions. The "Secret version replica storage" SKU was a top-10 line item on the bill. After cleanup, each checkpoint secret has 1 version. Cost dropped significantly.
Compute Engine: the GPU VM and the static IPs
Compute Engine was 6.6% of the bill. Most of that was expected: 10 VMs running various services (data science, form collection, OCR, migration tools). But one line item stood out.
A GPU VM with an NVIDIA L4 and 1 TB disk had been provisioned for OCR model fine-tuning. The fine-tuning work had finished weeks earlier. The VM was still running. It was the single most expensive Compute Engine resource. After confirming the fine-tuning was complete, we deleted it.
We also found unused static IP addresses being billed across both projects, some attached to services that no longer needed them. Small amounts individually, but they add up when nobody is watching.
Cloud SQL: fewer resources for the same workload
Cloud SQL was running multiple PostgreSQL instances 24/7. After reviewing actual query loads, we right-sized the instances. Total Cloud SQL savings: -36%.
Cloud Logging: paying for logs nobody reads
Cloud Logging on one project dropped 45% after reducing log levels and removing redundant logging. On the other project, logging actually increased 50% because the converter was processing more records after the backfill work. We still need to add exclusion filters there.
External API costs
We use LLM API calls for several features in the platform. One service makes an API call per record to validate field mappings during conversion. Another runs nightly to generate data quality summaries. A few others were experimental tools from earlier prototyping that were still hitting the API on a schedule. These API calls were costing more than Cloud Logging across both projects.
We reviewed each caller: how often it ran, how many API calls per run, and whether anyone was actually using the output. Two experimental tools had not had their output checked in over a month. The field validation service was running on every converter cycle, but we could batch the calls and reduce frequency without affecting quality. After removing the unused callers and batching the rest, the cost dropped 75%. The lesson here is the same as with schedulers: things that run automatically tend to keep running long after the reason for running them has passed.
Billing visibility: the foundation for everything else
We set up the GCP billing export to BigQuery in late April 2026. Before that, our only visibility was the monthly invoice and the GCP console billing dashboard. We could see the total was going up, but we could not see why. Once the billing export was in place, we could break down costs by SKU, by day, by service account. We could compare any two dates for a specific service and see exactly what changed. Every fix in this article started with a query against that billing export table. Without it, we would still be guessing.
The numbers
All numbers are from the GCP billing export. The daily cost is what matters, because it shows when each optimization took effect.
Daily cost trend (combined, both projects):
| Period | Daily cost | What happened |
|---|---|---|
| Baseline | 1x | Normal operations |
| Data volume growth | 1.5x - 2x | Backfill volume hits BQ costs |
| Peak day | 3x | Backfill volume + dedup cascade |
| BQ scans climbing | 2x - 2.8x | Scheduler scans compounding |
| BQ fixes deployed | 1.8x | Clustering + scheduler pauses |
| Cloud Run downsized | 0.7x | Utilization-based downsizing |
| HWM deployed | 0.6x | Converter BQ near-zero |
| Current | 0.6x | Lowest point so far |
From peak to current: an 80% reduction in daily cost, in six weeks.
Three optimization waves are visible in the data. First was BigQuery (clustering and scheduler pauses). Second was Cloud Run (utilization-based downsizing). Third was the high-water mark deployment.
Service-level changes (monthly):
| Service | Change |
|---|---|
| BigQuery (Enterprise Edition) | -41%, then deactivated (Jul: ~0) |
| Cloud Run | -26% |
| Cloud SQL | -21% |
| Secret Manager | -26% |
| Cloud Healthcare API | +24% (data ingestion spike) |
| Cloud Logging (one project) | -45% |
| External API | -75% |
Not everything went down, and that is worth understanding. Cloud Healthcare API increased 24%. The billing data explains why: upstream data providers were pushing large historical backfills during this period. One migration drove Healthcare API cost up 25x per day during the last two weeks of May. Another pushed daily Healthcare cost up 6x starting in early June. More records coming in means more FHIR writes, and that cost is proportional to actual data volume. It went up because the platform was doing what it was built to do.
Cloud Logging on one project increased 50% for a similar reason: more incoming data means more converter runs, more logs. We have not added exclusion filters there yet, so that is still on the list.
What we learned about the process
Start with the billing data. We did not guess which services were expensive. We exported billing data, queried INFORMATION_SCHEMA for scan volumes, and pulled utilization metrics from Cloud Run. Every optimization started with a number that told us where to look. The billing export showed us Secret Manager costs we would never have guessed. The utilization data showed us Cloud Run services at 1% capacity that had been running for months. Without the data, we would have started with the wrong services.
Schedulers are the silent cost multiplier. They get set up once and forgotten. We found scheduled queries running every 5 hours for dashboards refreshed once a day, and export queries running every 12 hours for data consumed once a week. The cost of each individual scheduler was small, but we had over 50 of them across both projects. Listing every scheduler alongside its actual consumer was the single most productive exercise we did.
The Cloud Run utilization review is a good example of how little effort some of these wins take. We spent about 30 minutes pulling p99 CPU and memory numbers for every service, sorted by gap between allocated and actual. That 30-minute exercise identified a large portion of total savings. The gap between "provisioned for the worst case" and "what actually happens" grows silently. We now plan to run this review monthly.
What is next
We still have optimization targets identified from the data. The FHIR export tables need partitioning by meta.lastUpdated so we can re-enable the dedup scheduler without recreating the scan cost problem. Cloud Logging on one project needs exclusion filters. Several Compute Engine VMs are running at low utilization. We also need to keep monitoring BigQuery on-demand costs to make sure they stay below what Enterprise Edition would have cost.
The daily cost is down 80% from peak. We are aiming to stabilize at the current level. The data will tell us when we get there.
What we run
We operate a federated health data platform deployed in two districts in Indonesia. The platform collects data from twelve different source systems: electronic medical records from hospitals, nutrition tracking from community health workers, TB surveillance from the national system, immunization registries, midwife apps, and several others. All of that gets converted into FHIR R4 resources and stored in a central FHIR datastore per district.
Each district runs its own GCP project. The converter runs on Cloud Run alongside about 30 other microservices per project. BigQuery handles analytics and dashboard queries. Cloud Healthcare API is the FHIR datastore. Cloud Scheduler triggers the converter and about 50 scheduled queries across both projects. Then there is Secret Manager for credentials and checkpoints, Cloud SQL for application databases, a handful of Compute Engine VMs for data science and tooling, and Cloud Logging collecting output from everything.
At peak, we were processing over 500,000 records per day across 80 source tables. Two projects, two districts, running 24/7. That is a lot of moving parts generating cost, and before the spike, we had never looked carefully at which parts were generating how much.
Following the data to find the cost drivers
We did not start by guessing. We set up the billing export queries, pulled Cloud Run utilization data, and looked at INFORMATION_SCHEMA.JOBS_BY_PROJECT to understand where the money was actually going:
SELECT
DATE(creation_time) AS dt,
ROUND(SUM(total_bytes_billed) / POW(2, 40), 2) AS tib_billed
FROM `region-asia-southeast2.INFORMATION_SCHEMA.JOBS_BY_PROJECT`
WHERE creation_time >= TIMESTAMP_SUB(CURRENT_TIMESTAMP(), INTERVAL 30 DAY)
GROUP BY dt ORDER BY dt
This query gave us daily scan volume in TiB, which we could plot against the billing export to see exactly when costs started climbing and why. BigQuery was roughly 36% of the total bill. Cloud Run was another 21%. But the percentages alone did not explain the spike.
What the data showed was a compounding pattern. Several upstream data providers had pushed large historical backfills, which is a normal part of operating a health data platform. Migrations happen. New data sources come online. One upstream migration brought in 427,000 rows in one week (43x the normal volume). Another pushed 2.27 million rows in a single week.
The system handled it correctly. The converter processed the records, and the FHIR store grew. The main resource table went from 150 GiB to 201 GiB. But a deduplication scheduler was scanning the full FHIR store three times per hour. At 150 GiB, each scan cost very little. At 201 GiB and growing, the same scheduler became the single most expensive query in the system. The scan volume data on one project tells the story clearly:
| Period | Daily BQ scan |
|---|---|
| Week 1 | 2.36 TiB |
| Week 4 | 4.0 TiB |
| Week 5 | 7.37 TiB |
| Week 6 | 9.26 TiB (peak) |
| After scheduler pauses | 1.34 TiB |
| After all fixes | 0.47 TiB |
From 2.36 TiB to 9.26 TiB in five weeks, without anyone changing a single scheduler configuration. The data volume changed, and the cost followed. That same pattern was happening on the other project too, going from 1.1 TiB to 2.85 TiB daily, though the absolute numbers were smaller.
What the data told us to do
Once we had the billing and utilization data in front of us, we split the work across BigQuery, Cloud Run, Compute Engine, and the smaller services. Each fix started with a specific metric.
BigQuery: query analysis pointed to the converter polling pattern
We used INFORMATION_SCHEMA.JOBS_BY_PROJECT to rank queries by bytes billed. The top consumer was the converter's polling query, which checks for unprocessed records using an anti-join against a multi-million-row ledger table. Every poll for every table scanned the entire ledger.
The scan data told us a CLUSTER BY table_name on the ledger, combined with adding a table_name filter to the JOIN clause, would let BigQuery prune 90% of the scan. The change was small:
-- Before: scans entire ledger on every poll
LEFT JOIN processing_report AS B
ON A.uuid = B.uuid
-- After: BigQuery prunes to just this table's slice (~94% fewer bytes)
LEFT JOIN processing_report AS B
ON A.uuid = B.uuid AND B.table_name = 'source_table_name'
After deploying that change, per-query scan dropped from 531 MiB to 53 MiB.
We then looked at the pattern more carefully. The data showed that on a typical hourly run, most tables have zero new rows. We built a high-water mark routing layer: if a table has no backlog, use a cheap timestamp query instead of the full anti-join.
def get_unprocessed_data(table_name):
"""Routes to HWM or anti-join based on backlog check."""
if table_name not in _HWM_READY_TABLES:
backlog = _check_backlog(table_name)
if backlog == 0:
_HWM_READY_TABLES.add(table_name)
else:
return _get_unprocessed_antijoin(table_name)
return _get_unprocessed_hwm(table_name)
The HWM path runs a simple WHERE timestamp > last_processed instead of joining millions of rows. This brought the converter's BigQuery cost down by over 95%.
BigQuery: evaluating Enterprise Edition after reducing query volume
Both projects were running BigQuery Enterprise Edition with autoscaling slots. Enterprise Edition pricing is based on slot usage, and when scan volume was high, the autoscaler was spinning up slots constantly. This was the single largest line item on the bill.
After deploying the clustering fix, the high-water mark routing, and pausing the expensive schedulers, our actual BigQuery compute usage dropped by over 90%. We looked at the numbers and asked whether Enterprise Edition still made sense. The autoscaling slots were now mostly idle, but we were still paying for the baseline. We deactivated Enterprise Edition on both projects and fell back to on-demand pricing.
The result was immediate. The projected BigQuery cost on on-demand is about 4% of what we were paying with Enterprise Edition. The query volume reductions we had already made meant that on-demand pricing was now far cheaper than even the minimum Enterprise Edition commitment. This single change, deactivating a pricing model that no longer matched our usage, was the largest cost reduction in the entire effort. There is a risk: if scan volume spikes again (another large backfill, or re-enabling the dedup scheduler), on-demand could become more expensive than Enterprise Edition was. We are monitoring on-demand costs weekly to catch that before it happens.
Scheduled query audit: frequency, dead joins, and dead code
We listed every scheduled query and its run frequency, then compared it against when the output data was actually consumed. The gaps were wide.
One dedup scheduler ran a UNION ALL full-scan of every FHIR resource table three times per hour. One scheduler, hundreds of dollars per month. We checked its downstream consumers and found it could safely be paused while we work on a better dedup approach.
We found schedulers with a dev- prefix running in the production project. One ran every 7 minutes, another ran every hour. They did not produce errors, so they never surfaced in monitoring. They only showed up when we audited the scheduler list against actual usage.
Export queries ran every 12 hours but were consumed once a day. Dashboard refresh queries ran every 5 hours but updated a table used in a daily report. We matched frequency to actual consumption patterns across both projects and paused or reduced 26+ schedulers.
Rewriting scheduled query SQL: the dead JOIN that ate 96% of slot-time
Frequency was not the only problem. Some scheduled queries were expensive because the SQL itself was inefficient.
Two scheduled queries that build a maternal health dashboard table started timing out at 6 hours. The SQL had not changed. The data had. Both queries contained a LEFT JOIN against a clinical orders table. The joined alias was referenced only in the ON clause, never in any SELECT or WHERE. It did nothing except multiply rows. The orders table stores one row per ordered lab test, so each visit record (full screening panel) fanned out 113 times. With 362,000 visit records, the intermediate result was 41 million rows before SELECT DISTINCT collapsed them back down.
This was harmless when the orders table was small. Then bulk conversion of two data sources landed 8 million new rows in one week. The fan-out blew past the 6-hour query timeout. When we queried INFORMATION_SCHEMA.JOBS_BY_PROJECT for slot usage, these two queries had consumed 96.1% of all project slot-time over the previous three days (62.6% and 33.5% respectively), leaving 3.9% for the 35,582 other queries.
The fix was one line: comment out the dead JOIN. Run time dropped from 6 hours (timeout) to 25 minutes. We also rewrote the partitioned update query to read from a 52 MB computed dashboard table instead of scanning 1 TB of raw FHIR tables directly. That cut its monthly cost from about USD 130 to USD 3.
The 4-billion-row table that should have been 12,000
One table had 4,040,397,903 rows. The actual data was 12,033 records.
A scheduled query had been running with WRITE_APPEND since February, duplicating every row on each run. The table grew to 764 GiB. This showed up when we sorted scheduled query outputs by storage size. We rebuilt the table from source, switching to WRITE_TRUNCATE. If we had not looked at storage sizes, this table would still be growing.
Cloud Run: utilization metrics vs. allocated resources
We pulled 30-day p99 CPU and memory utilization for every Cloud Run service across both projects. The gap between allocated and actual was large. One service had 4 CPU and 12 GiB allocated, with peak utilization at 12.3% CPU and 1.0% memory. Several services had similar profiles: provisioned for a load that never came, or provisioned for a peak that happened once and never returned.
Based on the utilization data, we downsized 44 services in one session, deleted 4 with zero traffic in 30 days, and switched 17 idle always-on services to request-based billing. The Cloud Run revision history shows it all happened in about 13 minutes. One project's daily compute cost dropped by 57% the next day.
We also right-sized the converter itself from 4 CPU / 16 GiB to 2 CPU / 4 GiB, and reduced the trigger frequency from every 30 minutes to hourly. During the backfill peak, the higher spec made sense. The converter was processing 571,000 records per day across 80 tables, running 20 worker threads with batches of 25 records. After the backfill cleared, utilization data showed the converter sitting well below capacity. We lowered workers and batch sizes to match the new steady state: 2 table workers, batches of 5, query limit down from 100,000 to 15,000.
Secret Manager: storage billing we were not tracking
We also went through every line item in the billing export, including services we assumed were cheap. Secret Manager was not cheap. Our patient deduplication service stores its checkpoint as a secret version, creating a new version on every run. Since late 2025, old versions were never destroyed. We found over 30,000 accumulated versions. The "Secret version replica storage" SKU was a top-10 line item on the bill. After cleanup, each checkpoint secret has 1 version. Cost dropped significantly.
Compute Engine: the GPU VM and the static IPs
Compute Engine was 6.6% of the bill. Most of that was expected: 10 VMs running various services (data science, form collection, OCR, migration tools). But one line item stood out.
A GPU VM with an NVIDIA L4 and 1 TB disk had been provisioned for OCR model fine-tuning. The fine-tuning work had finished weeks earlier. The VM was still running. It was the single most expensive Compute Engine resource. After confirming the fine-tuning was complete, we deleted it.
We also found unused static IP addresses being billed across both projects, some attached to services that no longer needed them. Small amounts individually, but they add up when nobody is watching.
Cloud SQL: fewer resources for the same workload
Cloud SQL was running multiple PostgreSQL instances 24/7. After reviewing actual query loads, we right-sized the instances. Total Cloud SQL savings: -36%.
Cloud Logging: paying for logs nobody reads
Cloud Logging on one project dropped 45% after reducing log levels and removing redundant logging. On the other project, logging actually increased 50% because the converter was processing more records after the backfill work. We still need to add exclusion filters there.
External API costs
We use LLM API calls for several features in the platform. One service makes an API call per record to validate field mappings during conversion. Another runs nightly to generate data quality summaries. A few others were experimental tools from earlier prototyping that were still hitting the API on a schedule. These API calls were costing more than Cloud Logging across both projects.
We reviewed each caller: how often it ran, how many API calls per run, and whether anyone was actually using the output. Two experimental tools had not had their output checked in over a month. The field validation service was running on every converter cycle, but we could batch the calls and reduce frequency without affecting quality. After removing the unused callers and batching the rest, the cost dropped 75%. The lesson here is the same as with schedulers: things that run automatically tend to keep running long after the reason for running them has passed.
Billing visibility: the foundation for everything else
We set up the GCP billing export to BigQuery in late April 2026. Before that, our only visibility was the monthly invoice and the GCP console billing dashboard. We could see the total was going up, but we could not see why. Once the billing export was in place, we could break down costs by SKU, by day, by service account. We could compare any two dates for a specific service and see exactly what changed. Every fix in this article started with a query against that billing export table. Without it, we would still be guessing.
The numbers
All numbers are from the GCP billing export. The daily cost is what matters, because it shows when each optimization took effect.
Daily cost trend (combined, both projects):
| Period | Daily cost | What happened |
|---|---|---|
| Baseline | 1x | Normal operations |
| Data volume growth | 1.5x - 2x | Backfill volume hits BQ costs |
| Peak day | 3x | Backfill volume + dedup cascade |
| BQ scans climbing | 2x - 2.8x | Scheduler scans compounding |
| BQ fixes deployed | 1.8x | Clustering + scheduler pauses |
| Cloud Run downsized | 0.7x | Utilization-based downsizing |
| HWM deployed | 0.6x | Converter BQ near-zero |
| Current | 0.6x | Lowest point so far |
From peak to current: an 80% reduction in daily cost, in six weeks.
Three optimization waves are visible in the data. First was BigQuery (clustering and scheduler pauses). Second was Cloud Run (utilization-based downsizing). Third was the high-water mark deployment.
Service-level changes (monthly):
| Service | Change |
|---|---|
| BigQuery (Enterprise Edition) | -41%, then deactivated (Jul: ~0) |
| Cloud Run | -26% |
| Cloud SQL | -21% |
| Secret Manager | -26% |
| Cloud Healthcare API | +24% (data ingestion spike) |
| Cloud Logging (one project) | -45% |
| External API | -75% |
Not everything went down, and that is worth understanding. Cloud Healthcare API increased 24%. The billing data explains why: upstream data providers were pushing large historical backfills during this period. One migration drove Healthcare API cost up 25x per day during the last two weeks of May. Another pushed daily Healthcare cost up 6x starting in early June. More records coming in means more FHIR writes, and that cost is proportional to actual data volume. It went up because the platform was doing what it was built to do.
Cloud Logging on one project increased 50% for a similar reason: more incoming data means more converter runs, more logs. We have not added exclusion filters there yet, so that is still on the list.
What we learned about the process
Start with the billing data. We did not guess which services were expensive. We exported billing data, queried INFORMATION_SCHEMA for scan volumes, and pulled utilization metrics from Cloud Run. Every optimization started with a number that told us where to look. The billing export showed us Secret Manager costs we would never have guessed. The utilization data showed us Cloud Run services at 1% capacity that had been running for months. Without the data, we would have started with the wrong services.
Schedulers are the silent cost multiplier. They get set up once and forgotten. We found scheduled queries running every 5 hours for dashboards refreshed once a day, and export queries running every 12 hours for data consumed once a week. The cost of each individual scheduler was small, but we had over 50 of them across both projects. Listing every scheduler alongside its actual consumer was the single most productive exercise we did.
The Cloud Run utilization review is a good example of how little effort some of these wins take. We spent about 30 minutes pulling p99 CPU and memory numbers for every service, sorted by gap between allocated and actual. That 30-minute exercise identified a large portion of total savings. The gap between "provisioned for the worst case" and "what actually happens" grows silently. We now plan to run this review monthly.
What is next
We still have optimization targets identified from the data. The FHIR export tables need partitioning by meta.lastUpdated so we can re-enable the dedup scheduler without recreating the scan cost problem. Cloud Logging on one project needs exclusion filters. Several Compute Engine VMs are running at low utilization. We also need to keep monitoring BigQuery on-demand costs to make sure they stay below what Enterprise Edition would have cost.
The daily cost is down 80% from peak. We are aiming to stabilize at the current level. The data will tell us when we get there.
Top comments (0)