Scoring a boosted tree ensemble is cheap per row and expensive in aggregate, and almost nobody knows which of their parameters is driving the bill. It is the tree count. Here is the arithmetic that shows why, with every input stated so you can substitute your own.
Every price below is an assumption, not a quote. Cloud compute prices vary by provider, region, instance family and commitment, and they change. The number to keep from this page is the structure of the calculation; substitute the price from your own bill and rerun it.
The cost model
Predicting one row with a gradient boosted ensemble means walking every tree from root to leaf and summing the leaf values. The work per row is therefore
comparisons_per_row = n_trees x average_depth
total_comparisons = n_rows x n_trees x average_depth
wall_clock_seconds = total_comparisons / (comparisons_per_second_per_core x cores)
cost = (wall_clock_seconds / 3600) x cores x price_per_vcpu_hour
Three properties of that expression are worth naming before any numbers go into it.
It is linear in rows, linear in trees, and linear in depth. Nothing here is superlinear, which is why boosted-tree batch scoring stays predictable as it scales — unlike a neural model, where batching effects and memory transfers dominate.
The feature count does not appear. A tree of depth 6 makes six comparisons regardless of whether the table has 20 columns or 2,000. Feature count affects how long it takes to assemble the input batch, which is a data-engineering cost rather than a scoring one, and it is frequently the larger of the two.
And the tree count is the only term in the expression that you chose during model selection. Row count is the business. Depth is bounded and usually single-digit. Tree count ranges from 100 to 5,000 depending on a learning rate you picked, which makes it a 50-fold lever.
The arithmetic, with every input named
A nightly scoring job over 50 million rows. Stated inputs, all of them assumptions except the ones you can read off your own model:
- Rows: 50,000,000 per night.
- Trees: 800, which is what early stopping kept at a learning rate of 0.05.
- Average depth: 6.
- Throughput assumption: 20 million node comparisons per second per core. This is the input most worth replacing with a measurement of your own; it is an order-of-magnitude figure for a modern x86 core running a compiled predictor on batched, contiguous float32 input, and it can be off by 3× in either direction depending on cache behaviour.
- Cores: 16, on one instance.
- Price assumption: $0.04 per vCPU-hour. Substitute your own; on-demand general-purpose instances and committed-use discounts differ by more than a factor of two.
comparisons_per_row = 800 x 6 = 4,800
total_comparisons = 50,000,000 x 4,800 = 240,000,000,000 (2.4e11)
core-seconds = 2.4e11 / 2.0e7 = 12,000 core-seconds
wall clock on 16 = 12,000 / 16 = 750 s (12.5 minutes)
vCPU-hours = 12,000 / 3600 = 3.33
cost = 3.33 x $0.04 = $0.13 per night
per million rows = $0.13 / 50 = $0.0027
per year (365) = $0.13 x 365 = $49
The headline is that the model scoring itself is close to free. Fifty million rows for about thirteen cents of compute, at the stated price. Anyone whose batch scoring bill is materially larger than this is not paying for tree traversal, and the next two sections are about what they are paying for.
Change one input to see the sensitivity. Drop the learning rate to 0.02 and early stopping might keep 2,500 trees instead of 800: total comparisons go to 7.5e11, wall clock to 39 minutes, cost to $0.42 a night, $153 a year. Same data, same accuracy to within a fraction of a point, three times the bill. That is the trade named in gradient boosting hyperparameter tuning made concrete.
What actually sets the throughput
The 20-million-comparisons-per-second assumption hides most of the engineering. Four things move it, and they move it a long way.
Batching. Calling predict() once per row from Python costs one interpreter round trip per row, and the per-call overhead swamps the traversal entirely — single-row calls in a loop are commonly two orders of magnitude slower per row than one call on a large array. Score in chunks of tens of thousands of rows.
Memory layout. The predictor walks trees for a batch of rows; contiguous float32 in the layout the library expects avoids a copy and a dtype conversion of the entire batch. A pandas DataFrame with mixed dtypes will be converted, and that conversion can cost more than the prediction.
Compiled predictors. Tools that compile an ensemble to native code — Treelite, ONNX Runtime with a tree ensemble operator, or a library’s own C++ predictor — remove the per-node branching overhead of a generic traversal. This is where the largest single-digit multipliers come from, and it is worth measuring rather than assuming.
Cache behaviour. A large ensemble does not fit in L2. The traversal order — all trees for one row, versus one tree for all rows — changes the hit rate substantially, which is why library predictors process a block of rows through a block of trees.
The costs that are not the model
In a real nightly job, the thirteen cents is rarely the line item. The things around it usually are:
- Reading the input. 50 million rows × 200 columns × 8 bytes is 80 GB uncompressed, perhaps 15–25 GB as compressed Parquet depending on the data. Object-storage requests, egress if it crosses a boundary, and the decompression itself are all real.
- Building the features. The joins and window aggregations that produce those 200 columns are usually a warehouse query, and warehouse compute is billed at a much higher rate than plain vCPUs. On most teams this dominates by an order of magnitude or more.
- Writing the output. 50 million predictions plus keys, plus whatever index or table update the consumer needs.
- Orchestration and idle time. An instance held for a two-hour window to do twelve minutes of work is billed for two hours. This alone can make the scoring step cost ten times its compute.
The general treatment of pricing a per-unit inference operation is in cost per request; the structure is the same even though the per-unit numbers here are three or four orders of magnitude smaller than for a hosted language model.
The levers, in order of effect
- Fewer trees. Directly proportional. Raise the learning rate, tighten early stopping, or prune the ensemble to the first N trees and measure what the metric actually loses. Often it is nothing.
- Batch properly and use a compiled predictor. The same model, the same accuracy, a multiple of the throughput.
- Score fewer rows. Most batch jobs re-score rows whose features have not changed since last night. Scoring only changed entities is frequently a 90% reduction and costs nothing in accuracy.
- Right-size and release the instance. Twelve minutes of work should not hold a machine for two hours.
- Only then, shrink the model. And weigh it against what an added model would cost — stacking a second model roughly doubles every term in the expression above.
Top comments (0)