GPU spend behaves differently from general cloud compute. Rates run from roughly $0.35 to $7 per hour depending on the hardware, the provider, and what you committed to, and the same training job can cost an order of magnitude more or less depending on where it runs and how well it uses the card.
The tracking problem is not the rate. It is that spend scatters across experiments, jobs, and teams, and cloud billing dashboards report it hours or days after the fact. By the time the number is visible, the budget is already spent.
Short answer
GPU cost tracking needs two layers. An internal observability layer attributes spend to pods, experiments, and models so engineers can find waste. A metering and billing layer converts GPU seconds into customer-facing charges with margin visible per account. Cloud-native billing tools cover neither well: they lag, they aggregate by resource rather than experiment, and they report cost without utilisation.
What to actually measure
Four metrics carry most of the signal.
GPU hours consumed, broken down by job, experiment, model version, and team. The aggregate number tells you nothing actionable.
Utilisation rate. A card allocated at 100% and used at 20% costs the same as one used well. This is where the waste hides, and it does not appear on an invoice.
Cost per model or experiment. The unit that lets you compare a hyperparameter sweep against the result it produced.
Idle spend. Reserved capacity, failed jobs that held allocation, and instances left running after a notebook session ended.
Why cloud-native billing tools are not enough
AWS, Google Cloud, and Azure all provide billing dashboards with tagging and labelling, and for an early team that is a reasonable starting point. Four limitations show up as operations scale.
Data lag. Native billing updates hours or days behind actual usage, which rules out enforcing a budget or reacting during a run.
Attribution granularity. Spend aggregates by instance and tag. Splitting it across individual experiments, model versions, or the specific hyperparameter run that went wrong is manual work when dozens run in parallel.
No utilisation context. Billing shows what was paid, not how efficiently the hardware was used. Idle time, failed jobs, and oversized instances all bill identically to productive work.
Fragmented views. Data scientists look at model metrics, infrastructure teams look at Kubernetes dashboards, finance looks at the cloud invoice. Three sources, no agreement.
There are secondary frictions too. Egress charges accumulate quietly when data moves out. Fixed VM and GPU pairings mean paying for CPU and memory you did not need. And on shared infrastructure, performance variability means the same job costs differently on different days without any change on your side.
Layer one: internal cost observability
Kubecost allocates cost inside Kubernetes, mapping spend to pods, namespaces, and deployments. With NVIDIA DCGM Exporter it surfaces GPU efficiency and idle spend, supports custom pricing sheets for on-prem hardware, and is OpenCost compliant. This is the right tool when workloads run on Kubernetes and the question is which namespace is burning the budget.
Weights and Biases logs GPU utilisation alongside runtime, job status, and model performance. Because the resource data sits next to the experiment metadata, inefficient runs are visible in the same view as the results they produced.
MLflow does not track cost natively, and that is sometimes the point. GPU hours and computed cost can be logged as custom parameters on a run, which unifies model outcomes and resource consumption inside an existing experiment workflow with no new platform.
Run:AI handles orchestration and scheduling for GPU clusters, with real-time allocation tracking and job-level metrics. It integrates with Kubernetes, Kubeflow, and MLflow, and it is aimed at the case where the primary problem is packing expensive hardware efficiently.
Pick based on the actual question: Kubernetes attribution, experiment-aware logging, custom tracking inside an existing pipeline, or cluster orchestration.
Layer two: metering and billing
The observability layer answers where the spend went. It does not answer what a given customer should be charged, or what the margin on that account is.
That requires the GPU consumption to become a metered event tied to a customer, priced, and invoiced.
import os, time, uuid, requests
def run_job(customer_id, model, fn, *args):
start = time.monotonic()
try:
return fn(*args)
finally:
elapsed = time.monotonic() - start
requests.post(
"https://api.cloud.flexprice.io/v1/events",
headers={"x-api-key": os.environ["FLEXPRICE_API_KEY"]},
json={
"event_id": str(uuid.uuid4()),
"event_name": "gpu_seconds",
"external_customer_id": customer_id,
"properties": {
"model": model,
"gpu_seconds": round(elapsed, 3),
"device": "a100-80gb",
"device_weight": 2.5,
"experiment": os.environ.get("EXPERIMENT_ID"),
},
},
timeout=2,
)
Emitting in a finally block matters. A job that crashes still consumed GPU time, and a metering path that only records successful completions systematically undercounts.
The experiment property is what makes the same event stream useful for both layers: billing aggregates by customer, internal reporting aggregates by experiment.
Once events arrive, aggregations turn GPU seconds into a billable metric, and cost sheets hold what the compute actually costs so margin is computed per call, per model, and per account rather than reconstructed at month end.
Pricing GPU workloads
Raw GPU seconds are an honest metric and a difficult one to sell, because a customer cannot predict them. Three patterns work better in practice.
Credits with a per-model burn rate. The customer buys credits, and each model consumes them at a rate reflecting its cost. One number to watch, and the rate card absorbs the hardware differences. Wallets handle prepaid, promotional, and recurring grants with separate expiry rules.
Base fee plus metered overage. Included capacity for predictability, per-second charges above it.
Weighted units. GPU seconds multiplied by a device coefficient, so an H100 second and a T4 second are not the same billable unit. Send the coefficient on the event as a property, then a custom expression such as gpu_seconds * device_weight computes the billable quantity per event and sums it. One meter covers every device type.
Whichever model you pick, the enforcement question follows immediately. When a customer's budget runs out mid-training, the options are stop the job, continue and bill the overage, or refuse new jobs while letting running ones finish. That is a product decision, and it should be configuration rather than a code path.
Making the numbers agree
The recurring organisational failure is three teams holding three different cost figures. Engineering sees utilisation, finance sees the cloud invoice, and the customer sees a number neither of them can reproduce.
One event stream fixes that. The same GPU seconds feed internal attribution, customer-facing usage, and the invoice, so a disputed charge resolves by looking at events rather than by arguing between dashboards. Customers get usage widgets and threshold alerts against the same data.
Flexprice is enterprise-grade, open source usage based billing infrastructure for AI and SaaS companies. It can be deployed in your own VPC, on-prem, or on Flexprice's managed cloud, and all three run the same engine. For teams running their own GPU fleet, that means usage and revenue data can stay entirely inside your own infrastructure and never reach a vendor's cloud.
Wiring it up
The integration path is short because the usage data usually already exists. Connect the existing metrics source, whether that is Kubernetes, DCGM, job logs, or an API. Define the meters and pricing rules as configuration. Then attribute and invoice by customer, project, or experiment from the same stream.
Getting started
AI cost tracking covers the cost sheet model and how margin is computed per model and per customer, and ingesting AI usage covers the event shape. The source is on GitHub.
Top comments (0)