A question on r/devops this week, asked in exactly these words: "How much money are you wasting on GPUs doing nothing?"
The asker's server has four 24GB cards. Two of them are spoken for:
A fine-tuned Llama 3 8B running a background data-extraction job
A Qwen model answering an internal bot
Combined traffic: about fifty requests an hour. But the weights sit in VRAM around the clock, so two whole cards are — their words — "held hostage doing almost nothing all day."
Now a new coding workflow needs a 70B model , and there is nowhere to put it. The choice on the table: ask for budget to buy more hardware, or pack both quiet models onto one card and free the other.
Held is not the same as busy
A GPU's cost does not care about your request rate. The card is bought, powered and occupied whether it serves fifty requests an hour or fifty thousand. What most teams measure is "is the model loaded?" What matters is "how much work did the card do?"
Before any purchase order, three questions:
What is the real request rate? Count it for a week, not a vibe. Peaks included.
What actually needs the VRAM? Two small models that idle most of the day can share one card — loading on demand or co-locating beats a card each.
What is the queue, not the load? If nothing ever waits, the card is a very expensive shelf.
The biggest tenant on most boxes is the thing doing nothing — an idle model here, a forgotten session there. Find it before finance finds the invoice.
My agents work the 3AM shift so no human has to. They also do not get to hold a whole GPU hostage for fifty requests an hour.
How many of your cards could pass the fifty-requests test?
Disclaimer
This post describes my personal experience with my own systems and accounts, in my own words. It is a first-person account of what I observed and what I do in my own work — not a statement of any company’s intentions, policies, or internal decisions. Where I do not know something, the post says so.
I am not affiliated with, endorsed by, or speaking for any company, product, or vendor named in this post. All product and company names are trademarks of their respective owners and are mentioned only to describe the experience accurately.
Any error messages, documentation excerpts, or figures quoted here are reproduced from my own records or from the public sources linked in the post, believed accurate at the time of writing. Product behavior changes over time; verify against current official documentation before relying on any detail.
Nothing in this post is professional, financial, legal, or security advice for your specific situation. Examples are illustrative — validate in your own environment before running anything in production.
Corrections are welcome. If any company named here publishes an official statement, or if any detail is wrong, I will update this post and note the correction. The goal is an accurate public record, not a verdict.
© 2026 Honey Kiran. All rights reserved.
Honey Kiran · DevOps & Platform Engineering
AI agents transforming DevOps, infra, platform, security & SRE. The shift nobody works anymore: deep dives, 3AM Files, and OpsBites.
Examples are illustrative — validate in your own environment before running in production.

Top comments (0)