DEV Community: DigitalOcean

How do I cap inference billing to avoid surprise costs from idle GPUs or runaway agents?

DigitalOcean — Wed, 14 Oct 2026 16:31:01 +0000

You cap inference billing by layering controls, not by flipping one switch. On DigitalOcean, Serverless Inference is prepaid, so your balance is a real ceiling, and scoped keys, alerts, and cleanup jobs cover the rest. Spend alerts alone won't stop anything: they email you when spending crosses a threshold, but your usage keeps running.

Key takeaways:

Serverless Inference stops when funds run out, so fund a dedicated balance with only what you plan to spend.
Dedicated Inference and GPU Droplets bill by the hour until destroyed, so automate cleanup.
Agents need caps in your own code, because rate limits slow spending but don't cap it.

Use prepaid balances as a ceiling

Serverless usage draws from promotional credits, then the Inference and Agents balance, then your account prepayment balance. Only Serverless Inference and Managed Agents draw from the first, so it works as a separate AI budget. Auto-reload is on by default, so turn it off for a hard stop. A zero balance is also an outage, so reserve this for experiments and non-critical workloads.

Add alerts and scope your keys

Spend alerts email you at percentage thresholds of your monthly budget, within an hour of each threshold being crossed. Their sole purpose is notification. Give each service its own model access key, scoped to specific models and a VPC, so one leak or loop has a small blast radius. Regenerating a key takes effect immediately, which is how you want it to work during an incident.

Cap agents in code

Rate limits (up to 120 requests and 1.5M tokens per minute at the lowest serverless tier) limit speed, not total spend. Harness Runtime has no per-session or per-product spend limit, so an agent stuck in an unintended loop can quickly drain balances. Set a max output token count per call, a max step count per task, and a running spend counter that stops the run. You should also pause idle sessions, since paused sessions incur no compute charges.

Stop paying for idle GPUs

Dedicated Inference bills per GPU-hour, and GPU Droplets bill per second until destroyed. Powering off doesn't stop the meter. At \$30.32 per hour for 8x H100, an idle Dedicated Inference endpoint costs roughly \$728 a day.

To cap inference billing and avoid surprise costs, tag experimental GPU Droplets and destroy them nightly:

doctl compute droplet delete --tag-name ephemeral-gpu --force

Inference billing cap FAQ

Can I get usage alerts or hard spending caps on a managed AI platform before bills run away?

DigitalOcean spend alerts notify you at budget thresholds but don't limit usage. The closest hard cap is Serverless Inference's prepaid model, where access is suspended when credits and balances run out.

Does powering off a GPU Droplet stop billing?

No. The GPU stays reserved, so billing continues until you destroy the Droplet.

Which inference option doesn't penalize variable workloads?

Serverless Inference, which bills per token instead of reserving GPUs, is a solid choice for variable workloads. Dedicated Inference bills per GPU-hour, whether or not anyone calls it, so it suits steady traffic better than spiky traffic.

References & further reading

Spend alerts: worth setting up first.
Manage Serverless Inference prepayment: explains the order balances are drawn down.
Learn more about model access keys and inference limits.
Inference pricing: worth checking directly before you build.
Droplet pricing and billing
Harness Runtime pricing: worth bookmarking before you run agents at scale.

We moved a production backend from a phone

Vinay K — Tue, 13 Oct 2026 13:19:33 +0000

Published by Vinay Kumar, Chief Product and Technology Officer at DigitalOcean

Every era of the cloud has a unit of work. For agents, it’s a human who directs, a controller who runs the operation, and a worker who does the heavy lifting. Last week we watched all three move a production backend, from a phone.

I’ve spent fifteen years building cloud services: S3 and EBS in the early days of AWS, Oracle Cloud Infrastructure from inception, and now the agentic cloud at DigitalOcean. In every era, moving a backend meant the same thing: a project plan, a maintenance window, and a team working late into the night.

Last week, Rohit Bhattacharjee on our team moved an entire production backend from his phone. Database, APIs, file storage, and about 31,000 AI search embeddings. He never opened a laptop. A message while waiting for coffee. A decision between meetings. By the end of the week, it was done.

This wasn’t a demo. It was a real workload with a bill.

Three parties, three jobs

The worker was a DigitalOcean Managed Agent, an AI that does real work in the cloud, not just chat. It ran in our Harness Runtime: a private, isolated computer that spins up for each session, with a workspace that persists when the agent pauses. Through our Action Gateway, a single connection to 20,000+ tools, it could use GitHub, Supabase, a web browser, and the whole DigitalOcean API.

The agent never saw a password. The gateway applies credentials at the moment of use, so they never reach the model.

It provisioned the database, loaded all 40 access policies, and mapped every Supabase feature to its DigitalOcean counterpart. It wasn’t advising on a migration. It was performing one.

The controller was a personal agent running on his phone, the agent Rohit actually talked to. He ran the migration in plain chat, and the personal agent turned his messages into work: tasking the worker, tracking the run, and reporting back. When he asked it to check a claim, it did. The worker said 145,432 rows had been loaded, so the controller counted them. Validation was the side job. The main job was holding the whole operation together in one conversation, so a person never had to leave the flow of his day.

The worker and controller come from different model families. That’s the point of an open platform: you choose the right model for each job.

Rohit provided the credentials, his judgment, and a credit card. He reviewed a $180 proposed plan and saved almost 80% on a revision with just a few taps. Everything else, the two agents handled between themselves.

What moved

The workload was huffl.ai, an agentic canvas that puts docs, spreadsheets, whiteboard, presentations, and search in one place. Underneath: 31 tables, 145,432 rows, three storage buckets, 24 functions, and 30,650 embeddings, the numeric fingerprints that make search work.

Before moving anything, the worker read all of it and caught two problems that would have poisoned the migration. Two overloaded functions that looked like duplicates. A “vector count” that was really measuring string lengths.

The new home: managed Postgres ($15), one App Platform app running four services ($20), and one object store ($5). About $40 a month. The data moved with per-table parity checks, and every one of the 30,650 embeddings verified non-null.

Pay for the work, not the wait

An agent spends most of its session waiting: on a model, a tool, or a person. The runtime paused the worker whenever it waited and resumed it in about 300 milliseconds. While it slept, the meter stopped. We bill for active CPU, not provisioned capacity. The agent runtime for the entire run cost a few dollars.

What this changes

Moving a backend used to mean a team, a plan, a maintenance window, and a war room. Last week it meant chat bubbles. The human says what needs to happen, the controller runs the operation, the worker does the work, and every claim gets checked along the way.

The work didn’t get smaller. What one person can accomplish got bigger. See it in action: watch the video:

Try DigitalOcean Managed Agents today. Bring your preferred harness, connect your tools, and give your agents a complete environment with our Cuttlefish mobile application coming soon. Get started today.

What would you trust an agent to move for you, and what would you want checking its work?

Jev: an AI model that can’t write a sentence

DigitalOcean — Fri, 09 Oct 2026 14:00:00 +0000

ICYMI AI: The latest AI trends, delivered to you on Dev.to

Tech Talk: TypeSafe’s Jev is a model that only returns probabilities

TypeSafe's Jev, released into early access on September 15, is a general-purpose classification model. It reads up to 32K tokens of input (such as a support ticket, a customer record, or a row of data). It answers any number of typed questions about it in a single parallel pass, and returns probabilities rather than prose.

Outputs are limited to three options: Choice, Score, and Noul (yes or no), so structured-output errors and bad tool calls drop to zero by construction. A valid answer isn't always the right one, and Jev doesn't claim otherwise.

What it does is tell you how sure it is. TypeSafe trains it for calibration, so its probabilities match real-world frequencies. When Jev says it's 80% confident, it's right about 80% of the time. That lets you set a threshold, automatically acting on high-confidence calls and routing the rest to a human or a larger model.

The pricing is what makes Jev worth a look. At \$0.042 per million input tokens with free output, TypeSafe estimates about \$0.0004 per decision, roughly 1/76th the cost of GPT-5.6 Terra on its benchmark. Within days, imitators appeared: Laya, Bespoke Nimble, and Kev, several of them open-weight fine-tunes of Qwen 3.5 you can run locally. OpenAI likewise followed with a Decisions API of its own.

For developers, the change is in how you structure the call. Instead of sending an LLM one broad question like "is this spam?" and parsing a paragraph back, you pass Jev the email once, along with a set of narrower typed questions, such as: Does the sender domain match the claimed company? Does it ask for credentials? How urgent is the language? Each comes back as a probability. Your own code then applies the thresholds and makes the call, so the routing policy lives in your repo, versioned and auditable, and the model only scores the inputs.

Jev is available today on Serverless Inference through the DigitalOcean Model Library, at Typesafe’s list price—no separate TypeSafe account needed. Pair it with the DigitalOcean Inference Router to send routing and classification traffic to Jev while keeping frontier models in reserve for the calls that need them.

Event calendar

Open Intelligence Summit

October 12-13th, San Francisco

The argument is no longer “Is open source AI cheaper?” It’s now “Who owns intelligence?” A movement is forming around open intelligence and generating the belief that the intelligence powering our AI products and companies should be more open, portable, and controllable.

Join DigitalOcean, NVIDIA, RadixArk, Inferact, and more for deep dives, demos, and discussions about the technologies that keep intelligence portable, interoperable, and in the hands of builders.

Industry news

Near-flagship intelligence at a fifth of the price was the headline at OpenAI DevDay, but the bigger shift is the stack forming around the model.

OpenAI announced GPT-6.1 Sol, which it says delivers near-Astra intelligence at a fifth of Astra's token prices, plus an Ultrafast tier with up to 6x faster generation in the API. It also added computer use to its Agents API and opened ChatGPT, which it puts at 1.2B weekly users, to developer-built plugins and partners like Devin, Notion, and Vercel.

Agent safety is moving below the application layer.

NVIDIA's Open Agent Safety Platform pairs OpenShell, an open-source runtime that traces agent actions and enforces policy, with Sentry, an out-of-band watchdog on BlueField-4 DPUs that NVIDIA says can quarantine an agent that leaves its boundaries within milliseconds. NVIDIA says more than 100 organizations, including Anthropic and CrowdStrike, are working with the technology, and DigitalOcean is among the companies supporting the platform.

New models available on DigitalOcean Inference Engine

We recently launched models from TypeSafeAI, DeepSeek, Z.ai, NVIDIA, OpenAI, and Anthropic, all with different strengths and use cases. Learn more about the models in our What’s New blog.

Start using them through the DigitalOcean API or directly in the Cloud Console.

DeepSeek-V4.1-Flash

DeepSeek-V4.1-Flash cuts the KV cache footprint to 1/4 of V4-Flash while outperforming DeepSeek's own V4-Pro flagship on coding and agentic benchmarks, addressing the cache-hit charges that often dominate agent costs. The 552B MoE model activates just 8B parameters on input, delivering frontier performance with infrastructure economics that improve as you scale.

GLM-5.3 and GLM-5.3-Flash from Z.ai

Z.ai's post-training translates to significant performance improvements for GLM-5.3, achieving a 50% lower cost per completed coding task than GLM-5.2 while jumping to 28.3 on Terminal-Bench 3.0. GLM-5.3-Flash adds native image and video input with the same agentic strength, giving teams both reasoning-heavy and efficient multimodal paths on a single platform.

Nemotron™ 3 Diarization from NVIDIA

NVIDIA's Nemotron™ 3 Diarization replaces the multi-stage speaker attribution pipeline (VAD, embeddings, clustering, post-processing) with a single 100M-parameter model, keeping audio within DigitalOcean's infrastructure rather than routing it to third-party APIs. It identifies up to 8 speakers (double the previous limit) with a 14.72% error rate.

GPT-6.1 Sol, GPT-6 Astra, Sol, and Luna from OpenAI

One week after GPT-6 Sol’s launch, OpenAI releases GPT-6.1 Sol, which offers near-Astra-level coding and substantial improvements in document understanding, computer use, and scientific work. OpenAI's three-tier GPT-6 family also debuts on DigitalOcean: Astra for computer use and coding, Sol for frontier reasoning at 90% lower cost than Opus 5, and Luna for high-volume tasks.

Claude Fable 5.1, Opus 5.5, and Sonnet 5.5 from Anthropic

Anthropic's latest point releases deliver focused gains: Fable 5.1 reaches 55.8% on Terminal-Bench 4.0 with 75% cheaper cache reads, while Opus 5.5 matches high-effort results at default effort using half the tokens and 20% lower per-token cost. Sonnet 5.5 runs 30% faster with equivalent cost savings, with a shift toward better token efficiency and lower verification burden.

Community tutorials

How naive failover between inference providers makes outages worse

In a six-hour fault-injection run on an H200 GPU Droplet, the standard retry-with-backoff policy pushed 3.54x normal load onto a degraded provider, while a retry budget plus circuit breaker held it to 1.02x, and naive retries left 3,743 zombie requests generating 959,232 output tokens for clients that had already disconnected. Learn how to classify inference errors before failing over, replace fixed retry counts with a budget, and verify whether your provider actually stops billing when you cancel a stream.

What vision-language models cost to serve at production scale

A single uploaded photo can add more than a thousand input tokens before the model writes a word, and at 500,000 requests a month, that gap is the difference between \$155 on serverless Nemotron Nano 12B VL and \$3,175.20 for a dedicated H100 running 24/7. Walk through the per-request cost math, the roughly 10.2 million-request breakeven, where dedicated wins, and six levers for cutting image-token spend.

Speculative decoding's hidden cost: the KV cache capacity you give up, and where throughput turns negative

On a 141 GB H200 serving Qwen3-14B with vLLM, a separate Qwen3-0.6B draft model consumed about 43.8 percent of baseline batch slots, and even n-gram speculation, predicted to cost nothing, measured a 2.8 percent capacity tax. See where each method's throughput crosses below baseline as concurrency rises, and get the config.json arithmetic to predict a draft model's KV cost before spending GPU hours.

Cache reliability is a pricing term: what I learned measuring DeepSeek-V4.1-Flash on DigitalOcean Serverless Inference

DeepSeek-V4.1-Flash has the deepest cache discount in the Serverless Inference catalog at \$0.006 per million cached tokens, yet across 844 API calls it hit the cache on only 78.6% of warm turns, and with a miss priced at 50x a hit, it cost 2.2x what GLM-5.3-Flash did for the same 15-turn, 200K-token sessions. Discover how DigitalOcean's 64-token-block prefix cache works, how to log your own hit rate from the usage object, and why reasoning_effort: max cost up to 1.88x more without improving accuracy.

Why DeepSeek-V4.1-Flash splits reading from writing

DeepSeek-V4.1-Flash's Causal Encoder-Decoder splits a 40-layer model into a 20-layer encoder that runs prefill at 8B active parameters and a full stack that decodes at 16B, shrinking the global KV cache to about 890 bytes per token, roughly a quarter of V4-Flash's. Learn why that asymmetry, not the 74.2 versus 62.7 DeepSWE score against V4-Pro, is what lowers the bill for read-heavy coding agents.

Product updates

Introducing DigitalOcean Managed Agents, now in Public Preview

AI agents are ready to take on more, and we built the infrastructure to match. DigitalOcean Managed Agents brings together Harness Runtime, a purpose-built runtime for running agent sessions at scale, and Action Gateway, governed access to the tools agents need to get real work done, all in one platform.

Deploy your preferred agent harness, Claude Code, Codex CLI, OpenCode, and more, or bring your own, and connect to 16,000+ tools across 500+ providers through a single managed endpoint. Sessions are ready in under a second, resume in about 300 milliseconds with full context intact, and you only pay for the compute you actually use.

Whether you're automating support workflows, refactoring code across repositories, or building the next agent-powered product, Managed Agents gives you the infrastructure to do it, without building it yourself.

DigitalOcean MicroVMs now in private preview

Isolated, Firecracker-based VMs that pause when idle and resume with state intact, built for agent workloads that run in bursts. The same layer that powers Managed Agents is now available directly to teams building their own agent platforms, sandboxes, and short-lived jobs.

What is disaggregated prefill and decode in LLM inference?

DigitalOcean — Tue, 06 Oct 2026 18:03:00 +0000

What is disaggregated prefill and decode in LLM inference?

Prefill is compute-bound, decode is memory-bound. Learn why colocating them hurts TTFT and ITL, and when disaggregated serving pays off.

Every LLM request runs through two phases with effectively opposite hardware profiles. Understanding the split explains most of the latency and cost behavior you see in production, and why disaggregated prefill and decode has become a standard serving pattern.

Key takeaways:

Prefill is compute-bound and sets TTFT; decode is memory-bandwidth-bound and sets ITL/TPOT.
Running both on the same GPUs creates prefill/decode interference, visible as TTFT spikes and ITL jitter.
Chunked prefill and continuous batching reduce interference; disaggregating the phases onto separate pools removes it, at the cost of a KV cache transfer.
Disaggregation is worth it for long prompts, high concurrency, and strict streaming SLOs. A managed option like DigitalOcean Serverless Inference gives you it without the setup; for small self-hosted deployments, a well-tuned single pool on a GPU Droplet running vLLM is often enough.

The two phases of LLM inference

Prefill processes the entire prompt in one forward pass and builds the KV cache, the attention state every later token reads from. It is compute-bound: arithmetic intensity is high and the GPU spends its time doing math, not waiting on memory. Prefill governs time to first token (TTFT), and its attention cost grows quadratically with prompt length.

Decode generates output one token at a time. Each step reloads the model weights and the growing KV cache from HBM to produce a single token, so it is memory-bandwidth-bound. Decode governs inter-token latency (ITL) and time per output token (TPOT). A GPU with higher memory bandwidth streams tokens faster even if its FLOPs are unchanged.

Why colocating prefill and decode causes latency problems

Most serving engines batch prefill and decode on the same GPUs. That is where the interference shows up: a long prompt's prefill can monopolize compute while dozens of in-flight decode requests stall, and users see ITL jitter or a TTFT spike. The two phases also want different batch sizes and parallelism strategies, but colocated they must share one configuration.

Chunked prefill and continuous batching soften this by interleaving prompt chunks with decode steps, which is why our inference trilemma guide recommends them for latency-sensitive deployments.

What disaggregation changes

Disaggregated serving, formalized in the DistServe paper, runs prefill and decode on separate GPU pools. A prefill worker computes the KV cache, ships it over a fast interconnect (RDMA, RoCE, or a transfer library like NIXL or LMCache) to a decode worker, and the decode pool streams tokens without prefill interference. Because each pool is sized and tuned independently, you can hit TTFT and ITL targets at the same time rather than trading one for the other. The cost is added complexity plus a KV transfer step; that transfer can land on second-token latency rather than TTFT depending on how orchestration is done.

This is the architecture DigitalOcean already runs in production: Serverless Inference uses disaggregated serving over a RoCE network, paired with prefix-aware routing so shared system prompts skip redundant prefill entirely.

References & further reading

DistServe: Disaggregating Prefill and Decoding for Goodput-optimized LLM Serving: the primary source for the pattern, worth reading before you build.
LLM inference benchmarking: measure what matters: a clear walkthrough of TTFT, TPOT, ITL, and how each maps to prefill or decode.
The LLM inference trilemma: practical tuning advice, including when to reach for chunked prefill.
Learn more about debugging p99 TTFT and deploying NVIDIA Dynamo on GPU Droplets, a framework built around disaggregated serving.
GPU Droplets: current hardware and regions, worth checking directly since availability changes.

How to route LLM requests by cost vs. latency

DigitalOcean — Fri, 02 Oct 2026 18:33:17 +0000

Routing LLM requests by cost and latency means sending each request to the cheapest or fastest model that still meets the quality bar, rather than hardcoding a single model for everything. Production traffic isn't uniform: routine lookups, complex troubleshooting, and background jobs have different cost and latency requirements, even within a single app.

Using an inference router like DigitalOcean Inference Router involves defining request categories, assigning each a selection policy (cost, latency, or a benchmarked best fit), and setting fallback models so an unavailable model doesn't break the request. Cost is the price per token (which can vary by up to 100x across a model catalog); latency is mainly the time to first token (TTFT). The two don't move together, so route per task rather than using a single blended score.

Key takeaways:

Routing is a per-workload policy, not a search for "the best model."
Cost and latency usually need separate, explicit priorities.
Fallback and cache-aware routing protect the savings that routing is meant to deliver.
The DigitalOcean Inference Router implements this as configuration, not custom infrastructure.

LLM routing strategies

Static rules: Hardcode model per request type. Simple, but brittle.
Policy-based routing: A model pool per task with a cost/latency/optimal policy. Most production systems land here.
Dynamic routing: A classifier infers the task and policy automatically.

Routing LLM requests: A quick decision framework

Workload	Priority	Policy
Real-time chat	Latency	Speed-optimized (TTFT)
Routine lookups	Cost	Cost-optimized
Complex reasoning	Quality	Benchmarked "optimal"
Batch/background	Cost	Cost-optimized

Fallback models keep requests completing when the preferred model is down or rate-limited.

Cache-aware routing matters too: switching models to save a fraction of a cent can break a cached prompt and cost more overall.

How the DigitalOcean Inference Router routes LLM requests

Inference Router tasks pair a model pool with a selection policy: preset tasks default to a benchmarked Optimal policy; custom tasks choose Cost Efficiency, Speed Optimization (TTFT), or Manual Ranking.

Fallback models handle unmatched traffic. X-Model-Affinity preserves cache reuse, and X-Routing-Max-Switch-Spend-Pct (default 20%) caps the cost of switching mid-session.

It's a drop-in change. Set "model": "router:your-router-name". Routing decisions typically resolve in about 200ms, billed at the serving model's standard rate with no separate router charge during public preview.

LLM routing FAQ

What is LLM routing?

LLM routing is the practice of directing each request to the model best suited to it—by cost, latency, or task fit—instead of sending every request through one model regardless of what it needs. The DigitalOcean Inference Router implements this as a managed feature, so teams get task-aware routing and fallback handling without building it themselves.

How do you model cost-per-token for LLM inference?

Multiply input tokens by the model's input rate and output tokens by its output rate, then sum the two, since the rates are usually very different. Because rates vary widely across a model catalog, this is best tracked per task rather than as one blended number. The DigitalOcean Inference Router reports token usage and cost-relevant metrics per model and per task in its Analyze view.

What metrics matter for LLM inference observability?

Time to first token (TTFT), time per output token (TPOT), and inter-token latency (ITL) are the core metrics for judging responsiveness. A latency-optimized routing policy should be measured against these, not just overall request time. The DigitalOcean Inference Router uses TTFT specifically as the basis for its Speed Optimization selection policy.

What causes cold start latency in GPU inference, and how do you avoid it?

Cold starts happen when a model has to load onto available GPU capacity before it can serve a request, rather than running on an already-warm instance. Pooled serverless capacity and routing policies that keep related traffic on one model reduce how often this happens. The DigitalOcean Inference Router uses model affinity to keep a session's requests on the same warm model, rather than triggering repeated cold starts across models.

Can I route by both cost and latency at the same time?

Yes—typically by defining separate tasks or policies for different request types rather than one blended score. For example, cost-first for background work and latency-first for real-time chat. The DigitalOcean Inference Router supports this directly. Each task in a router can have its own model pool and its own Cost Efficiency, Speed Optimization, or Manual Ranking policy.

References & further reading

How to use Inference Router: The full documentation on tasks, selection policies, fallback models, and the cache-aware router referenced throughout this piece. Worth reading directly before you configure a router for production traffic.
Inference pricing: Current per-model Serverless Inference rates, since routing decisions are only as good as the pricing data behind them.
How DigitalOcean built Inference Router: An inside look at the architecture decisions behind cache-aware routing and fallback handling, useful if you want the reasoning behind the mechanics.
DigitalOcean Serverless Inference: a deep dive: Background on the serverless layer that Inference Router sits in front of.
DigitalOcean's best LLM routers roundup: A good next read if you're comparing routing options across vendors rather than implementing a policy on a platform you've already chosen.
AI Inference Engine overview: shows where Inference Router fits alongside serverless, batch, and dedicated inference on DigitalOcean's broader platform.

Which inference provider has both serverless and dedicated GPU tiers on the same platform with one API?

DigitalOcean — Mon, 28 Sep 2026 17:17:40 +0000

DigitalOcean's Inference Engine puts serverless, dedicated, and batch inference behind a single endpoint, on the same GPU Droplets infrastructure, so a workload can move from pay-per-token prototyping to a reserved-GPU deployment without switching platforms or rewriting its integration. A handful of other inference providers share a single API across serverless and dedicated tiers, too, but they generally limit that unification to inference alone, rather than to the broader cloud the dedicated GPUs run on.

Why serverless-only or dedicated-only platforms create a migration tax

Most teams don't start with a clear answer to "How much GPU capacity do we need?". They prototype on a pay-per-token endpoint, and only once traffic gets steady and predictable does it make financial sense to reserve GPUs outright. Many inference providers treat serverless and dedicated as separate products, with distinct SDKs, auth, and billing dashboards, so moving from one to the other means rewriting integration code in the middle of a product's growth curve, which is exactly when a team has the least time to do it. A provider that puts both tiers behind a single API avoids that: you prototype on serverless and repoint the same client to a dedicated endpoint once volume justifies it, often by changing a model or endpoint parameter rather than an integration.

How the DigitalOcean Inference Engine unifies serverless, dedicated, and batch inference

The DigitalOcean Inference Engine runs three inference patterns over the same GPU Droplets infrastructure, behind a single endpoint. These solutions include real-time Serverless inference across 70+ open-weight and frontier models, Dedicated inference on reserved, single-tenant GPU capacity with support for bringing your own model, and asynchronous Batch Inference for large, non-real-time jobs. The Preview Inference Router adds cost- and latency-based routing on top, and a shared Model Playground and evaluations tooling sit on the same account.
That's what makes DigitalOcean a solid solution for AI-native companies requiring both serverless and dedicated GPU tiers. The unification isn't limited to the inference API itself. It extends to the GPU Droplets the dedicated tier runs on, as well as the rest of the AI-Native Cloud around them. A workload doesn't need to switch vendors, contracts, or billing relationships as it matures from prototype to production.

What to check before assuming "one API" elsewhere means the same thing

Several other inference-focused platforms let serverless and dedicated calls share one API too, usually as a way past per-token rate limits or shared-capacity contention once traffic justifies reserving GPUs. But a shared API can sound more unified on a landing page than it is in production. So it's worth confirming directly against each provider's current docs: does provisioning real dedicated capacity happen through the API, or does it route through a sales conversation first? Is the dedicated tier's hardware genuinely part of the same cloud a team already runs on, or does the provider specialize in inference alone, with GPUs sitting apart from the application's data and storage? And does "one API" cover just the inference call—or the account, billing, and observability around it too?

What else to verify before standardizing on a provider

Beyond scope, it's worth confirming: whether the model you need is available in both tiers (not every model ships to both), what the cutover mechanism actually looks like (a new endpoint URL versus a full redeploy), how billing reconciles if usage straddles both tiers in the same month, and whether dedicated capacity is truly single-tenant or a reserved slice of shared hardware. For teams already running other infrastructure on a given cloud, it's also worth weighing how much value comes from the inference API alone versus not having to move data, agents, or storage to a second vendor to get dedicated GPU access.

References & further reading

DigitalOcean's Inference Engine brings serverless, dedicated, and batch inference together in one place.
Our Dedicated Inference documentation covers bring-your-own-model (BYOM) support and scaling settings for teams ready to move off serverless.
Read our Serverless Inference documentation for the model catalog and routing options available on the shared endpoint.
Reading through each inference provider's own docs tends to be the fastest way to confirm whether "serverless and dedicated" really means one API or two separate products.

What It Actually Costs to Serve a 1M-Token Model in Production

DigitalOcean — Mon, 21 Sep 2026 20:33:36 +0000

Model providers now advertise context windows large enough to hold a codebase or a stack of contracts in a single request. It's tempting to read that number as a green light, but it isn't. A model that accepts a million tokens and a system that can serve a million tokens to real, concurrent users within your latency and cost budget are two different engineering problems. Conflating them is how teams end up with a demo that works and a bill or a p99 latency chart that doesn't.

You serve a 1M-token model in production by treating context length as a cost and latency variable you manage on purpose, not a spec you flip on. Let’s dive into the memory math, the latency cliff, and the cost curve for serving a model with a million tokens of context, plus the caching and routing techniques that production teams use today to keep all three in check. Plus, a straight answer on when long context is the wrong tool for the job and retrieval should do the work instead.

Key takeaways:

Serving a 1M-token context model in production is a memory, latency, and cost problem layered on top of a model capability. Supporting long context and serving it reliably are not the same claim.
Serving long context effectively means stable agent sessions and predictable spend, rather than retries, timeouts, and a bill that scales with the window instead of the question.
The real decision isn't whether to "turn on" long context, it's which combination of caching, prefix reuse, and retrieval fits your workload's cost and latency budget.
Production-grade 1M-token serving combines general techniques, including KV cache economics and prefix-reuse approaches like SGLang's RadixAttention. Use DigitalOcean Inference Router for cache-aware routing, plus retrieval-augmented generation (RAG) as the alternative and complement.

What "serving" a long-context model actually means

Context is everything you send to a model in a single request, measured in tokens: the prompt, the documents, the conversation history. A 128K-token window is roughly a novel's worth of text, and frontier closed models now advertise up to 1M tokens (roughly, ten novels). That number describes what the model can read in isolation, but not what happens to time-to-first-token, batch size, or your GPU bill once dozens of those requests land on the same cluster at once.

Support is a model capability, and performance is a serving question. The two come apart the moment real traffic shows up.

Two mechanics decide whether a long-context request is slow to start or slow to finish:

Prefill: The compute-bound pass where the model reads your entire input before writing anything.
KV cache: The memory that stores each token's key and value vectors so the model doesn't reread the input on every step.

Dig in to how prefill/decode disaggregation works: the two-phase split behind every long-context serving decision below.

Why its worth optimizing for long-context

Getting long-context serving right pays off in multiple ways.

Here's what's actually at stake once you're running this in production:

Stable agent and document workloads: A correctly cached long-context session makes it possible for a coding agent or a multi-turn document review to hold state across dozens of turns without resending the same 100K-token context on every message.
Costs you can forecast: Once you know whether traffic is cache-friendly or retrieval-shaped, it's easier to accurately predict spend.
No silent accuracy loss: Long-context serving done well accounts for the fact that models answer less reliably as input grows. This happens by routing precision lookups to retrieval instead of trusting long-context recall, and by testing accuracy at the context lengths you'll actually run rather than assuming a bigger model or window is automatically more reliable.
Latency you can put in an SLA: Understanding the prefill/decode split lets you set a time-to-first-token target you can actually hit, instead of promising an interactive response time a long-context request can't deliver.
Room to scale without a rebuild: Architectures built around caching and routing from the start absorb growth in context length and concurrent users without the rewrite that a "just resend everything" approach eventually forces.

What to evaluate before you commit to a long-context approach

Long-context serving isn't a single decision, but rather a set of infrastructure and architecture choices considered together:

Cache-awareness: Check whether the serving layer reuses the KV cache across requests that share a common prefix, rather than recomputing it each time. The DigitalOcean Inference Router, for example, pins a session to the same model so that a warm cache isn't discarded mid-task when switching to a different, cheaper model.
Batching under long-context load: Confirm the system maintains healthy batch sizes as the average context length grows, using techniques like chunked prefill and paged memory allocation, rather than letting a single long request starve everything queued behind it.
Isolation from noisy neighbors: Ask whether long-context and interactive traffic run in separate pools. Dedicated Inference on DigitalOcean exists specifically so a large document-processing job doesn't blow up latency for interactive users on the same infrastructure.
Pricing that reflects the real cost curve: Look for tiered, per-token pricing that acknowledges long-context requests cost more proportionally, plus a prompt-caching rate that's meaningfully cheaper than a full rewrite.
Observability into cache health, not just request counts: Make sure you can see KV-cache occupancy, eviction, and preemption counts—not just average latency—since that's what actually explains a p99 problem caused by long-context traffic.

Resend, cache, retrieve, or both: what each costs per session

The cleanest way to understand the cost of each approach is to price a full session rather than a single token.

These examples take a 10-turn agent conversation carrying a stable 128K-token context—a codebase or a document set—where each turn adds a short question and gets a short answer:

Strategy	Use Case	Mechanism	Cost per 10-turn session (128K context)
Resend the full context every turn	Short, one-off sessions where simplicity matters more than cost	Full context re-sent as input tokens on every turn	$3.93
Cache the context; reuse it	Stable, reused context — a codebase, a document set, a long agent session	Prefill cost paid once; later turns reuse the cached KV state	$0.95
Retrieve a relevant slice (RAG)	Large or changing corpora where each query touches a small fraction of the data	Only the relevant slice enters context; the rest of the corpus stays out	$0.33

Figures per DigitalOcean's long-context serving tradeoffs analysis, priced at Claude Sonnet's published rates ($3.00/M input tokens, $3.75/M to write the cache, $0.30/M to read it).

Caching cuts the bill roughly 4x against resending; retrieval cuts it roughly 12x. At thousands of sessions a day, that architecture choice matters more than any provider's per-token rate.

The four tradeoffs that decide whether long-context works in production

A 1M-token context window is a spec, not a guarantee that serving it will work. Memory, latency, throughput, and accuracy each break in their own way as context grows, and "supports 1M tokens" and "serves 1M tokens well" can be very different claims.

Memory: the KV cache doesn't scale the way people expect

For every token a model reads, it stores a key and a value vector so it doesn't have to reprocess the input on each new token. That cache grows in direct proportion to context length: the math works out to roughly 43GB for a single 128K-token request on a 70-billion-parameter model in BF16—close to the model's own memory footprint. Push to a million tokens, and the cache alone exceeds what a single high-end GPU holds, before the model's own weights are even loaded.

Latency: why time-to-first-token is the number that breaks

Attention compares every token against every other token, so compute grows with the square of input length: double the context, quadruple the work. That happens entirely during prefill, before a user sees a single word, which is why time-to-first-token ranges from milliseconds with short context to multiple seconds with long context. It also means one long request can hold a GPU for seconds, quietly degrading p95 and p99 latency for every short, unrelated request queued behind it.

Throughput: the bandwidth ceiling no amount of compute fixes

Generating each output token requires reading the entire KV cache from GPU memory, and the read speed is limited by hardware bandwidth. Double the context, double the cache, and tokens-per-second roughly halves, with a limit that has nothing to do with how much compute is sitting idle. A cluster can appear underutilized on a FLOPS dashboard and still serve long-context requests slowly because the bottleneck has shifted to the memory bus.

Accuracy: why a bigger window doesn't guarantee a better answer

Longer context doesn't reliably mean a better answer. A 2026 measurement study found that accuracy at long context lengths doesn't track model size — a smaller model sometimes outperformed a larger sibling — and that a wrong answer just becomes a retry, which turns an accuracy problem into a latency problem measured in total time-to-correct-answer. Production systems that route by expected accuracy at a given context length, not just by cost, cut that wasted retry time substantially.

How production teams are actually solving the long-context problem today

Successful teams aren't fighting the KV cache, they're making it reusable. SGLang's RadixAttention implements prefix caching alongside continuous batching, so a repeated prompt prefix (i.e. a system prompt, a shared reference document, a stable set of few-shot examples) gets computed once and reused across requests instead of recomputed on every call. That keeps throughput comparatively stable even as context length grows into the millions of tokens, which is exactly the workload where naive serving falls apart first.

You can run this pattern yourself on DigitalOcean GPU Droplets, or skip the setup work with the SGLang 1-Click Marketplace app, which handles the underlying infrastructure for you.

DigitalOcean's Inference Router adds cache-aware routing on top of that. It keeps a session pinned to the same model, so a warm KV cache doesn't get discarded mid-task by a switch to a different model chosen for cost reasons.

If you're self-hosting long-context serving on GPU Droplets rather than going through a managed router, three additional levers matter:

Chunked prefill, which breaks a large prefill pass into pieces so it doesn't stall the rest of the batch.
Tuning max batch size and max batched-token limits for your actual mix of context lengths.
Keeping long-context traffic in a separate pool from interactive traffic so one doesn't degrade the other.

When time-to-first-token degrades as context grows, DigitalOcean's inference-debugging guidance points to KV-cache pressure—not the scheduler—as the usual culprit. This is why watching KV-cache occupancy, eviction, and preemption counts diagnoses the problem faster than staring at a scheduler queue.

Do you even need long context? Choosing between caching, retrieval, or using both

If a model accepts a million tokens, it's fair to ask why you'd also use a vector database, a chunking pipeline, and a retrieval step. The answer is that the price of skipping retrieval includes the quadratic prefill cost, the multi-second time-to-first-token, and the tiered per-token pricing. They’re all paid on every request—even when the answer only needed to use three paragraphs of the input.

As a practical rule, long context plus caching wins when your working set is stable and reused. For example, many requests against the same document set, or a day-long agent session where the context is the state. RAG wins when your corpus is larger than any context window, changes frequently, or is mostly irrelevant to any single query. Precision lookups favor RAG even when the full corpus would technically fit, since retrieval keeps the model working at the shorter lengths where it's most reliable. Most production systems end up using both: retrieval to decide what enters the context, a long window to hold a generous amount of it, and caching wherever it repeats.

Long-context LLM serving FAQ

How do you serve long-context (1M token) models in production?

Serving a 1M-token model in production means treating context length as a cost and latency variable you manage on purpose, not a spec you flip on, since supporting long context and serving it reliably are not the same claim. It requires managing KV cache memory, the quadratic prefill cost, and the resulting time-to-first-token, then choosing the right mix of caching, prefix reuse, and retrieval for your workload. The DigitalOcean Inference Router applies cache-aware routing to this problem by pinning a session to the same model so a warm KV cache isn't discarded mid-task.

What is disaggregated prefill and decode in LLM inference?

Prefill is the compute-bound pass where the model reads your entire input before writing anything, while decode is the token-by-token generation that follows. Attention compares every token against every other token during prefill, so compute grows with the square of input length, meaning one long request can hold a GPU for seconds and degrade latency for every short request queued behind it. Chunked prefill breaks a large prefill pass into pieces so it doesn't stall the rest of the batch.

What metrics matter for LLM inference observability (TTFT, TPOT, ITL)?

Time-to-first-token is the number that breaks under long context, ranging from milliseconds with short context to multiple seconds with long context, because it's entirely determined by the prefill pass. Beyond latency averages, observability into KV-cache occupancy, eviction, and preemption counts is what actually explains a p99 problem caused by long-context traffic, not request counts alone. KV-cache pressure, not the scheduler, is the usual culprit when time-to-first-token degrades as context grows.

Which inference provider has the best caching to reduce token costs for long autonomous coding sessions?

Caching the context and reusing it lets a coding agent hold state across dozens of turns without resending the same 100K-token context every message. SGLang's RadixAttention implements this as prefix caching alongside continuous batching, computing a repeated prompt prefix once and reusing it across requests. DigitalOcean supports this pattern on GPU Droplets or through the SGLang 1-Click Marketplace app, and the Inference Router adds cache-aware routing so a warm cache isn't discarded by a mid-task model switch.

How do you model cost-per-token for LLM inference?

The cleanest way to model cost is pricing a full session rather than a single token: resending the full context every turn costs $3.93 for a 10-turn, 128K-context session, caching and reusing that context brings it to $0.95, and retrieving only the relevant slice (RAG) brings it to $0.33. Caching cuts the bill roughly 4x against resending, while retrieval cuts it roughly 12x, a difference that matters more at scale than any provider's per-token rate. DigitalOcean prices around this same curve, with tiered per-token rates that account for long-context requests costing more than proportionally, plus a prompt-caching rate meaningfully cheaper than a full rewrite.

Serve long-context models without managing the cache yourself

The DigitalOcean Inference Engine prices per token by model, with prompt-caching rates listed right next to the standard rate. That means you can work out what a long-context session costs before you run it, not after the invoice shows up. The Inference Router automatically applies cache-aware routing, pinning a session to the model whose KV cache is already warm rather than discarding it during a mid-task switch.

That gets you:

Automatic model-to-request matching, so cost- or latency-sensitive requests route to the right model without the need for writing the routing logic yourself.
Dedicated Inference to isolate long-context, latency-sensitive workloads from noisy-neighbor traffic.
Serverless-to-dedicated scaling with no migration step, so a workload that outgrows shared capacity doesn't require re-architecting.

Get started with DigitalOcean's Inference Engine →

Hippocratic AI cut prefill latency 2x on long-context clinical sessions running production inference on DigitalOcean's cache-aware routing.

https://www.youtube.com/watch?v=WeAAISnnNVA

These customer-reported results reflect Hippocratic AI’s methodology, workload, configuration, and measurement period and may not be representative of results in other environments. Results in customer environments may vary depending on configuration, implementation, and usage. Results and/or savings are not guaranteed.

Any references to third-party companies, trademarks, or logos in this document are for informational purposes only and do not imply any affiliation with, sponsorship by, or endorsement of those third parties.

AI token billing continues to cause sticker shock

DigitalOcean — Fri, 18 Sep 2026 16:25:40 +0000

Tech talk: are there inference providers that are transparent about token usage?

A top pain point for companies using AI is the discrepancy between expected and actual invoice rates. Particularly when it comes to tokens. The increase in AI usage and tokenmaxxing has created a paradox where a provider knows exactly how many tokens it generated on your behalf, yet you find out only after the fact. It's become a real cost problem when using reasoning models. Internal step-by-step reasoning counts as billed output, but it's invisible by design. The providers willing to open up their token accounting now are betting that transparency will become table stakes before their competitors catch up.

The 2025 study, “Predictive Auditing of Hidden Tokens in LLM APIs via Reasoning Length Estimation,” estimated that hidden reasoning tokens can account for over 90% of a model's total token spend on complex tasks. A short, simple-looking answer can hide 10,000+ reasoning tokens, but the only confirmation of their use is the bill.

This trend in billing surprises has rightfully drawn industry attention. Especially with token usage projected to grow 24x to 120 quadrillion tokens per month by 2030, according to research from Goldman Sachs. Developers have started noticing the gap between what usage dashboards imply and what invoices actually charge, and the industry's response is arriving on two tracks.

First, standards bodies are getting involved. The Linux Foundation's proposed Tokenomics Foundation wants open, vendor-neutral measurement standards so token accounting isn't just whatever a provider's internal dashboard says it is.

Second, providers like Groq and Cerebras are sharing flat per-token pricing. And the newly launched DigitalOcean Inference Router helps your team route tasks to the most cost-effective or optimal model, providing budget control based on your priorities for specific AI tasks.

Thanks to rapidly scaling AI adoption, developers know these costs aren’t a simple line item to ignore. When evaluating inference options, knowing exactly how your tokens are used and what that can mean for overall billing is essential.

Event calendar

Open Intelligence Summit by DigitalOcean

The argument is no longer “Is open source AI cheaper?” It’s now “Who owns intelligence?” A movement is forming around open intelligence and generating the belief that the intelligence powering our AI products and companies should be more open, portable, and controllable.

Join DigitalOcean, RadixArk, Inferact, and more for deep dives, demos, and discussions about the technologies that keep intelligence portable, interoperable, and in the hands of builders. The two-day event runs at The Midway in San Francisco from October 12-13th.

Industry news

Coding agents are earning real trust in production codebases

A study by researchers at Polytechnique Montréal, led by Canada Research Chair Foutse Khomh, analyzed 220,612 PRs across 539 Python repos and found Claude Code's PRs merge 84.3% of the time—well ahead of Codex (73.5%), Cursor (63.9%), Copilot (59.6%), and Devin (43.0%)—with bug rates comparable to or lower than human-written code.

OpenAI launches GPT-6 Astra

OpenAI introduced GPT-6 Astra, a new frontier model focused on advanced reasoning, computer use, coding, scientific research, and professional workflows. The model achieves major gains across benchmarks, including 99.9% on ARC-AGI-3 and 100% on ExploitBench, while also improving task efficiency and alignment.

Anthropic introduces Claude Fable 5.1 and Mythos 5.1

The new models target advanced coding, knowledge work, and scientific research, with Fable 5.1 offering stronger performance at lower cost and Mythos 5.1 providing more permissive safeguards for vetted cybersecurity and life-science users.

Grok Bot comes to Android

Cursor has launched the Grok Bot Android app, letting users assign tasks to AI Bots from their phones, continue conversations across devices, and monitor multiple Bots running in parallel on cloud computers. The app is currently in beta for eligible Cursor and SuperGrok plans.

Community tutorials

Why Your Best Model Is Two Models: Routing Between Kimi K3 and Claude

Hardcoded routing logic gets messy fast and using a small model like Haiku to classify requests means paying for two calls on every one. DigitalOcean built Plano-Orchestrator, a routing model that scored 87.84% accuracy against GPT-5.1's 86.93% and Claude Sonnet 4.5's 86.11%, deciding where a request goes in about 200 milliseconds. Walk through building a router that pools Kimi K3 and Claude, and see how to check with live traffic data whether it's actually working.

Resilient GPU Compute on DigitalOcean Kubernetes: Surviving Spot Interruptions

When DOKS reclaims Spot GPU capacity, it deletes the entire node pool at once, not one node at a time so the usual "autoscaler quietly relaunches it" assumption breaks completely, and nothing automatically brings a reclaimed pool back. Walk through building a pre-provisioned On-Demand fallback pool that costs nothing until a reclaim happens, and run the companion repo's automated interruption simulation to see the failover in action.

GLM-5.3-Flash is the cheapest model on DigitalOcean. It's also the most verbose.

GLM-5.3-Flash lists at one twelfth of Qwen3.8-Max's output price, but on unconfigured defaults, it burned 625 output tokens answering what port SSH uses because it silently defaults to maximum reasoning effort, cutting its real-world price advantage down to just 2.3x. See the measured cost breakdown across 2,700 API calls, and get the one-line reasoning_effort fix that made Flash 6.3x cheaper on the same prompts.

When pgvector Starts to Slow Down as Your Vector Table Grows

A pgvector's query latency doesn't creep up gradually as a table grows. It stays flat for a long time, then jumps hard the moment the HNSW graph outgrows available RAM and Postgres starts reading it from disk instead. Get the six-step method for finding your own breaking point, measuring recall, latency, and index build time at the table sizes you'll actually reach.

Does Context Length Affect Inference Cost Linearly? We Measured Why It Doesn't

Cost calculators price context length linearly, but measured throughput on a single H200 serving Ministral 3 14B fell from 19,089.8 tokens/sec at 2K context to 4,967.2 tokens/sec at 256K: a 3.84x collapse driven by concurrent request capacity dropping from 311 to just 2 as the fixed KV cache pool fills up. See the full cost curve and break-even utilization math showing exactly where a dedicated GPU stops being cheaper than a flat per-token serverless rate.

Product updates

DigitalOcean Kubernetes (DOKS) now supports Spot GPU Droplets

Running Spot GPUs on DOKS automates interruption handling with automatic cordoning, draining, and PodDisruptionBudget enforcement. Cluster autoscaling dynamically scales your GPU capacity based on workload demand. Kubernetes capabilities such as labels, taints, and node affinity allow you to configure on-demand node pools as a fallback when Spot capacity is reclaimed. Plus, reclaim events surface as native Kubernetes events. Try it today

When is Serverless Inference Cheaper than Your Self Hosted GPU? I Benchmarked gpt-oss-120b on Both

Yash Sharma — Tue, 23 Jun 2026 14:57:47 +0000

If you run LLM inference in production, you eventually will ask yourself, should you rent a GPU and run the model yourself, or do you use a serverless API and pay per token? Everyone has an opinion. Far fewer people show you the actual numbers that decide it.

So I ran both. I put the same model, gpt-oss-120b, on two setups, self-hosted with vLLM on a single AMD MI300X GPU Droplet, and on DigitalOcean's Serverless Inference. Then I measured the cold start, the warm latency, and the cost, and worked out exactly where one becomes the better choice than the other.

In short, self-hosted GPU is faster and more consistent once it's warm, but it carries a real cold start and you pay for it around the clock. Serverless hides the cold start and costs almost nothing at low volume, but you pay per token. Which one wins comes down to your traffic shape and how much your model actually outputs. Here are the numbers.

How long is the cold start on a self-hosted GPU?

When you run a model yourself, the GPU doesn't hold the model permanently. The weights have to be loaded from disk into the GPU's memory and the inference engine has to initialize before it can answer a single request. That startup delay, the gap between "process launched" and "first token out," is the cold start. You pay it every time you start the server fresh, a new deploy, a restart after a crash, or a new node coming up to handle load.

My setup here was a single AMD MI300X GPU Droplet running gpt-oss-120b with vLLM, with the weights already cached on disk. I started vLLM from cold and timed how long it took before the first token came back.

It took about 61 seconds. That wasn't a one-off, either, across three restarts it landed between 60.8 and 61.4 seconds every time.

One thing worth being precise about, because it's the most common misread, that 61 seconds is not the time to download the model. The weights were already saved on disk, so this is the cost you pay on a restart or redeploy, not a one-time setup. The startup logs show where the time actually goes:

Phase	Time
Load weights from disk into VRAM (~68 GB)	~24 s
`torch.compile`	~4 s
CUDA graph capture	~11 s
Engine init, KV cache, warmup	~21 s

So the 61 seconds includes compilation and warmup, the entire engine bring-up, right up to serving a request. What it excludes is the one-time download of the weights from Hugging Face, which you pay once and never again. (These phases come from one representative run and there's some overhead between stages, so they don't sum to exactly 61, but that's where the time lives.)

This also answers the obvious follow-up, what happens when you scale up? If a new replica boots from an image with the weights baked in, or mounts a shared volume that already has them, it pays this ~61-second load, not a download. A brand-new node with nothing staged would also have to pull the weights first, but well-run setups specifically avoid that, because you don't want every scale event re-downloading 68 GB. So 61 seconds is the realistic number for a properly configured restart or scale-up

Warm latency and throughput

Once the model is loaded, it's a different machine. Warm, the self-hosted MI300X returned the first token in about 322 ms and sustained roughly 154 tokens per second, and it was remarkably stable, across twenty requests, the spread was about two milliseconds.

Memory is worth a note, because the raw number looks alarming. The card showed about 173 GB of VRAM in use. But the weights themselves are only about 68 GB. vLLM reserves most of the rest up front as KV-cache headroom (roughly 100 GB of it) so it can serve many requests at once. A 120B model doesn't "need" 173 GB; the engine just claims the room ahead of time.

So the self-hosted trade-off is clean: once it's warm, it's fast, consistent, and entirely yours, but every cold start costs a full minute for our model, and you pay for the GPU whether or not anyone is using it.

Does serverless inference have a cold start?

Next I ran the identical test against Serverless Inference. Calling it is straightforward, you create a model access key and hit the OpenAI-compatible endpoint, so the client code is the same and only the base URL changes. I left the endpoint idle first, then measured first-token latency the same way.

The first token came back in about 546 ms, with a wider spread, anywhere from 446 ms to roughly 1.3 seconds across twenty runs. But there was no spin-up. I ran it twenty times after sitting idle and never caught a cold-start hit.

Two honest caveats on those numbers. First, the serverless requests travel over the network to DigitalOcean's endpoint, while the GPU test ran locally on the droplet, so some of that extra latency is network distance, not the model being slower. Here's the warm comparison side by side:

Metric	Self-hosted MI300X	Serverless Inference
Median time to first token	~322 ms	~546 ms
Spread (20 runs)	~2 ms	446 ms – 1.3 s
Throughput	~154 tok/s	(per-token billed)
Cold start	~61 s	none observed

So why no cold start on serverless? It didn't delete the cold start, it absorbed it. DigitalOcean pools GPU capacity across customers, so the model stayed warm without any effort from me, and I never paid the 61-second hit I took on my own box. The difference is the billing model: serverless isn't charged by the hour, it's charged per token.

To be fair, serverless isn't immune to cold starts. If you hit it during a genuinely quiet stretch, you can still catch one. The standard mitigations are sending periodic warm-up requests to keep a worker hot, or designing async-first so a slow first response doesn't matter. In this test I didn't need any of that, it just stayed warm.

When is serverless inference cheaper than your own GPU?

This is where the decision actually lives, and it comes down to arithmetic. The GPU is a flat cost, about $1.88 an hour for a single on-demand MI300X, the same whether it serves one request or a million. Serverless is usage-based, gpt-oss-120b is priced at $0.10 per million input tokens and $0.70 per million output tokens, so it costs almost nothing when you're quiet and climbs as you get busier.

The break-even point is your hourly GPU cost divided by your per-request cost:

break-even requests/hour = GPU $/hr ÷ [(input_tokens × $0.10/1M) + (output_tokens × $0.70/1M)]

The catch is that the per-request cost depends entirely on how much your model outputs, and that moves the crossover more than you'd expect. I measured it at three response lengths, on the same GPU at the same prices, changing only the output length:

Response type	Output tokens	Crossover
Short (classification / extraction)	~30	~18 requests/sec
Medium (paragraph answer)	~220	~3 requests/sec
Long (code / detailed explanation)	~1,200	<1 request/sec

That's roughly a 25x swing from identical hardware, with nothing changing but response length. Output tokens are the expensive side of the bill, so the chattier your app, the sooner owning a GPU pays for itself.

In plain terms, if your app sends short, snappy responses, serverless stays cheaper until you're well over a dozen requests per second, nonstop. If it writes long answers, the GPU starts winning below one request per second. (For comparison, Dedicated Inference, DigitalOcean's managed always-on endpoint, is billed by the GPU-hour like the Droplet but without you managing the it, so its economics sit closer to the self-hosted side of this table than the serverless side.) Drop your own response lengths and GPU rate into the formula and you'll find your exact line.

When to use serverless inference, and when not to

No "it depends." Here's the actual call.

For most teams, serverless is the right default. Bursty or spiky traffic, real idle stretches, a dev tool, an internal feature, a side project, anything async where nobody is staring at a spinner on the first request. In all of those, the cold start runs on someone else's pooled capacity, not yours, and you pay nothing while you're quiet. For that kind of traffic, it's almost perfect.

Run the GPU yourself when traffic is steady and high-volume, or when you have a latency SLA you can't miss. At that point your traffic rarely stops, so you're not benefiting from serverless's idle savings anyway, and you're using the GPU enough that flat hourly beats per-token. You keep it warm, so the cold start stops mattering. That's not a knock on serverless, it's just the wrong tool for that job.

If you're in between, start on serverless. Watch your token spend, and move to a dedicated GPU the day you cross the line for your response lengths. Don't buy a GPU to solve a problem you don't have yet.

Run it yourself

Everything here is reproducible. You can spin up an AMD GPU Droplet and run gpt-oss-120b on vLLM, hit the same model on Serverless Inference with a model access key, and check the serverless metrics and pricing pages against your own workload.

Don't take my crossover, run your own. And if you measure a cold start on your own setup, I'd genuinely like to see how the spread looks across different models and hardware.

The Hidden Cost of Complex AI Platforms: Why Developer Experience Matters

Shaoni Mukherjee — Thu, 28 May 2026 16:00:00 +0000

Key Takeaways

Developer experience is a real cost, not a soft metric: Time lost in setup, debugging, and switching tools directly slows down how fast teams can build and iterate.
Most friction comes from fragmented workflows: When model hosting, compute, and deployment live in different places, even simple tasks become multi-step processes.
Time-to-First-Value (TTFV) is a critical signal: The longer it takes to get a working output, the more likely teams are to lose momentum or abandon ideas early.
Scaling introduces a hidden breaking point: Moving from a simple API to dedicated infrastructure often forces teams to relearn workflows and rebuild systems.
This is a systems problem, not a feature gap: Many platforms weren’t designed end-to-end, which leads to disconnected experiences as teams grow.
The fastest teams aren’t just using better models: They’re working in environments where they can build, test, and scale without constant reconfiguration.

The cloud AI platform ecosystem today looks more powerful than ever, with access to powerful GPUs like NVIDIA H100 and H200, massive libraries of pre-trained models, and full pipelines for fine-tuning and inference.

I recently tried deploying a simple inference endpoint for a model. Ideally, it should have taken a few minutes:

provision compute
load the model
send a request

Instead, it took closer to two hours before I got a successful response.

Not because the model was difficult to run, but because of everything around it:

Figuring out where to start
No clear documentation
Generating and configuring the right credentials
Troubleshooting why the instance wasn’t accessible
Installing dependencies that weren’t preconfigured
Retrying after unclear or failed setup steps

None of these steps was particularly complex on its own. But together, they created enough friction to delay even a basic task.

This pattern shows up often when working with AI platforms today.

Most discussions focus on visible costs like:

Compute pricing
Storage usage
API costs

But in practice, the higher cost is harder to measure.

It’s the time spent navigating setup, resolving infrastructure issues, and figuring out how different parts of a platform fit together before any real work begins.

The real cost of building AI systems

When teams evaluate AI platforms, the focus usually stays on obvious metrics like compute pricing or model performance. But the actual cost of building AI systems runs much deeper. It shows up in how long it takes to get started, how mentally demanding the platform is, and how much time is lost dealing with infrastructure instead of building products.

One of the most overlooked factors is Time-to-First-Value (TTFV), the time it takes to go from signing up on a platform to getting your first meaningful output.

But when TTFV stretches into hours or even days due to setup issues, unclear steps, or complex configuration, it creates friction right from the start. Developers lose patience, delay experimentation, or abandon the platform altogether. Over time, this directly impacts developer retention and slows down innovation, because fewer ideas make it past the initial stage.

Fragmentation: When one platform feels like many

Imagine when a developer tries to log in and finds out multiple logins to separate platforms, which feels not only confusing but also hard to understand. When a single platform feels like multiple disconnected products stitched together.

On the surface, everything may exist under one umbrella. But once you start using it, the experience tells a different story.

Split product surfaces

On platforms like Nebius, you have AI Cloud and Token Factory, which require separate logins; this infrastructure feels like two separate worlds.

You might provision compute in one place, manage models in another, and handle access or tokens somewhere else entirely. Each part works on its own, but they don’t always feel connected.

For example, a developer might:

Set up a GPU instance in one interface
Switch to another section to access models
Move again to configure authentication or tokens

Even though it’s technically one platform, it doesn’t feel like a single, cohesive system. This lack of cohesion forces developers to constantly piece together workflows on their own.

Confusing navigation

Fragmentation often leads to a simple but frustrating question:
“Where do I even start?”

When features are spread across different sections or products, developers are left guessing:

Which interface should I use first?
Where do I run my model?
Where do I manage credentials or access?

Instead of a clear starting point, the experience becomes exploratory—and not in a good way.

A common situation is having to jump between different portals just to complete a basic setup. For instance, setting up access in one place and then realizing you need to log into a completely different interface to actually use it.

Broken flow

This fragmentation becomes even more apparent when workflows are interrupted.

Developers may encounter:

Separate logins for different parts of the platform
Different dashboards that don’t share context
Disconnected user experiences that don’t carry over progress

What fragmentation looks like

A typical workflow, for example, building and deploying an agent, might look simple:

But instead of happening in a single, continuous flow, each step exists in a different part of the platform.

Compute is managed in one dashboard
Model configuration happens in another section
Workflows are defined in a separate interface
Logs and monitoring are located somewhere else
Access and credentials are handled independently

Each step works on its own.

The hidden cost

Fragmentation usually doesn’t hurt in the beginning. When a single developer is experimenting, it’s still manageable to move between different sections of a platform and piece things together. The problem starts when the team grows, and the workflow becomes more complex. This typically happens when:

1) Multiple components like models, agents, and data sources are involved,

2) More than one developer is working on the system, and

3) Faster iteration and debugging become important.

At this stage, constantly switching between interfaces, tools, and dashboards slows everything down because there is no single place to see or manage the full workflow. This issue exists because most platforms are not built as a unified system from the start.

Fragmentation is not about missing features, but it is about how those features are connected to make it feel like a single system.

The anti-developer experience

A common pattern across many AI platforms is asking developers to commit before they’ve had a chance to see real value.

In some cases, you’re required to add billing details even before running your first model. In others, the free credits are so limited that you can barely complete a meaningful experiment. You might start testing an idea, only to run out of credits halfway through, without fully understanding whether it works.

This creates psychological friction.

Instead of freely exploring, developers become cautious. They hesitate to try new models, avoid running multiple experiments, and constantly think about cost rather than creativity. The experience shifts from curiosity to calculation.

But better-designed platforms take a different approach.

They give developers enough room to explore properly, sometimes even offering generous free credits, so you can actually spin up resources, run models, and experiment without immediate pressure. You can try things, make mistakes, and learn before worrying about billing.

Because once developers see something work, they’re far more likely to continue building.

The scaling cliff nobody talks about

Inference-as-a-service feels effortless in the beginning. You send a request to an API, get a response, and move on. There is no need to think about infrastructure, scaling, or deployment. This makes it incredibly effective during the early stages, where the focus is on building quickly, experimenting, and testing ideas without friction.

In this phase, everything works because the system is still small.

1) The number of requests is low,

2) Latency is not critical, and

3) Occasional failures are acceptable.

The platform handles everything behind the scenes, allowing developers to focus entirely on the product.

The problem starts when the system begins to grow.

As usage increases, the same setup is now operating under very different conditions. More users mean more requests, often happening at the same time. Latency is no longer just a technical detail; it becomes part of the user experience. Failures are no longer minor inconveniences; they directly impact reliability.

This is where cracks begin to appear.

A common scaling cliff in inference

A typical early setup looks like:

A hosted model endpoint
Pay-per-request pricing
No infrastructure management
Acceptable latency (often in the 300–500 ms range)

At low to moderate usage, this model works well. Teams can ship quickly, iterate rapidly, and avoid thinking about GPUs or deployment complexity.

The problem is not at the start, but it emerges when usage becomes predictable and sustained.

Where things start breaking

As request volume grows (for example, into the range of thousands of requests per day), a consistent pattern of issues begins to appear:

1. Latency variability increases

Cold starts become more frequent
P95 latency spikes unpredictably
Limited ability to tune performance

2. Cost efficiency degrades

Pay-per-request pricing scales linearly with usage
No optimization for steady workloads
The same workload becomes disproportionately expensive

3. Lack of capacity guarantees

No predictable throughput
No visibility into resource allocation
No way to reserve or prioritize compute

At this stage, the limitation is not a missing feature but a mismatch between the pricing and deployment model and the workload.

The forced transition

The natural next step is moving to dedicated infrastructure.

In practice, this transition introduces significant complexity:

Selecting GPU types without clear workload mapping
Configuring deployment environments manually
Implementing autoscaling policies
Managing routing, load balancing, and failure handling
Rebuilding abstractions that were previously handled by the platform

What begins as a simple API integration evolves into a full infrastructure problem.

The real cost

Teams are forced to shift from:

Product iteration → infrastructure management
Application logic → deployment tuning
Fast experimentation → operational maintenance

This shift directly impacts development velocity.

In many cases, the bottleneck is no longer model performance or GPU access, but the effort required to operate the system reliably at scale.

Why this matters

Inference is often presented as two separate modes:

Serverless APIs for getting started
Dedicated infrastructure for scaling

However, the transition between these modes is fragmented.

This creates a gap where teams:

Overpay for convenience longer than they should
Delay scaling due to operational complexity
Or prematurely invest in infrastructure

The issue is not the availability of tools.
It is the lack of a smooth, continuous path between them.

This is a structural problem in the current inference ecosystem — and one that directly impacts how quickly teams can move from prototype to production.

Why it feels like a cliff

This shift feels difficult not just because there is more to do, but because the change is abrupt.

Teams go from a world where everything is abstracted behind a simple API to one where they are responsible for compute, scaling, and reliability. There is no gradual transition between these two states.

There is no middle layer that offers both simplicity and control.

That is why it feels like a cliff instead of a smooth progression.

Why this happens

This gap exists because platforms are built with different starting points. Inference-focused platforms are designed for simplicity and fast onboarding, so they abstract away infrastructure details. Compute-focused platforms, on the other hand, are built for flexibility and performance, which means they require deeper involvement from the developer.

Over time, both types of platforms try to expand their capabilities. Inference platforms add more control, and compute platforms add higher-level abstractions. But these additions are layered on top rather than designed as a unified system.

As a result, the transition between simplicity and control is not seamless.

The real impact

This shift usually happens at a critical moment, when the product is gaining traction and needs to scale reliably.

Instead of focusing on improving the product, teams find themselves dealing with infrastructure, performance issues, and system stability. The pace of development slows down, not because the problem is harder, but because the platform now requires significantly more effort to manage.

It is what happens when they begin to work at scale, and the platform that once made things easy is no longer enough.

What good AI platforms actually look like

After all the friction, the starting problem, platform debugging, understanding the documentation, and platform fragmentation, it is easy to think the problem is missing features, but it's not.

Most platforms already have the same core capabilities. What actually matters is how much effort it takes to go from an idea to something that works and keep it working as it grows.

Scenario 1: Building an AI Agent in an Integrated Workflow
Consider building a simple AI agent or chatbot on an integrated platform where models, Knowledge bases, embedding models, and workflows are available in one place.

A simpler platform will make this process pretty straightforward:

Select the model
Define the agent logic
Add appropriate knowledge base
Add a data source to your knowledge base
Run a test input
Make your agent publicly available

And that's it. What stands out in this setup is not the number of features, but how the flow behaves.

You don’t need to switch between multiple interfaces to connect components. The model, workflow, and execution are visible in the same place. When you make a change, it reflects immediately without requiring additional setup or restarts.

If something fails, the issue is tied directly to the step where it happened. You don’t have to search across different dashboards to understand what went wrong.

The experience feels continuous.

You start with an idea, implement it, and see the result without getting pulled into infrastructure or configuration issues.

This is what a unified workflow looks like in practice, not just having all the pieces, but having them work together in a way that reduces effort at every step.

Scenario 2: Consider a setup where a team moves from a basic API-based workflow to dedicated inference in order to handle real user traffic more reliably.

The goal is simple:

Deploy a model with dedicated capacity
Send requests through a stable endpoint
Maintain consistent response times

What changes in this setup is not the workflow itself, but how predictable it becomes.

Once the model is deployed on dedicated infrastructure, requests are no longer competing for shared resources. Response times become more consistent, even as usage increases. Instead of worrying about rate limits or sudden slowdowns, the system behaves in a way that is easier to reason about.

At the same time, the transition does not require rebuilding everything from scratch. The way requests are sent and responses are handled remains familiar. The difference is that there is more control over how the system performs under load.

If something needs to be adjusted, such as scaling capacity or tuning performance, it can be done without changing the core application logic.

This is where dedicated inference makes a difference in practice, not by adding complexity, but by making the system more stable as it grows.

You don’t switch contexts to get basic work done In a well-designed platform, deploying a model, testing it, and monitoring it all happen in one place. You’re not jumping between dashboards, CLI tools, and cloud consoles just to complete a single workflow.
Time-to-First-Value (TTFV) stays consistently low It shouldn’t take hours to figure out how to get a model running. A good platform makes the “first successful response” happen quickly — not just in ideal conditions, but even when you’re unfamiliar with the setup. If you’re spending time debugging environment issues instead of validating outputs, that’s a design failure, not a user error.
The path from prototype to scale doesn’t change shape One of the biggest failure points in current platforms is that the workflow breaks when you scale. A well-designed system keeps the same mental model — the way you deploy and interact with a model at a small scale should still work when traffic increases. You shouldn’t need to relearn everything just to handle more requests.
Infrastructure decisions are abstracted until they actually matter You shouldn’t need to think about GPU types, networking, or provisioning just to test an idea. Good platforms delay these decisions without hiding them completely — they only surface when you have a real reason to care, like optimizing latency or cost.
Failure modes are visible and easy to debug When something breaks, it’s obvious where and why. You’re not digging through multiple systems trying to trace a failed request. Logs, errors, and performance signals are tied directly to the workflow you’re already using, so debugging doesn’t become a separate project.

Conclusion

The hardest part of building AI systems today isn’t getting access to models or GPUs, but it’s everything that happens around them.

It’s the time lost moving between tools.

It’s the friction of stitching together workflows that were never designed to work as one.
It’s the moment when something that worked at a small scale suddenly forces a complete rewrite.

And most of this doesn’t show up in benchmarks or pricing comparisons. It shows up in delays, workarounds, and abandoned ideas.

The teams that will win on inference aren’t the ones with the most compute. They’re the ones that can move from idea to working system and then to scale without having to change how they build along the way.

The real question isn’t which platform has the best features.

It’s this:
How many times does your workflow break before you get to something that actually works?

References

How to Deploy Hermes' Self-Improving AI Agent

haimantika mitra — Tue, 26 May 2026 16:00:00 +0000

Key takeaways

You host Hermes on a DigitalOcean Droplet (Ubuntu 24.04, at least 2 vCPUs and 4 GB RAM), install with the official script, and run hermes setup for your LLM provider.
The Telegram bot uses hermes gateway setup and a persistent gateway service so Hermes stays reachable after you close SSH.
Skills are Markdown files under ~/.hermes/skills/. MCP servers go in Hermes config and expose tools such as grocery search and cart actions.
HTTP MCP with OAuth on a headless server uses the URL Hermes prints plus an SSH tunnel from your laptop browser to the Droplet loopback port.
The grocery walkthrough uses Swiggy Instamart where available; you swap in another MCP URL for your region or stack.

2026 is the year of agents doing the task for us rather than just guiding us to. We saw the rise of OpenClaw and now we have Hermes agent. It is an open-source, self-improving AI agent built by Nous Research. It has a built-in learning loop that creates skills from experience, improves them during use, nudges itself to persist knowledge, and builds a deepening model of who you are across sessions.

In this tutorial, you will deploy Hermes on a DigitalOcean Droplet, connect it to Telegram, and extend it with a custom skill. As a practical example, you will see how to build a grocery tracking agent that monitors daily consumption, alerts you when stock is low, and places orders automatically through a grocery delivery service.

The grocery example uses Swiggy Instamart, which is available in India. But the approach works with any MCP-compatible service, such as a local grocery API, a task manager, a calendar, or a home automation system. The pattern is the same no matter what you connect.

Prerequisites

Before you begin, you will need:

A DigitalOcean account. If you do not have one, sign up here.
A DigitalOcean Droplet running Ubuntu 24.04 with at least 2 vCPUs and 4 GB RAM.
A Telegram account.
API key from an LLM provider. Hermes supports Anthropic, OpenAI, OpenRouter, and others.

What are we building

Hermes is model-agnostic and platform-agnostic. It can connect to any tool that supports the Model Context Protocol (MCP), and it can send and receive messages on Telegram, WhatsApp, Discord, and Slack. You can find more information on their documentation.

The architecture in this tutorial looks like this:

Let’s get started building:

Step 1. Create the Droplet

On the creation page, select:

Region: The one closest to you
Image: Ubuntu 24.04 LTS
Plan: Basic, with at least 2 vCPUs and 4 GB RAM (the s-2vcpu-4gb size)
Authentication: SSH Key

If you need help adding an SSH key, follow the DigitalOcean SSH key guide.

Once the Droplet is created, SSH into it:

ssh root@YOUR_DROPLET_IP

Update the system before installing anything:

apt update && apt upgrade -y

Step 2. Install Hermes Agent

Hermes provides an install script that handles all dependencies including Python, uv, and the hermes binary itself.

curl -fsSL https://raw.githubusercontent.com/NousResearch/hermes-agent/main/scripts/install.sh | bash

Once it finishes, reload your shell so the hermes command is available:

source ~/.bashrc

Run the setup wizard:

hermes setup

The wizard will ask you to choose an LLM provider and enter your API key. Select your provider, paste your key, and Hermes will save it to ~/.hermes/.env.

Verify the installation:

hermes --version

Step 3. Connect Hermes to Telegram

Hermes connects to Telegram through a bot. Run the gateway setup:

hermes gateway setup

When it asks which platform to use, select Telegram. Hermes will walk you through creating a bot with Telegram's BotFather. Follow the steps in your terminal.

Once setup is complete, start the gateway:

hermes gateway start

To keep it running after you disconnect from the Droplet, enable it as a system service:

systemctl enable hermes-gateway
systemctl start hermes-gateway

Open Telegram, find the bot you just created, and send it a message like "hello." If it responds, Hermes is connected and ready.

You can now talk to Hermes from anywhere on your phone. Ask it to check the weather, set a reminder, search the web, or run a command on your Droplet. It handles all of this out of the box.

Step 4. Understand skills and MCP servers

Before building the automation example, let's understand two core Hermes concepts:

Skills - These are Markdown files that teach Hermes how to handle specific tasks. You write a skill file describing what the task is, what triggers it, and what steps to follow. Hermes reads all skill files in ~/.hermes/skills/ at startup and uses them to handle relevant requests.

MCP Servers - It gives Hermes access to external services. MCP (Model Context Protocol) is an open standard that lets AI agents communicate with APIs in a structured way. If a service publishes an MCP server, Hermes can search its products, manage carts, read calendars, create tasks, and more. You add MCP servers to your Hermes config and Hermes uses them automatically when relevant.

Together, skills and MCP servers let you build automations that are specific to your life and the tools you use.

Step 5. Build real-world automation: Grocery tracking

Personally, I am very bad at keeping a tab of groceries and often find myself out of essential stock just when I am about to cook. So I wanted to automate this process for me. I eat similar food everyday and also measure my calories. I used all of this information to automate grocery shopping for me.

The goal of Hermes was to track my daily grocery consumption, alert me on Telegram when something is running low, and place an order through a grocery delivery service when I say yes.

This example uses the Swiggy Instamart MCP server, which is available in India. If you are in a different country, you can swap it out for any MCP-compatible grocery or delivery service. The skill logic stays exactly the same.

Creating the skill file

Create a directory for the skill:

mkdir -p ~/.hermes/skills/grocery

Create the skill file:

nano ~/.hermes/skills/grocery/grocery_tracker.md

Paste the following and edit the groceries: section to match your actual diet and quantities:

# Grocery Auto-Order Skill

### My Daily Grocery List

groceries:
  - name: "Milk"
    unit: "ml"
    daily_consumption: 500
    reorder_quantity: 2000
    alert_threshold_days: 3
    search_query: "fresh milk 1 litre"

  - name: "Eggs"
    unit: "pieces"
    daily_consumption: 2
    reorder_quantity: 12
    alert_threshold_days: 3
    search_query: "eggs 12 pack"

  - name: "Oats"
    unit: "gm"
    daily_consumption: 60
    reorder_quantity: 1000
    alert_threshold_days: 7
    search_query: "rolled oats 1kg"

  # Add your own items following the same format

### Instructions for Hermes

#### Daily check
Read ~/.hermes/grocery_inventory.json. Subtract each item's
daily_consumption from its current quantity. For any item where
quantity / daily_consumption is less than or equal to
alert_threshold_days, send a Telegram alert listing the low
items and ask if the user wants to place an order.

#### On YES
Search each low item using its search_query at the user's saved
delivery address. Add reorder_quantity of each to cart. Send a
cart summary via Telegram with prices and total. Wait for CONFIRM.

#### On CONFIRM
Place the order. Update grocery_inventory.json with the restocked
quantities. Send a delivery confirmation message.

Save with Ctrl+O, then Enter, then exit with Ctrl+X.

Creating the inventory file

The inventory file tracks how much of each item you have at home right now.

nano ~/.hermes/grocery_inventory.json

{
  "last_updated": "2026-05-07",
  "delivery_address": "YOUR FULL ADDRESS HERE",
  "snooze_until": null,
  "items": {
    "Milk": { "quantity": 1000, "unit": "ml" },
    "Eggs": { "quantity": 6, "unit": "pieces" },
    "Oats": { "quantity": 300, "unit": "gm" }
  }
}

Replace the quantities with what you actually have at home. Save and exit.

Connect to an MCP server

Add the grocery service MCP to your Hermes config:

hermes config edit

Scroll to the bottom of the file and add your MCP server. For Swiggy Instamart:

mcp_servers:
  swiggy-instamart:
    url: "https://mcp.swiggy.com/im"
    auth: oauth

For any other MCP-compatible service, replace the name and URL with the values from that service's documentation.

Save and exit, then verify it appears:

hermes mcp list

Step 6. Authenticate with your MCP server

Most MCP servers require OAuth authentication. Because your Droplet is headless, the OAuth callback needs to reach your browser through an SSH tunnel.

Run the login command on your Droplet:

hermes mcp login swiggy-instamart

Hermes will print a URL containing a port number in the redirect URI, like http://127.0.0.1:45123/callback. Note that port number.

On your local machine, open a new terminal and run the SSH tunnel using that port:

ssh -i ~/.ssh/id_ed25519 -L 45123:127.0.0.1:45123 root@YOUR_DROPLET_IP

Keep that terminal open, then immediately open the URL from your Droplet in your browser and complete the login. You have about 30 seconds before the token expires, so have both terminals ready before you start.

When authentication succeeds, your Droplet terminal will show:

✓ Authenticated — tools available

Telling Hermes Your Current Stock

Start Hermes and initialize the inventory:

hermes

Type:

Set up my grocery tracker. Ask me for current stock levels for each item in the grocery_tracker skill.

Hermes will ask about each item one by one. Answer with what you currently have at home and it will update grocery_inventory.json automatically. Once you have given the update, you will receive a message similar to this:

Setting up the daily alert

Still inside Hermes, set up the cron job:

Add a daily cron at 8am: check my grocery inventory using the grocery_tracker skill, subtract daily consumption, and send me a Telegram message if anything is running low.

Hermes will schedule this and confirm. From now on, every morning it checks your stock and messages you on Telegram if you need to reorder. This is how it looks:

Step 7. Test the full flow

Send a test message to your Telegram bot:

Check my groceries and tell me what's running low.

You should receive a Telegram message like this:

Reply YES. Hermes searches your connected grocery service, builds a cart, and sends a summary.

Once everything is set up, you can manage your agent entirely from Telegram:

Message	What Hermes does
`Check my groceries`	Shows all items and days of stock remaining
`I bought 12 eggs`	Updates egg stock in the inventory file
`Order groceries now`	Skips the check and goes straight to ordering
`What's running low?`	Lists items near their alert threshold
`SKIP`	Snoozes today's alert until tomorrow

Conclusion

You now have a self-hosted AI agent running on DigitalOcean that works for you around the clock. Hermes connects to Telegram and more messaging platforms so you can reach it from anywhere, and skills plus MCP servers let you extend it to handle almost anything.

The grocery automation you built in this tutorial is a starting point. The pattern is reusable: write a skill file that describes the task, connect an MCP server that gives Hermes access to the right service, and set a cron job to trigger it automatically. The grocery automation is one example. The same pattern works for any repetitive task in your life. A few ideas to get you started:

Bill reminders- Create a skill that tracks recurring payments, calculates due dates, and alerts you three days before each bill is due.

Health tracking- Log your workouts or meals via Telegram and have Hermes summarize your week every Sunday.

Home automation- Connect Hermes to a smart home MCP server and have it adjust lights, thermostats, or appliances based on a schedule or your location.

April 2026 DigitalOcean Tutorials: Inference Optimization and AI Infrastructure

DigitalOcean — Fri, 22 May 2026 21:55:02 +0000

Most AI teams hit the same walls once they move past prototyping. The RAG pipeline that worked flawlessly in a demo starts hallucinating under real traffic. Inference costs climb without clear optimization levers. GPU resources sit underutilized while workloads spike elsewhere.

Most of the time, the root cause traces back to architecture decisions that weren't pressure-tested for production. This month's DigitalOcean tutorials focus on diagnosing and fixing those failure points across the AI infrastructure stack.

Why RAG Systems Fail in Production

Why do seemingly solid RAG demos collapse under real-world conditions? This article traces failures back to retrieval quality, latency tradeoffs, and embedding drift. You’ll get a clear picture of how upstream decisions—such as chunking strategy and ranking—directly affect downstream LLM outputs. If your team is building production pipelines, evaluation, monitoring, and retrieval engineering matter just as much as model choice.

Dedicated vs. Serverless Inference as You Scale

The choice between serverless and dedicated inference isn't a one-time decision but an evolution driven by how your workload changes over time. Early on, serverless makes sense because traffic is unpredictable and iteration speed matters more than performance optimization. As usage stabilizes, the cracks show up—latency variability frustrates users and per-request pricing gets expensive for always-on systems. Walk-throughs of Modal and Together.ai show where that transition point hits and why delaying it costs you.

Fine-Tuned LLMs on Serverless Architecture

Parameter-efficient methods like LoRA let platforms serve hundreds of fine-tuned model variants from a single GPU by layering small adapter weights on top of a shared frozen base model. This makes serverless, pay-per-token inference possible for custom models without dedicated GPU deployments. The tradeoff is cold starts: idle adapters get evicted from VRAM and need to be reloaded, adding a few hundred milliseconds of latency to the first token. You’ll learn how to minimize that with keep-alive requests, adapter rank tuning, and smarter layer targeting.

The Silent Versioning Problem in AI Inference

This one is a cautionary tale about what happens when the model behind your endpoint changes and nobody tells you. The serving stack is full of moving parts that can shift independently of the model name, and the result is silent regressions that break prompt tuning and invalidate your evaluations before you even know something moved. It includes a practical buyer's checklist for pressing inference platforms on snapshot pinning, retention commitments, and how they handle disclosure when something in the stack changes.

The Hidden Bottlenecks in LLM Inference and How to Fix Them

Faster GPUs are not the answer if the rest of your serving stack can't keep up. Spoiler: the bottlenecks are GPU underutilization from rigid batching, memory bandwidth constraints during decode, KV cache fragmentation, and CPU-side overhead from tokenization and prompt assembly. Click through for a deeper look at each one and practical fixes.

We Built a Private-Document AI App to Test Platform Security. Here Is What We Could Actually Verify

AI security should always be treated as a first-class concern, not an afterthought. This tutorial puts that to the test by building a private-document chatbot and running the same workflow across six inference platforms: DigitalOcean, Baseten, Nebius, Fireworks AI, Modal, and Together AI. Each platform is evaluated on access controls, data retention defaults, network isolation, audit logging, and shared responsibility clarity. It doubles as a practical framework for figuring out what you can actually verify before sensitive data is in flight.

Post-Inference Storage and Querying with MongoDB

Many inference tutorials stop at the model response. This one keeps going. You'll build a FastAPI app that sends images through a vision model, stores the structured predictions in MongoDB, and then exposes endpoints that let you filter by detected labels and confidence scores or run aggregation pipelines across your full dataset. It's a practical blueprint for turning raw model output into something queryable and operational.

How to Build a Multi-Agent AI System with Docker and DigitalOcean

Instead of routing everything through a single model, multi-agent systems let you split a workflow across specialized agents that each handle a different part of the problem and pass results between them. The tradeoff is coordination complexity. This walkthrough covers how to containerize each agent with Docker, manage communication between them, and deploy the full system on DigitalOcean. You'll come away with a working deployment pattern you can adapt to your own orchestration needs.

Building an AI-Powered GPU Fleet Optimizer with the DigitalOcean AI Platform ADK

A single idle GPU Droplet left running overnight can add hundreds of dollars to your monthly bill, and standard CPU monitoring won't catch it because it can't see whether the GPU is actually doing work. This tutorial builds an AI-powered agent using the DigitalOcean AI Platform ADK that scrapes NVIDIA DCGM metrics like VRAM usage, engine utilization, and power draw across your fleet in real time. It compares those metrics against configurable thresholds to flag idle resources before they inflate your cloud spend. The repo is designed to be forked and customized to your own workloads, including adding tools that let the agent take action like powering off idle nodes.