DEV Community: DigitalOcean

How do I cap inference billing to avoid surprise costs from idle GPUs or runaway agents?

DigitalOcean — Wed, 14 Oct 2026 16:31:01 +0000

You cap inference billing by layering controls, not by flipping one switch. On DigitalOcean, Serverless Inference is prepaid, so your balance is a real ceiling, and scoped keys, alerts, and cleanup jobs cover the rest. Spend alerts alone won't stop anything: they email you when spending crosses a threshold, but your usage keeps running.

Key takeaways:

Serverless Inference stops when funds run out, so fund a dedicated balance with only what you plan to spend.
Dedicated Inference and GPU Droplets bill by the hour until destroyed, so automate cleanup.
Agents need caps in your own code, because rate limits slow spending but don't cap it.

Use prepaid balances as a ceiling

Serverless usage draws from promotional credits, then the Inference and Agents balance, then your account prepayment balance. Only Serverless Inference and Managed Agents draw from the first, so it works as a separate AI budget. Auto-reload is on by default, so turn it off for a hard stop. A zero balance is also an outage, so reserve this for experiments and non-critical workloads.

Add alerts and scope your keys

Spend alerts email you at percentage thresholds of your monthly budget, within an hour of each threshold being crossed. Their sole purpose is notification. Give each service its own model access key, scoped to specific models and a VPC, so one leak or loop has a small blast radius. Regenerating a key takes effect immediately, which is how you want it to work during an incident.

Cap agents in code

Rate limits (up to 120 requests and 1.5M tokens per minute at the lowest serverless tier) limit speed, not total spend. Harness Runtime has no per-session or per-product spend limit, so an agent stuck in an unintended loop can quickly drain balances. Set a max output token count per call, a max step count per task, and a running spend counter that stops the run. You should also pause idle sessions, since paused sessions incur no compute charges.

Stop paying for idle GPUs

Dedicated Inference bills per GPU-hour, and GPU Droplets bill per second until destroyed. Powering off doesn't stop the meter. At \$30.32 per hour for 8x H100, an idle Dedicated Inference endpoint costs roughly \$728 a day.

To cap inference billing and avoid surprise costs, tag experimental GPU Droplets and destroy them nightly:

doctl compute droplet delete --tag-name ephemeral-gpu --force

Inference billing cap FAQ

Can I get usage alerts or hard spending caps on a managed AI platform before bills run away?

DigitalOcean spend alerts notify you at budget thresholds but don't limit usage. The closest hard cap is Serverless Inference's prepaid model, where access is suspended when credits and balances run out.

Does powering off a GPU Droplet stop billing?

No. The GPU stays reserved, so billing continues until you destroy the Droplet.

Which inference option doesn't penalize variable workloads?

Serverless Inference, which bills per token instead of reserving GPUs, is a solid choice for variable workloads. Dedicated Inference bills per GPU-hour, whether or not anyone calls it, so it suits steady traffic better than spiky traffic.

References & further reading

Spend alerts: worth setting up first.
Manage Serverless Inference prepayment: explains the order balances are drawn down.
Learn more about model access keys and inference limits.
Inference pricing: worth checking directly before you build.
Droplet pricing and billing
Harness Runtime pricing: worth bookmarking before you run agents at scale.

We moved a production backend from a phone

DigitalOcean — Tue, 13 Oct 2026 13:19:33 +0000

Published by Vinay Kumar, Chief Product and Technology Officer at DigitalOcean

Every era of the cloud has a unit of work. For agents, it’s a human who directs, a controller who runs the operation, and a worker who does the heavy lifting. Last week we watched all three move a production backend, from a phone.

I’ve spent fifteen years building cloud services: S3 and EBS in the early days of AWS, Oracle Cloud Infrastructure from inception, and now the agentic cloud at DigitalOcean. In every era, moving a backend meant the same thing: a project plan, a maintenance window, and a team working late into the night.

Last week, Rohit Bhattacharjee on our team moved an entire production backend from his phone. Database, APIs, file storage, and about 31,000 AI search embeddings. He never opened a laptop. A message while waiting for coffee. A decision between meetings. By the end of the week, it was done.

This wasn’t a demo. It was a real workload with a bill.

Three parties, three jobs

The worker was a DigitalOcean Managed Agent, an AI that does real work in the cloud, not just chat. It ran in our Harness Runtime: a private, isolated computer that spins up for each session, with a workspace that persists when the agent pauses. Through our Action Gateway, a single connection to 20,000+ tools, it could use GitHub, Supabase, a web browser, and the whole DigitalOcean API.

The agent never saw a password. The gateway applies credentials at the moment of use, so they never reach the model.

It provisioned the database, loaded all 40 access policies, and mapped every Supabase feature to its DigitalOcean counterpart. It wasn’t advising on a migration. It was performing one.

The controller was a personal agent running on his phone, the agent Rohit actually talked to. He ran the migration in plain chat, and the personal agent turned his messages into work: tasking the worker, tracking the run, and reporting back. When he asked it to check a claim, it did. The worker said 145,432 rows had been loaded, so the controller counted them. Validation was the side job. The main job was holding the whole operation together in one conversation, so a person never had to leave the flow of his day.

The worker and controller come from different model families. That’s the point of an open platform: you choose the right model for each job.

Rohit provided the credentials, his judgment, and a credit card. He reviewed a $180 proposed plan and saved almost 80% on a revision with just a few taps. Everything else, the two agents handled between themselves.

What moved

The workload was huffl.ai, an agentic canvas that puts docs, spreadsheets, whiteboard, presentations, and search in one place. Underneath: 31 tables, 145,432 rows, three storage buckets, 24 functions, and 30,650 embeddings, the numeric fingerprints that make search work.

Before moving anything, the worker read all of it and caught two problems that would have poisoned the migration. Two overloaded functions that looked like duplicates. A “vector count” that was really measuring string lengths.

The new home: managed Postgres ($15), one App Platform app running four services ($20), and one object store ($5). About $40 a month. The data moved with per-table parity checks, and every one of the 30,650 embeddings verified non-null.

Pay for the work, not the wait

An agent spends most of its session waiting: on a model, a tool, or a person. The runtime paused the worker whenever it waited and resumed it in about 300 milliseconds. While it slept, the meter stopped. We bill for active CPU, not provisioned capacity. The agent runtime for the entire run cost a few dollars.

What this changes

Moving a backend used to mean a team, a plan, a maintenance window, and a war room. Last week it meant chat bubbles. The human says what needs to happen, the controller runs the operation, the worker does the work, and every claim gets checked along the way.

The work didn’t get smaller. What one person can accomplish got bigger. See it in action: watch the video:

Try DigitalOcean Managed Agents today. Bring your preferred harness, connect your tools, and give your agents a complete environment with our Cuttlefish mobile application coming soon. Get started today.

What would you trust an agent to move for you, and what would you want checking its work?

Jev: an AI model that can’t write a sentence

DigitalOcean — Fri, 09 Oct 2026 14:00:00 +0000

ICYMI AI: The latest AI trends, delivered to you on Dev.to

Tech Talk: TypeSafe’s Jev is a model that only returns probabilities

TypeSafe's Jev, released into early access on September 15, is a general-purpose classification model. It reads up to 32K tokens of input (such as a support ticket, a customer record, or a row of data). It answers any number of typed questions about it in a single parallel pass, and returns probabilities rather than prose.

Outputs are limited to three options: Choice, Score, and Noul (yes or no), so structured-output errors and bad tool calls drop to zero by construction. A valid answer isn't always the right one, and Jev doesn't claim otherwise.

What it does is tell you how sure it is. TypeSafe trains it for calibration, so its probabilities match real-world frequencies. When Jev says it's 80% confident, it's right about 80% of the time. That lets you set a threshold, automatically acting on high-confidence calls and routing the rest to a human or a larger model.

The pricing is what makes Jev worth a look. At \$0.042 per million input tokens with free output, TypeSafe estimates about \$0.0004 per decision, roughly 1/76th the cost of GPT-5.6 Terra on its benchmark. Within days, imitators appeared: Laya, Bespoke Nimble, and Kev, several of them open-weight fine-tunes of Qwen 3.5 you can run locally. OpenAI likewise followed with a Decisions API of its own.

For developers, the change is in how you structure the call. Instead of sending an LLM one broad question like "is this spam?" and parsing a paragraph back, you pass Jev the email once, along with a set of narrower typed questions, such as: Does the sender domain match the claimed company? Does it ask for credentials? How urgent is the language? Each comes back as a probability. Your own code then applies the thresholds and makes the call, so the routing policy lives in your repo, versioned and auditable, and the model only scores the inputs.

Jev is available today on Serverless Inference through the DigitalOcean Model Library, at Typesafe’s list price—no separate TypeSafe account needed. Pair it with the DigitalOcean Inference Router to send routing and classification traffic to Jev while keeping frontier models in reserve for the calls that need them.

Event calendar

Open Intelligence Summit

October 12-13th, San Francisco

The argument is no longer “Is open source AI cheaper?” It’s now “Who owns intelligence?” A movement is forming around open intelligence and generating the belief that the intelligence powering our AI products and companies should be more open, portable, and controllable.

Join DigitalOcean, NVIDIA, RadixArk, Inferact, and more for deep dives, demos, and discussions about the technologies that keep intelligence portable, interoperable, and in the hands of builders.

Industry news

Near-flagship intelligence at a fifth of the price was the headline at OpenAI DevDay, but the bigger shift is the stack forming around the model.

OpenAI announced GPT-6.1 Sol, which it says delivers near-Astra intelligence at a fifth of Astra's token prices, plus an Ultrafast tier with up to 6x faster generation in the API. It also added computer use to its Agents API and opened ChatGPT, which it puts at 1.2B weekly users, to developer-built plugins and partners like Devin, Notion, and Vercel.

Agent safety is moving below the application layer.

NVIDIA's Open Agent Safety Platform pairs OpenShell, an open-source runtime that traces agent actions and enforces policy, with Sentry, an out-of-band watchdog on BlueField-4 DPUs that NVIDIA says can quarantine an agent that leaves its boundaries within milliseconds. NVIDIA says more than 100 organizations, including Anthropic and CrowdStrike, are working with the technology, and DigitalOcean is among the companies supporting the platform.

New models available on DigitalOcean Inference Engine

We recently launched models from TypeSafeAI, DeepSeek, Z.ai, NVIDIA, OpenAI, and Anthropic, all with different strengths and use cases. Learn more about the models in our What’s New blog.

Start using them through the DigitalOcean API or directly in the Cloud Console.

DeepSeek-V4.1-Flash

DeepSeek-V4.1-Flash cuts the KV cache footprint to 1/4 of V4-Flash while outperforming DeepSeek's own V4-Pro flagship on coding and agentic benchmarks, addressing the cache-hit charges that often dominate agent costs. The 552B MoE model activates just 8B parameters on input, delivering frontier performance with infrastructure economics that improve as you scale.

GLM-5.3 and GLM-5.3-Flash from Z.ai

Z.ai's post-training translates to significant performance improvements for GLM-5.3, achieving a 50% lower cost per completed coding task than GLM-5.2 while jumping to 28.3 on Terminal-Bench 3.0. GLM-5.3-Flash adds native image and video input with the same agentic strength, giving teams both reasoning-heavy and efficient multimodal paths on a single platform.

Nemotron™ 3 Diarization from NVIDIA

NVIDIA's Nemotron™ 3 Diarization replaces the multi-stage speaker attribution pipeline (VAD, embeddings, clustering, post-processing) with a single 100M-parameter model, keeping audio within DigitalOcean's infrastructure rather than routing it to third-party APIs. It identifies up to 8 speakers (double the previous limit) with a 14.72% error rate.

GPT-6.1 Sol, GPT-6 Astra, Sol, and Luna from OpenAI

One week after GPT-6 Sol’s launch, OpenAI releases GPT-6.1 Sol, which offers near-Astra-level coding and substantial improvements in document understanding, computer use, and scientific work. OpenAI's three-tier GPT-6 family also debuts on DigitalOcean: Astra for computer use and coding, Sol for frontier reasoning at 90% lower cost than Opus 5, and Luna for high-volume tasks.

Claude Fable 5.1, Opus 5.5, and Sonnet 5.5 from Anthropic

Anthropic's latest point releases deliver focused gains: Fable 5.1 reaches 55.8% on Terminal-Bench 4.0 with 75% cheaper cache reads, while Opus 5.5 matches high-effort results at default effort using half the tokens and 20% lower per-token cost. Sonnet 5.5 runs 30% faster with equivalent cost savings, with a shift toward better token efficiency and lower verification burden.

Community tutorials

How naive failover between inference providers makes outages worse

In a six-hour fault-injection run on an H200 GPU Droplet, the standard retry-with-backoff policy pushed 3.54x normal load onto a degraded provider, while a retry budget plus circuit breaker held it to 1.02x, and naive retries left 3,743 zombie requests generating 959,232 output tokens for clients that had already disconnected. Learn how to classify inference errors before failing over, replace fixed retry counts with a budget, and verify whether your provider actually stops billing when you cancel a stream.

What vision-language models cost to serve at production scale

A single uploaded photo can add more than a thousand input tokens before the model writes a word, and at 500,000 requests a month, that gap is the difference between \$155 on serverless Nemotron Nano 12B VL and \$3,175.20 for a dedicated H100 running 24/7. Walk through the per-request cost math, the roughly 10.2 million-request breakeven, where dedicated wins, and six levers for cutting image-token spend.

Speculative decoding's hidden cost: the KV cache capacity you give up, and where throughput turns negative

On a 141 GB H200 serving Qwen3-14B with vLLM, a separate Qwen3-0.6B draft model consumed about 43.8 percent of baseline batch slots, and even n-gram speculation, predicted to cost nothing, measured a 2.8 percent capacity tax. See where each method's throughput crosses below baseline as concurrency rises, and get the config.json arithmetic to predict a draft model's KV cost before spending GPU hours.

Cache reliability is a pricing term: what I learned measuring DeepSeek-V4.1-Flash on DigitalOcean Serverless Inference

DeepSeek-V4.1-Flash has the deepest cache discount in the Serverless Inference catalog at \$0.006 per million cached tokens, yet across 844 API calls it hit the cache on only 78.6% of warm turns, and with a miss priced at 50x a hit, it cost 2.2x what GLM-5.3-Flash did for the same 15-turn, 200K-token sessions. Discover how DigitalOcean's 64-token-block prefix cache works, how to log your own hit rate from the usage object, and why reasoning_effort: max cost up to 1.88x more without improving accuracy.

Why DeepSeek-V4.1-Flash splits reading from writing

DeepSeek-V4.1-Flash's Causal Encoder-Decoder splits a 40-layer model into a 20-layer encoder that runs prefill at 8B active parameters and a full stack that decodes at 16B, shrinking the global KV cache to about 890 bytes per token, roughly a quarter of V4-Flash's. Learn why that asymmetry, not the 74.2 versus 62.7 DeepSWE score against V4-Pro, is what lowers the bill for read-heavy coding agents.

Product updates

Introducing DigitalOcean Managed Agents, now in Public Preview

AI agents are ready to take on more, and we built the infrastructure to match. DigitalOcean Managed Agents brings together Harness Runtime, a purpose-built runtime for running agent sessions at scale, and Action Gateway, governed access to the tools agents need to get real work done, all in one platform.

Deploy your preferred agent harness, Claude Code, Codex CLI, OpenCode, and more, or bring your own, and connect to 16,000+ tools across 500+ providers through a single managed endpoint. Sessions are ready in under a second, resume in about 300 milliseconds with full context intact, and you only pay for the compute you actually use.

Whether you're automating support workflows, refactoring code across repositories, or building the next agent-powered product, Managed Agents gives you the infrastructure to do it, without building it yourself.

DigitalOcean MicroVMs now in private preview

Isolated, Firecracker-based VMs that pause when idle and resume with state intact, built for agent workloads that run in bursts. The same layer that powers Managed Agents is now available directly to teams building their own agent platforms, sandboxes, and short-lived jobs.

What is disaggregated prefill and decode in LLM inference?

DigitalOcean — Tue, 06 Oct 2026 18:03:00 +0000

What is disaggregated prefill and decode in LLM inference?

Prefill is compute-bound, decode is memory-bound. Learn why colocating them hurts TTFT and ITL, and when disaggregated serving pays off.

Every LLM request runs through two phases with effectively opposite hardware profiles. Understanding the split explains most of the latency and cost behavior you see in production, and why disaggregated prefill and decode has become a standard serving pattern.

Key takeaways:

Prefill is compute-bound and sets TTFT; decode is memory-bandwidth-bound and sets ITL/TPOT.
Running both on the same GPUs creates prefill/decode interference, visible as TTFT spikes and ITL jitter.
Chunked prefill and continuous batching reduce interference; disaggregating the phases onto separate pools removes it, at the cost of a KV cache transfer.
Disaggregation is worth it for long prompts, high concurrency, and strict streaming SLOs. A managed option like DigitalOcean Serverless Inference gives you it without the setup; for small self-hosted deployments, a well-tuned single pool on a GPU Droplet running vLLM is often enough.

The two phases of LLM inference

Prefill processes the entire prompt in one forward pass and builds the KV cache, the attention state every later token reads from. It is compute-bound: arithmetic intensity is high and the GPU spends its time doing math, not waiting on memory. Prefill governs time to first token (TTFT), and its attention cost grows quadratically with prompt length.

Decode generates output one token at a time. Each step reloads the model weights and the growing KV cache from HBM to produce a single token, so it is memory-bandwidth-bound. Decode governs inter-token latency (ITL) and time per output token (TPOT). A GPU with higher memory bandwidth streams tokens faster even if its FLOPs are unchanged.

Why colocating prefill and decode causes latency problems

Most serving engines batch prefill and decode on the same GPUs. That is where the interference shows up: a long prompt's prefill can monopolize compute while dozens of in-flight decode requests stall, and users see ITL jitter or a TTFT spike. The two phases also want different batch sizes and parallelism strategies, but colocated they must share one configuration.

Chunked prefill and continuous batching soften this by interleaving prompt chunks with decode steps, which is why our inference trilemma guide recommends them for latency-sensitive deployments.

What disaggregation changes

Disaggregated serving, formalized in the DistServe paper, runs prefill and decode on separate GPU pools. A prefill worker computes the KV cache, ships it over a fast interconnect (RDMA, RoCE, or a transfer library like NIXL or LMCache) to a decode worker, and the decode pool streams tokens without prefill interference. Because each pool is sized and tuned independently, you can hit TTFT and ITL targets at the same time rather than trading one for the other. The cost is added complexity plus a KV transfer step; that transfer can land on second-token latency rather than TTFT depending on how orchestration is done.

This is the architecture DigitalOcean already runs in production: Serverless Inference uses disaggregated serving over a RoCE network, paired with prefix-aware routing so shared system prompts skip redundant prefill entirely.

References & further reading

DistServe: Disaggregating Prefill and Decoding for Goodput-optimized LLM Serving: the primary source for the pattern, worth reading before you build.
LLM inference benchmarking: measure what matters: a clear walkthrough of TTFT, TPOT, ITL, and how each maps to prefill or decode.
The LLM inference trilemma: practical tuning advice, including when to reach for chunked prefill.
Learn more about debugging p99 TTFT and deploying NVIDIA Dynamo on GPU Droplets, a framework built around disaggregated serving.
GPU Droplets: current hardware and regions, worth checking directly since availability changes.

How to route LLM requests by cost vs. latency

DigitalOcean — Fri, 02 Oct 2026 18:33:17 +0000

Routing LLM requests by cost and latency means sending each request to the cheapest or fastest model that still meets the quality bar, rather than hardcoding a single model for everything. Production traffic isn't uniform: routine lookups, complex troubleshooting, and background jobs have different cost and latency requirements, even within a single app.

Using an inference router like DigitalOcean Inference Router involves defining request categories, assigning each a selection policy (cost, latency, or a benchmarked best fit), and setting fallback models so an unavailable model doesn't break the request. Cost is the price per token (which can vary by up to 100x across a model catalog); latency is mainly the time to first token (TTFT). The two don't move together, so route per task rather than using a single blended score.

Key takeaways:

Routing is a per-workload policy, not a search for "the best model."
Cost and latency usually need separate, explicit priorities.
Fallback and cache-aware routing protect the savings that routing is meant to deliver.
The DigitalOcean Inference Router implements this as configuration, not custom infrastructure.

LLM routing strategies

Static rules: Hardcode model per request type. Simple, but brittle.
Policy-based routing: A model pool per task with a cost/latency/optimal policy. Most production systems land here.
Dynamic routing: A classifier infers the task and policy automatically.

Routing LLM requests: A quick decision framework

Workload	Priority	Policy
Real-time chat	Latency	Speed-optimized (TTFT)
Routine lookups	Cost	Cost-optimized
Complex reasoning	Quality	Benchmarked "optimal"
Batch/background	Cost	Cost-optimized

Fallback models keep requests completing when the preferred model is down or rate-limited.

Cache-aware routing matters too: switching models to save a fraction of a cent can break a cached prompt and cost more overall.

How the DigitalOcean Inference Router routes LLM requests

Inference Router tasks pair a model pool with a selection policy: preset tasks default to a benchmarked Optimal policy; custom tasks choose Cost Efficiency, Speed Optimization (TTFT), or Manual Ranking.

Fallback models handle unmatched traffic. X-Model-Affinity preserves cache reuse, and X-Routing-Max-Switch-Spend-Pct (default 20%) caps the cost of switching mid-session.

It's a drop-in change. Set "model": "router:your-router-name". Routing decisions typically resolve in about 200ms, billed at the serving model's standard rate with no separate router charge during public preview.

LLM routing FAQ

What is LLM routing?

LLM routing is the practice of directing each request to the model best suited to it—by cost, latency, or task fit—instead of sending every request through one model regardless of what it needs. The DigitalOcean Inference Router implements this as a managed feature, so teams get task-aware routing and fallback handling without building it themselves.

How do you model cost-per-token for LLM inference?

Multiply input tokens by the model's input rate and output tokens by its output rate, then sum the two, since the rates are usually very different. Because rates vary widely across a model catalog, this is best tracked per task rather than as one blended number. The DigitalOcean Inference Router reports token usage and cost-relevant metrics per model and per task in its Analyze view.

What metrics matter for LLM inference observability?

Time to first token (TTFT), time per output token (TPOT), and inter-token latency (ITL) are the core metrics for judging responsiveness. A latency-optimized routing policy should be measured against these, not just overall request time. The DigitalOcean Inference Router uses TTFT specifically as the basis for its Speed Optimization selection policy.

What causes cold start latency in GPU inference, and how do you avoid it?

Cold starts happen when a model has to load onto available GPU capacity before it can serve a request, rather than running on an already-warm instance. Pooled serverless capacity and routing policies that keep related traffic on one model reduce how often this happens. The DigitalOcean Inference Router uses model affinity to keep a session's requests on the same warm model, rather than triggering repeated cold starts across models.

Can I route by both cost and latency at the same time?

Yes—typically by defining separate tasks or policies for different request types rather than one blended score. For example, cost-first for background work and latency-first for real-time chat. The DigitalOcean Inference Router supports this directly. Each task in a router can have its own model pool and its own Cost Efficiency, Speed Optimization, or Manual Ranking policy.

References & further reading

How to use Inference Router: The full documentation on tasks, selection policies, fallback models, and the cache-aware router referenced throughout this piece. Worth reading directly before you configure a router for production traffic.
Inference pricing: Current per-model Serverless Inference rates, since routing decisions are only as good as the pricing data behind them.
How DigitalOcean built Inference Router: An inside look at the architecture decisions behind cache-aware routing and fallback handling, useful if you want the reasoning behind the mechanics.
DigitalOcean Serverless Inference: a deep dive: Background on the serverless layer that Inference Router sits in front of.
DigitalOcean's best LLM routers roundup: A good next read if you're comparing routing options across vendors rather than implementing a policy on a platform you've already chosen.
AI Inference Engine overview: shows where Inference Router fits alongside serverless, batch, and dedicated inference on DigitalOcean's broader platform.

Which inference provider has both serverless and dedicated GPU tiers on the same platform with one API?

DigitalOcean — Mon, 28 Sep 2026 17:17:40 +0000

DigitalOcean's Inference Engine puts serverless, dedicated, and batch inference behind a single endpoint, on the same GPU Droplets infrastructure, so a workload can move from pay-per-token prototyping to a reserved-GPU deployment without switching platforms or rewriting its integration. A handful of other inference providers share a single API across serverless and dedicated tiers, too, but they generally limit that unification to inference alone, rather than to the broader cloud the dedicated GPUs run on.

Why serverless-only or dedicated-only platforms create a migration tax

Most teams don't start with a clear answer to "How much GPU capacity do we need?". They prototype on a pay-per-token endpoint, and only once traffic gets steady and predictable does it make financial sense to reserve GPUs outright. Many inference providers treat serverless and dedicated as separate products, with distinct SDKs, auth, and billing dashboards, so moving from one to the other means rewriting integration code in the middle of a product's growth curve, which is exactly when a team has the least time to do it. A provider that puts both tiers behind a single API avoids that: you prototype on serverless and repoint the same client to a dedicated endpoint once volume justifies it, often by changing a model or endpoint parameter rather than an integration.

How the DigitalOcean Inference Engine unifies serverless, dedicated, and batch inference

The DigitalOcean Inference Engine runs three inference patterns over the same GPU Droplets infrastructure, behind a single endpoint. These solutions include real-time Serverless inference across 70+ open-weight and frontier models, Dedicated inference on reserved, single-tenant GPU capacity with support for bringing your own model, and asynchronous Batch Inference for large, non-real-time jobs. The Preview Inference Router adds cost- and latency-based routing on top, and a shared Model Playground and evaluations tooling sit on the same account.
That's what makes DigitalOcean a solid solution for AI-native companies requiring both serverless and dedicated GPU tiers. The unification isn't limited to the inference API itself. It extends to the GPU Droplets the dedicated tier runs on, as well as the rest of the AI-Native Cloud around them. A workload doesn't need to switch vendors, contracts, or billing relationships as it matures from prototype to production.

What to check before assuming "one API" elsewhere means the same thing

Several other inference-focused platforms let serverless and dedicated calls share one API too, usually as a way past per-token rate limits or shared-capacity contention once traffic justifies reserving GPUs. But a shared API can sound more unified on a landing page than it is in production. So it's worth confirming directly against each provider's current docs: does provisioning real dedicated capacity happen through the API, or does it route through a sales conversation first? Is the dedicated tier's hardware genuinely part of the same cloud a team already runs on, or does the provider specialize in inference alone, with GPUs sitting apart from the application's data and storage? And does "one API" cover just the inference call—or the account, billing, and observability around it too?

What else to verify before standardizing on a provider

Beyond scope, it's worth confirming: whether the model you need is available in both tiers (not every model ships to both), what the cutover mechanism actually looks like (a new endpoint URL versus a full redeploy), how billing reconciles if usage straddles both tiers in the same month, and whether dedicated capacity is truly single-tenant or a reserved slice of shared hardware. For teams already running other infrastructure on a given cloud, it's also worth weighing how much value comes from the inference API alone versus not having to move data, agents, or storage to a second vendor to get dedicated GPU access.

References & further reading

DigitalOcean's Inference Engine brings serverless, dedicated, and batch inference together in one place.
Our Dedicated Inference documentation covers bring-your-own-model (BYOM) support and scaling settings for teams ready to move off serverless.
Read our Serverless Inference documentation for the model catalog and routing options available on the shared endpoint.
Reading through each inference provider's own docs tends to be the fastest way to confirm whether "serverless and dedicated" really means one API or two separate products.

What It Actually Costs to Serve a 1M-Token Model in Production

DigitalOcean — Mon, 21 Sep 2026 20:33:36 +0000

Model providers now advertise context windows large enough to hold a codebase or a stack of contracts in a single request. It's tempting to read that number as a green light, but it isn't. A model that accepts a million tokens and a system that can serve a million tokens to real, concurrent users within your latency and cost budget are two different engineering problems. Conflating them is how teams end up with a demo that works and a bill or a p99 latency chart that doesn't.

You serve a 1M-token model in production by treating context length as a cost and latency variable you manage on purpose, not a spec you flip on. Let’s dive into the memory math, the latency cliff, and the cost curve for serving a model with a million tokens of context, plus the caching and routing techniques that production teams use today to keep all three in check. Plus, a straight answer on when long context is the wrong tool for the job and retrieval should do the work instead.

Key takeaways:

Serving a 1M-token context model in production is a memory, latency, and cost problem layered on top of a model capability. Supporting long context and serving it reliably are not the same claim.
Serving long context effectively means stable agent sessions and predictable spend, rather than retries, timeouts, and a bill that scales with the window instead of the question.
The real decision isn't whether to "turn on" long context, it's which combination of caching, prefix reuse, and retrieval fits your workload's cost and latency budget.
Production-grade 1M-token serving combines general techniques, including KV cache economics and prefix-reuse approaches like SGLang's RadixAttention. Use DigitalOcean Inference Router for cache-aware routing, plus retrieval-augmented generation (RAG) as the alternative and complement.

What "serving" a long-context model actually means

Context is everything you send to a model in a single request, measured in tokens: the prompt, the documents, the conversation history. A 128K-token window is roughly a novel's worth of text, and frontier closed models now advertise up to 1M tokens (roughly, ten novels). That number describes what the model can read in isolation, but not what happens to time-to-first-token, batch size, or your GPU bill once dozens of those requests land on the same cluster at once.

Support is a model capability, and performance is a serving question. The two come apart the moment real traffic shows up.

Two mechanics decide whether a long-context request is slow to start or slow to finish:

Prefill: The compute-bound pass where the model reads your entire input before writing anything.
KV cache: The memory that stores each token's key and value vectors so the model doesn't reread the input on every step.

Dig in to how prefill/decode disaggregation works: the two-phase split behind every long-context serving decision below.

Why its worth optimizing for long-context

Getting long-context serving right pays off in multiple ways.

Here's what's actually at stake once you're running this in production:

Stable agent and document workloads: A correctly cached long-context session makes it possible for a coding agent or a multi-turn document review to hold state across dozens of turns without resending the same 100K-token context on every message.
Costs you can forecast: Once you know whether traffic is cache-friendly or retrieval-shaped, it's easier to accurately predict spend.
No silent accuracy loss: Long-context serving done well accounts for the fact that models answer less reliably as input grows. This happens by routing precision lookups to retrieval instead of trusting long-context recall, and by testing accuracy at the context lengths you'll actually run rather than assuming a bigger model or window is automatically more reliable.
Latency you can put in an SLA: Understanding the prefill/decode split lets you set a time-to-first-token target you can actually hit, instead of promising an interactive response time a long-context request can't deliver.
Room to scale without a rebuild: Architectures built around caching and routing from the start absorb growth in context length and concurrent users without the rewrite that a "just resend everything" approach eventually forces.

What to evaluate before you commit to a long-context approach

Long-context serving isn't a single decision, but rather a set of infrastructure and architecture choices considered together:

Cache-awareness: Check whether the serving layer reuses the KV cache across requests that share a common prefix, rather than recomputing it each time. The DigitalOcean Inference Router, for example, pins a session to the same model so that a warm cache isn't discarded mid-task when switching to a different, cheaper model.
Batching under long-context load: Confirm the system maintains healthy batch sizes as the average context length grows, using techniques like chunked prefill and paged memory allocation, rather than letting a single long request starve everything queued behind it.
Isolation from noisy neighbors: Ask whether long-context and interactive traffic run in separate pools. Dedicated Inference on DigitalOcean exists specifically so a large document-processing job doesn't blow up latency for interactive users on the same infrastructure.
Pricing that reflects the real cost curve: Look for tiered, per-token pricing that acknowledges long-context requests cost more proportionally, plus a prompt-caching rate that's meaningfully cheaper than a full rewrite.
Observability into cache health, not just request counts: Make sure you can see KV-cache occupancy, eviction, and preemption counts—not just average latency—since that's what actually explains a p99 problem caused by long-context traffic.

Resend, cache, retrieve, or both: what each costs per session

The cleanest way to understand the cost of each approach is to price a full session rather than a single token.

These examples take a 10-turn agent conversation carrying a stable 128K-token context—a codebase or a document set—where each turn adds a short question and gets a short answer:

Strategy	Use Case	Mechanism	Cost per 10-turn session (128K context)
Resend the full context every turn	Short, one-off sessions where simplicity matters more than cost	Full context re-sent as input tokens on every turn	$3.93
Cache the context; reuse it	Stable, reused context — a codebase, a document set, a long agent session	Prefill cost paid once; later turns reuse the cached KV state	$0.95
Retrieve a relevant slice (RAG)	Large or changing corpora where each query touches a small fraction of the data	Only the relevant slice enters context; the rest of the corpus stays out	$0.33

Figures per DigitalOcean's long-context serving tradeoffs analysis, priced at Claude Sonnet's published rates ($3.00/M input tokens, $3.75/M to write the cache, $0.30/M to read it).

Caching cuts the bill roughly 4x against resending; retrieval cuts it roughly 12x. At thousands of sessions a day, that architecture choice matters more than any provider's per-token rate.

The four tradeoffs that decide whether long-context works in production

A 1M-token context window is a spec, not a guarantee that serving it will work. Memory, latency, throughput, and accuracy each break in their own way as context grows, and "supports 1M tokens" and "serves 1M tokens well" can be very different claims.

Memory: the KV cache doesn't scale the way people expect

For every token a model reads, it stores a key and a value vector so it doesn't have to reprocess the input on each new token. That cache grows in direct proportion to context length: the math works out to roughly 43GB for a single 128K-token request on a 70-billion-parameter model in BF16—close to the model's own memory footprint. Push to a million tokens, and the cache alone exceeds what a single high-end GPU holds, before the model's own weights are even loaded.

Latency: why time-to-first-token is the number that breaks

Attention compares every token against every other token, so compute grows with the square of input length: double the context, quadruple the work. That happens entirely during prefill, before a user sees a single word, which is why time-to-first-token ranges from milliseconds with short context to multiple seconds with long context. It also means one long request can hold a GPU for seconds, quietly degrading p95 and p99 latency for every short, unrelated request queued behind it.

Throughput: the bandwidth ceiling no amount of compute fixes

Generating each output token requires reading the entire KV cache from GPU memory, and the read speed is limited by hardware bandwidth. Double the context, double the cache, and tokens-per-second roughly halves, with a limit that has nothing to do with how much compute is sitting idle. A cluster can appear underutilized on a FLOPS dashboard and still serve long-context requests slowly because the bottleneck has shifted to the memory bus.

Accuracy: why a bigger window doesn't guarantee a better answer

Longer context doesn't reliably mean a better answer. A 2026 measurement study found that accuracy at long context lengths doesn't track model size — a smaller model sometimes outperformed a larger sibling — and that a wrong answer just becomes a retry, which turns an accuracy problem into a latency problem measured in total time-to-correct-answer. Production systems that route by expected accuracy at a given context length, not just by cost, cut that wasted retry time substantially.

How production teams are actually solving the long-context problem today

Successful teams aren't fighting the KV cache, they're making it reusable. SGLang's RadixAttention implements prefix caching alongside continuous batching, so a repeated prompt prefix (i.e. a system prompt, a shared reference document, a stable set of few-shot examples) gets computed once and reused across requests instead of recomputed on every call. That keeps throughput comparatively stable even as context length grows into the millions of tokens, which is exactly the workload where naive serving falls apart first.

You can run this pattern yourself on DigitalOcean GPU Droplets, or skip the setup work with the SGLang 1-Click Marketplace app, which handles the underlying infrastructure for you.

DigitalOcean's Inference Router adds cache-aware routing on top of that. It keeps a session pinned to the same model, so a warm KV cache doesn't get discarded mid-task by a switch to a different model chosen for cost reasons.

If you're self-hosting long-context serving on GPU Droplets rather than going through a managed router, three additional levers matter:

Chunked prefill, which breaks a large prefill pass into pieces so it doesn't stall the rest of the batch.
Tuning max batch size and max batched-token limits for your actual mix of context lengths.
Keeping long-context traffic in a separate pool from interactive traffic so one doesn't degrade the other.

When time-to-first-token degrades as context grows, DigitalOcean's inference-debugging guidance points to KV-cache pressure—not the scheduler—as the usual culprit. This is why watching KV-cache occupancy, eviction, and preemption counts diagnoses the problem faster than staring at a scheduler queue.

Do you even need long context? Choosing between caching, retrieval, or using both

If a model accepts a million tokens, it's fair to ask why you'd also use a vector database, a chunking pipeline, and a retrieval step. The answer is that the price of skipping retrieval includes the quadratic prefill cost, the multi-second time-to-first-token, and the tiered per-token pricing. They’re all paid on every request—even when the answer only needed to use three paragraphs of the input.

As a practical rule, long context plus caching wins when your working set is stable and reused. For example, many requests against the same document set, or a day-long agent session where the context is the state. RAG wins when your corpus is larger than any context window, changes frequently, or is mostly irrelevant to any single query. Precision lookups favor RAG even when the full corpus would technically fit, since retrieval keeps the model working at the shorter lengths where it's most reliable. Most production systems end up using both: retrieval to decide what enters the context, a long window to hold a generous amount of it, and caching wherever it repeats.

Long-context LLM serving FAQ

How do you serve long-context (1M token) models in production?

Serving a 1M-token model in production means treating context length as a cost and latency variable you manage on purpose, not a spec you flip on, since supporting long context and serving it reliably are not the same claim. It requires managing KV cache memory, the quadratic prefill cost, and the resulting time-to-first-token, then choosing the right mix of caching, prefix reuse, and retrieval for your workload. The DigitalOcean Inference Router applies cache-aware routing to this problem by pinning a session to the same model so a warm KV cache isn't discarded mid-task.

What is disaggregated prefill and decode in LLM inference?

Prefill is the compute-bound pass where the model reads your entire input before writing anything, while decode is the token-by-token generation that follows. Attention compares every token against every other token during prefill, so compute grows with the square of input length, meaning one long request can hold a GPU for seconds and degrade latency for every short request queued behind it. Chunked prefill breaks a large prefill pass into pieces so it doesn't stall the rest of the batch.

What metrics matter for LLM inference observability (TTFT, TPOT, ITL)?

Time-to-first-token is the number that breaks under long context, ranging from milliseconds with short context to multiple seconds with long context, because it's entirely determined by the prefill pass. Beyond latency averages, observability into KV-cache occupancy, eviction, and preemption counts is what actually explains a p99 problem caused by long-context traffic, not request counts alone. KV-cache pressure, not the scheduler, is the usual culprit when time-to-first-token degrades as context grows.

Which inference provider has the best caching to reduce token costs for long autonomous coding sessions?

Caching the context and reusing it lets a coding agent hold state across dozens of turns without resending the same 100K-token context every message. SGLang's RadixAttention implements this as prefix caching alongside continuous batching, computing a repeated prompt prefix once and reusing it across requests. DigitalOcean supports this pattern on GPU Droplets or through the SGLang 1-Click Marketplace app, and the Inference Router adds cache-aware routing so a warm cache isn't discarded by a mid-task model switch.

How do you model cost-per-token for LLM inference?

The cleanest way to model cost is pricing a full session rather than a single token: resending the full context every turn costs $3.93 for a 10-turn, 128K-context session, caching and reusing that context brings it to $0.95, and retrieving only the relevant slice (RAG) brings it to $0.33. Caching cuts the bill roughly 4x against resending, while retrieval cuts it roughly 12x, a difference that matters more at scale than any provider's per-token rate. DigitalOcean prices around this same curve, with tiered per-token rates that account for long-context requests costing more than proportionally, plus a prompt-caching rate meaningfully cheaper than a full rewrite.

Serve long-context models without managing the cache yourself

The DigitalOcean Inference Engine prices per token by model, with prompt-caching rates listed right next to the standard rate. That means you can work out what a long-context session costs before you run it, not after the invoice shows up. The Inference Router automatically applies cache-aware routing, pinning a session to the model whose KV cache is already warm rather than discarding it during a mid-task switch.

That gets you:

Automatic model-to-request matching, so cost- or latency-sensitive requests route to the right model without the need for writing the routing logic yourself.
Dedicated Inference to isolate long-context, latency-sensitive workloads from noisy-neighbor traffic.
Serverless-to-dedicated scaling with no migration step, so a workload that outgrows shared capacity doesn't require re-architecting.

Get started with DigitalOcean's Inference Engine →

Hippocratic AI cut prefill latency 2x on long-context clinical sessions running production inference on DigitalOcean's cache-aware routing.

https://www.youtube.com/watch?v=WeAAISnnNVA

These customer-reported results reflect Hippocratic AI’s methodology, workload, configuration, and measurement period and may not be representative of results in other environments. Results in customer environments may vary depending on configuration, implementation, and usage. Results and/or savings are not guaranteed.

Any references to third-party companies, trademarks, or logos in this document are for informational purposes only and do not imply any affiliation with, sponsorship by, or endorsement of those third parties.

AI token billing continues to cause sticker shock

DigitalOcean — Fri, 18 Sep 2026 16:25:40 +0000

Tech talk: are there inference providers that are transparent about token usage?

A top pain point for companies using AI is the discrepancy between expected and actual invoice rates. Particularly when it comes to tokens. The increase in AI usage and tokenmaxxing has created a paradox where a provider knows exactly how many tokens it generated on your behalf, yet you find out only after the fact. It's become a real cost problem when using reasoning models. Internal step-by-step reasoning counts as billed output, but it's invisible by design. The providers willing to open up their token accounting now are betting that transparency will become table stakes before their competitors catch up.

The 2025 study, “Predictive Auditing of Hidden Tokens in LLM APIs via Reasoning Length Estimation,” estimated that hidden reasoning tokens can account for over 90% of a model's total token spend on complex tasks. A short, simple-looking answer can hide 10,000+ reasoning tokens, but the only confirmation of their use is the bill.

This trend in billing surprises has rightfully drawn industry attention. Especially with token usage projected to grow 24x to 120 quadrillion tokens per month by 2030, according to research from Goldman Sachs. Developers have started noticing the gap between what usage dashboards imply and what invoices actually charge, and the industry's response is arriving on two tracks.

First, standards bodies are getting involved. The Linux Foundation's proposed Tokenomics Foundation wants open, vendor-neutral measurement standards so token accounting isn't just whatever a provider's internal dashboard says it is.

Second, providers like Groq and Cerebras are sharing flat per-token pricing. And the newly launched DigitalOcean Inference Router helps your team route tasks to the most cost-effective or optimal model, providing budget control based on your priorities for specific AI tasks.

Thanks to rapidly scaling AI adoption, developers know these costs aren’t a simple line item to ignore. When evaluating inference options, knowing exactly how your tokens are used and what that can mean for overall billing is essential.

Event calendar

Open Intelligence Summit by DigitalOcean

The argument is no longer “Is open source AI cheaper?” It’s now “Who owns intelligence?” A movement is forming around open intelligence and generating the belief that the intelligence powering our AI products and companies should be more open, portable, and controllable.

Join DigitalOcean, RadixArk, Inferact, and more for deep dives, demos, and discussions about the technologies that keep intelligence portable, interoperable, and in the hands of builders. The two-day event runs at The Midway in San Francisco from October 12-13th.

Industry news

Coding agents are earning real trust in production codebases

A study by researchers at Polytechnique Montréal, led by Canada Research Chair Foutse Khomh, analyzed 220,612 PRs across 539 Python repos and found Claude Code's PRs merge 84.3% of the time—well ahead of Codex (73.5%), Cursor (63.9%), Copilot (59.6%), and Devin (43.0%)—with bug rates comparable to or lower than human-written code.

OpenAI launches GPT-6 Astra

OpenAI introduced GPT-6 Astra, a new frontier model focused on advanced reasoning, computer use, coding, scientific research, and professional workflows. The model achieves major gains across benchmarks, including 99.9% on ARC-AGI-3 and 100% on ExploitBench, while also improving task efficiency and alignment.

Anthropic introduces Claude Fable 5.1 and Mythos 5.1

The new models target advanced coding, knowledge work, and scientific research, with Fable 5.1 offering stronger performance at lower cost and Mythos 5.1 providing more permissive safeguards for vetted cybersecurity and life-science users.

Grok Bot comes to Android

Cursor has launched the Grok Bot Android app, letting users assign tasks to AI Bots from their phones, continue conversations across devices, and monitor multiple Bots running in parallel on cloud computers. The app is currently in beta for eligible Cursor and SuperGrok plans.

Community tutorials

Why Your Best Model Is Two Models: Routing Between Kimi K3 and Claude

Hardcoded routing logic gets messy fast and using a small model like Haiku to classify requests means paying for two calls on every one. DigitalOcean built Plano-Orchestrator, a routing model that scored 87.84% accuracy against GPT-5.1's 86.93% and Claude Sonnet 4.5's 86.11%, deciding where a request goes in about 200 milliseconds. Walk through building a router that pools Kimi K3 and Claude, and see how to check with live traffic data whether it's actually working.

Resilient GPU Compute on DigitalOcean Kubernetes: Surviving Spot Interruptions

When DOKS reclaims Spot GPU capacity, it deletes the entire node pool at once, not one node at a time so the usual "autoscaler quietly relaunches it" assumption breaks completely, and nothing automatically brings a reclaimed pool back. Walk through building a pre-provisioned On-Demand fallback pool that costs nothing until a reclaim happens, and run the companion repo's automated interruption simulation to see the failover in action.

GLM-5.3-Flash is the cheapest model on DigitalOcean. It's also the most verbose.

GLM-5.3-Flash lists at one twelfth of Qwen3.8-Max's output price, but on unconfigured defaults, it burned 625 output tokens answering what port SSH uses because it silently defaults to maximum reasoning effort, cutting its real-world price advantage down to just 2.3x. See the measured cost breakdown across 2,700 API calls, and get the one-line reasoning_effort fix that made Flash 6.3x cheaper on the same prompts.

When pgvector Starts to Slow Down as Your Vector Table Grows

A pgvector's query latency doesn't creep up gradually as a table grows. It stays flat for a long time, then jumps hard the moment the HNSW graph outgrows available RAM and Postgres starts reading it from disk instead. Get the six-step method for finding your own breaking point, measuring recall, latency, and index build time at the table sizes you'll actually reach.

Does Context Length Affect Inference Cost Linearly? We Measured Why It Doesn't

Cost calculators price context length linearly, but measured throughput on a single H200 serving Ministral 3 14B fell from 19,089.8 tokens/sec at 2K context to 4,967.2 tokens/sec at 256K: a 3.84x collapse driven by concurrent request capacity dropping from 311 to just 2 as the fixed KV cache pool fills up. See the full cost curve and break-even utilization math showing exactly where a dedicated GPU stops being cheaper than a flat per-token serverless rate.

Product updates

DigitalOcean Kubernetes (DOKS) now supports Spot GPU Droplets

Running Spot GPUs on DOKS automates interruption handling with automatic cordoning, draining, and PodDisruptionBudget enforcement. Cluster autoscaling dynamically scales your GPU capacity based on workload demand. Kubernetes capabilities such as labels, taints, and node affinity allow you to configure on-demand node pools as a fallback when Spot capacity is reclaimed. Plus, reclaim events surface as native Kubernetes events. Try it today

April 2026 DigitalOcean Tutorials: Inference Optimization and AI Infrastructure

DigitalOcean — Fri, 22 May 2026 21:55:02 +0000

Most AI teams hit the same walls once they move past prototyping. The RAG pipeline that worked flawlessly in a demo starts hallucinating under real traffic. Inference costs climb without clear optimization levers. GPU resources sit underutilized while workloads spike elsewhere.

Most of the time, the root cause traces back to architecture decisions that weren't pressure-tested for production. This month's DigitalOcean tutorials focus on diagnosing and fixing those failure points across the AI infrastructure stack.

Why RAG Systems Fail in Production

Why do seemingly solid RAG demos collapse under real-world conditions? This article traces failures back to retrieval quality, latency tradeoffs, and embedding drift. You’ll get a clear picture of how upstream decisions—such as chunking strategy and ranking—directly affect downstream LLM outputs. If your team is building production pipelines, evaluation, monitoring, and retrieval engineering matter just as much as model choice.

Dedicated vs. Serverless Inference as You Scale

The choice between serverless and dedicated inference isn't a one-time decision but an evolution driven by how your workload changes over time. Early on, serverless makes sense because traffic is unpredictable and iteration speed matters more than performance optimization. As usage stabilizes, the cracks show up—latency variability frustrates users and per-request pricing gets expensive for always-on systems. Walk-throughs of Modal and Together.ai show where that transition point hits and why delaying it costs you.

Fine-Tuned LLMs on Serverless Architecture

Parameter-efficient methods like LoRA let platforms serve hundreds of fine-tuned model variants from a single GPU by layering small adapter weights on top of a shared frozen base model. This makes serverless, pay-per-token inference possible for custom models without dedicated GPU deployments. The tradeoff is cold starts: idle adapters get evicted from VRAM and need to be reloaded, adding a few hundred milliseconds of latency to the first token. You’ll learn how to minimize that with keep-alive requests, adapter rank tuning, and smarter layer targeting.

The Silent Versioning Problem in AI Inference

This one is a cautionary tale about what happens when the model behind your endpoint changes and nobody tells you. The serving stack is full of moving parts that can shift independently of the model name, and the result is silent regressions that break prompt tuning and invalidate your evaluations before you even know something moved. It includes a practical buyer's checklist for pressing inference platforms on snapshot pinning, retention commitments, and how they handle disclosure when something in the stack changes.

The Hidden Bottlenecks in LLM Inference and How to Fix Them

Faster GPUs are not the answer if the rest of your serving stack can't keep up. Spoiler: the bottlenecks are GPU underutilization from rigid batching, memory bandwidth constraints during decode, KV cache fragmentation, and CPU-side overhead from tokenization and prompt assembly. Click through for a deeper look at each one and practical fixes.

We Built a Private-Document AI App to Test Platform Security. Here Is What We Could Actually Verify

AI security should always be treated as a first-class concern, not an afterthought. This tutorial puts that to the test by building a private-document chatbot and running the same workflow across six inference platforms: DigitalOcean, Baseten, Nebius, Fireworks AI, Modal, and Together AI. Each platform is evaluated on access controls, data retention defaults, network isolation, audit logging, and shared responsibility clarity. It doubles as a practical framework for figuring out what you can actually verify before sensitive data is in flight.

Post-Inference Storage and Querying with MongoDB

Many inference tutorials stop at the model response. This one keeps going. You'll build a FastAPI app that sends images through a vision model, stores the structured predictions in MongoDB, and then exposes endpoints that let you filter by detected labels and confidence scores or run aggregation pipelines across your full dataset. It's a practical blueprint for turning raw model output into something queryable and operational.

How to Build a Multi-Agent AI System with Docker and DigitalOcean

Instead of routing everything through a single model, multi-agent systems let you split a workflow across specialized agents that each handle a different part of the problem and pass results between them. The tradeoff is coordination complexity. This walkthrough covers how to containerize each agent with Docker, manage communication between them, and deploy the full system on DigitalOcean. You'll come away with a working deployment pattern you can adapt to your own orchestration needs.

Building an AI-Powered GPU Fleet Optimizer with the DigitalOcean AI Platform ADK

A single idle GPU Droplet left running overnight can add hundreds of dollars to your monthly bill, and standard CPU monitoring won't catch it because it can't see whether the GPU is actually doing work. This tutorial builds an AI-powered agent using the DigitalOcean AI Platform ADK that scrapes NVIDIA DCGM metrics like VRAM usage, engine utilization, and power draw across your fleet in real time. It compares those metrics against configurable thresholds to flag idle resources before they inflate your cloud spend. The repo is designed to be forked and customized to your own workloads, including adding tools that let the agent take action like powering off idle nodes.

Python Decorators: From Basics to Real-World Use Cases

DigitalOcean — Tue, 12 May 2026 21:04:07 +0000

This article was originally written by Shaoni Mukherjee (AI Technical Writer)

Key takeaways

Python decorators allow additional functionality to be added to functions without changing the original function code.
Decorators help reduce repeated code and improve code reusability.
The @decorator_name syntax is a cleaner way of wrapping functions.
Decorators are commonly used for logging, authentication, caching, validation, and performance monitoring.
*args and **kwargs make decorators flexible enough to work with different function arguments.
functools.wraps helps preserve the original function metadata and should be considered a best practice.
Multiple decorators can be chained together to add multiple layers of functionality.
Frameworks like Flask and Django rely heavily on decorators for routing, authentication, and request handling.
Decorators should be kept simple and focused to maintain readability and easier debugging.
Understanding decorators is important for writing cleaner and more maintainable Python applications.

Introduction

While building real-world Python applications, a common challenge is the repetition of certain logic codes, such as logging, authentication, validation, time, or performance monitoring across multiple functions. For instance, API endpoints often require user authentication checks, and performance-critical functions may need execution time tracking.

Adding the same logic code within each function often leads to cluttered code, reduced readability, and increased maintenance effort. Decorators address this problem by creating the separation of such cross-cutting concerns into reusable components that can be applied to functions in a clean and consistent manner. In frameworks like Flask, the @app.route("/") decorator links a URL to a function without requiring explicit routing logic, while in Django, decorators such as @login_required enforce access control by restricting views to authenticated users. This approach promotes modularity, improves code clarity, and simplifies the overall structure of applications.

What are Python decorators?

Decorators are basically a wrapper around a function to modify it for better use. The function remains the same, but the decorator adds an extra something to the function.

The core idea

Say you have a simple function:

def greet():
    print("Hello, world!")

Now imagine you want to print a line before and after every function you write, without modifying each one. A decorator lets you do exactly that:

def my_decorator(func):
    def wrapper():
        print("--- Before ---")
        func()           # calls the original function
        print("--- After ---")
    return wrapper

@my_decorator
def greet():
    print("Hello, world!")

greet()

Output:

--- Before ---
Hello, world!
--- After ---

The @my_decorator line is just shorthand for greet = my_decorator(greet). Python replaces your function with the wrapped version automatically.
To understand the concept better, let us take a real-world example of timing a function:

import time

def timer(func):
    def wrapper(*args, **kwargs):        # *args lets it work with ANY function
        start = time.time()
        result = func(*args, **kwargs)
        end = time.time()
        print(f"{func.__name__} took {end - start:.4f} seconds")
        return result
    return wrapper

@timer
def slow_task():
    time.sleep(1)
    print("Task done!")

slow_task()

Output:

Task done!
slow_task took 1.0012 seconds

Why decorators matter (especially in real projects)

They're everywhere in Python. Common use cases include:

@staticmethod / @classmethod — built into Python for class methods
@app.route('/home') — Flask/Django use them to define web routes
@login_required — Django uses this to protect pages behind authentication
Logging, caching, retrying failed requests — all cleanly handled with decorators

A decorator takes a function, adds behavior around it, and returns a new function without touching the original code.

How decorators work internally

To understand decorators better, we will first need to understand a few core Python concepts:

Foundation: Functions are objects in Python

In Python, functions aren't special, but they're just objects like integers or strings.

def say_hello():
    print("Hello!")

# Pass a function as an argument
def run_it(func):
    func()

run_it(say_hello)   # prints: Hello!

# Assign a function to a variable
my_func = say_hello
my_func()           # prints: Hello!

# Return a function from another function
def get_greeter():
    def say_hi():
        print("Hi!")
    return say_hi   # returning the function, not calling it

greeter = get_greeter()
greeter()           # prints: Hi!

This is the entire foundation that decorators are built on.

Why are decorators needed?

Imagine there are many functions in a project, and each function needs logging.

Without decorators:

def add(a, b):
    print("Function started")
    result = a + b
    print("Function ended")
    return result

def multiply(a, b):
    print("Function started")
    result = a * b
    print("Function ended")
    return result

Problem:

Repeated code
Hard to maintain in large projects
If logging changes, every function must be updated

Decorators solve this problem by reusing common functionality.

With decorators:

Using decorators, the repeated code ("Function started" and "Function ended") can be moved into a single reusable decorator.
Instead of writing the same lines inside every function, the decorator handles it automatically.

Step 1: Create the decorator

def log_function(func):

    def wrapper(a, b):
        print("Function started")

        result = func(a, b)

        print("Function ended")

        return result

    return wrapper

Step 2: Apply the Decorator

@log_function
def add(a, b):
    return a + b


@log_function
def multiply(a, b):
    return a * b

Calling the Functions

print(add(2, 3))
print(multiply(4, 5))

Output:

Function started
Function ended
5

Function started
Function ended
20

What changed?

The functions now only contain their main logic:

return a + b

and

return a * b

The extra behavior (logging) is handled by the decorator separately.

Visual understanding

When this runs:

add(2, 3)

Python internally does this:

add = log_function(add)

So the actual flow becomes:

wrapper()
    ├── print("Function started")
    ├── call original add()
    ├── print("Function ended")
    └── return result

Better Version Using *args and **kwargs

The previous decorator only works for functions with two arguments.
A more reusable decorator looks like this:

def log_function(func):

    def wrapper(*args, **kwargs):
        print("Function started")

        result = func(*args, **kwargs)

        print("Function ended")

        return result

    return wrapper

Now it works with:

any number of arguments
positional arguments
keyword arguments

Why this is powerful

Imagine 100 functions needing logging.

Without decorators:

repeated code everywhere

With decorators:

write logging once
reuse everywhere

This is one of the biggest reasons decorators are widely used in real-world Python projects and frameworks like:

Common practical examples of Python decorators

A few of the most common practical examples are listed here, from solo projects to production systems.

1. Timing and performance measurement

Useful when profiling slow functions or benchmarking code.

import time
from functools import wraps

def timer(func):
    @wraps(func)
    def wrapper(*args, **kwargs):
        start = time.perf_counter()
        result = func(*args, **kwargs)
        end = time.perf_counter()
        print(f"{func.__name__} ran in {end - start:.4f}s")
        return result
    return wrapper

@timer
def process_data(n):
    total = sum(range(n))
    return total

process_data(1_000_000)
# process_data ran in 0.0312s

perf_counter() is preferred over time.time() for short measurements, and it's higher resolution and is not affected by system clock adjustments.

2. Logging

Instead of adding print statements everywhere, a logging decorator handles it in one place.

import logging
from functools import wraps

logging.basicConfig(level=logging.INFO)

def log_calls(func):
    @wraps(func)
    def wrapper(*args, **kwargs):
        logging.info(f"Calling {func.__name__} | args={args} kwargs={kwargs}")
        result = func(*args, **kwargs)
        logging.info(f"{func.__name__} returned {result}")
        return result
    return wrapper

@log_calls
def multiply(a, b):
    return a * b

multiply(4, 5)
# INFO: Calling multiply | args=(4, 5) kwargs={}
# INFO: multiply returned 20

In production, you'd swap logging.info for a structured logger like structlog or a cloud logging sink.

3. Retry on failure

Critical for network calls, API requests, or anything that can fail transiently.

import time
from functools import wraps

def retry(times=3, delay=1):
    def decorator(func):
        @wraps(func)
        def wrapper(*args, **kwargs):
            for attempt in range(1, times + 1):
                try:
                    return func(*args, **kwargs)
                except Exception as e:
                    print(f"Attempt {attempt} failed: {e}")
                    if attempt < times:
                        time.sleep(delay)
            raise Exception(f"{func.__name__} failed after {times} attempts")
        return wrapper
    return decorator

@retry(times=3, delay=2)
def fetch_data(url):
    import requests
    response = requests.get(url, timeout=5)
    response.raise_for_status()
    return response.json()

fetch_data("https://api.example.com/data")
# Attempt 1 failed: Connection timeout
# Attempt 2 failed: Connection timeout
# Attempt 3 failed: Connection timeout
# Exception: fetch_data failed after 3 attempts

Notice this is a decorator factory — retry(times=3) returns the actual decorator. This is how you pass arguments to decorators.

4. Caching memoization

Avoids recomputing expensive results by storing previous outputs.

from functools import wraps

def memoize(func):
    cache = {}
    @wraps(func)
    def wrapper(*args):
        if args not in cache:
            cache[args] = func(*args)
            print(f"Cache miss — computing for {args}")
        else:
            print(f"Cache hit for {args}")
        return cache[args]
    return wrapper

@memoize
def fibonacci(n):
    if n <= 1:
        return n
    return fibonacci(n - 1) + fibonacci(n - 2)

fibonacci(6)
# Cache miss — computing for (6,)
# Cache miss — computing for (5,)
# ...
fibonacci(6)
# Cache hit for (6,)   ← instantly returns stored result

Python actually ships a production-grade version of this built in:

from functools import lru_cache

@lru_cache(maxsize=128)
def fibonacci(n):
    if n <= 1:
        return n
    return fibonacci(n - 1) + fibonacci(n - 2)

lru_cache (Least Recently Used) is thread-safe and evicts old entries when the cache is full — use it over a hand-rolled version in real projects.

5. Access control authorization

A staple in web frameworks like Flask and Django.

from functools import wraps

def require_role(role):
    def decorator(func):
        @wraps(func)
        def wrapper(user, *args, **kwargs):
            if user.get("role") != role:
                raise PermissionError(f"Access denied. Required role: {role}")
            return func(user, *args, **kwargs)
        return wrapper
    return decorator

@require_role("admin")
def delete_user(user, user_id):
    print(f"Deleting user {user_id}")

admin = {"name": "Shaoni", "role": "admin"}
guest = {"name": "Guest", "role": "viewer"}

delete_user(admin, 42)    # Deleting user 42
delete_user(guest, 42)    # PermissionError: Access denied. Required role: admin

Django's @login_required and @permission_required follow this exact pattern internally.

6. Input validation

Validate arguments before they even reach your function's logic.

from functools import wraps

def validate_positive(*arg_positions):
    def decorator(func):
        @wraps(func)
        def wrapper(*args, **kwargs):
            for i in arg_positions:
                if args[i] <= 0:
                    raise ValueError(
                        f"Argument at position {i} must be positive, got {args[i]}"
                    )
            return func(*args, **kwargs)
        return wrapper
    return decorator

@validate_positive(0, 1)
def calculate_area(width, height):
    return width * height

calculate_area(5, 10)    # 50
calculate_area(-3, 10)   # ValueError: Argument at position 0 must be positive

7. Rate Limiting

Preventing a function from being called too frequently is very common in API clients.

import time
from functools import wraps

def rate_limit(calls_per_second=1):
    min_interval = 1.0 / calls_per_second
    last_called = [0.0]   # mutable container to hold state in closure

    def decorator(func):
        @wraps(func)
        def wrapper(*args, **kwargs):
            elapsed = time.time() - last_called[0]
            wait = min_interval - elapsed
            if wait > 0:
                print(f"Rate limit: waiting {wait:.2f}s")
                time.sleep(wait)
            last_called[0] = time.time()
            return func(*args, **kwargs)
        return wrapper
    return decorator

@rate_limit(calls_per_second=2)
def call_api(endpoint):
    print(f"Calling {endpoint}")

call_api("/users")
call_api("/posts")    # Rate limit: waiting 0.49s
call_api("/comments") # Rate limit: waiting 0.49s

Quick reference

Decorator	Use Case	Real-world Equivalent
`@timer`	Measure execution time	Profiling, benchmarking
`@log_calls`	Audit function calls	Observability, debugging
`@retry`	Handle transient failures	API clients, DB connections
`@lru_cache`	Cache expensive results	ML inference, DB queries
`@require_role`	Guard endpoints by role	Django, Flask auth
`@validate_positive`	Sanitize inputs early	Data pipelines, APIs
`@rate_limit`	Throttle call frequency	External API clients

Real-world use cases in frameworks

Decorators are heavily used in modern Python frameworks because they provide a clean and reusable way to add functionality to applications without modifying the core business logic.
Frameworks such as Flask and Django use decorators for:

Routing
Authentication
Authorization
Caching
Request validation
Restricting HTTP methods
Logging

These decorators make applications cleaner, easier to maintain, and more readable.

Flask routing decorator

One of the most common examples of decorators appears in Flask routing.
Using Flask:

from flask import Flask

app = Flask(__name__)

@app.route("/")
def home():
   return "Homepage"

Here:

@app.route("/")

is a decorator.
It tells Flask:
“When a user visits /, execute the home() function.”

Flask authentication decorator

Decorators are also commonly used for authentication.
Example:

@app.route("/dashboard")
@login_required
def dashboard():
   return "Dashboard"

Here:

@login_required

checks whether the user is logged in before allowing access to the dashboard.

Why this is useful

Without decorators, authentication checks would need to be repeated inside every protected function.
Example without decorator:

def dashboard():
   if not logged_in:
       return "Please log in"
   return "Dashboard"

Using decorators:

avoids repeated code
keeps route definitions clean
centralizes authentication logic

This becomes extremely useful in large applications with many protected routes.

Django authentication decorator

Django also uses decorators extensively.
Example:

from django.contrib.auth.decorators import login_required
@login_required
def dashboard(request):
   return HttpResponse("Welcome")

The @login_required decorator ensures:

only authenticated users can access the view
unauthorized users are redirected to the login page

Benefits

Reusable security checks
Cleaner view functions
Better maintainability
Centralized authentication handling

Django HTTP method restriction

Django provides decorators to restrict HTTP request methods.

Example:

from django.views.decorators.http import require_POST
@require_POST
def submit(request):
   return HttpResponse("Submitted")

The decorator:

@require_POST

ensures the function only accepts POST requests.
If a GET request is sent, Django automatically returns an error.

Why this matters

This helps:

enforce API rules
improve security
prevent invalid request types
simplify validation logic

Without decorators, manual checks would be needed inside every function.

Django caching decorator

Decorators are also used for performance optimization.

Example:

from django.views.decorators.cache import cache_page
@cache_page(60)
def my_view(request):
   return HttpResponse("Cached")

Here:

@cache_page(60)

stores the response for 60 seconds.
If another user requests the same page during that time:

Django serves the cached version
the function does not run again

Advanced decorator concepts

Once the basic concepts are understood, the next step is to learn how decorators are implemented in production-grade Python applications. Advanced decorator patterns solve practical problems such as preserving function metadata, creating configurable decorators, and combining multiple decorators together.

These concepts are widely used in frameworks, libraries, and enterprise-level Python applications.

Preserving function metadata with `functools.wraps`

One common issue with decorators is that they replace the original function with the wrapper function. As a result, important metadata such as the function name, documentation string, annotations, and debugging information may be lost.

Consider the following decorator:

def decorator(func):

   def wrapper(*args, **kwargs):
       return func(*args, **kwargs)

   return wrapper

Using it:

@decorator
def greet():
   """This function greets the user"""
   print("Hello")

Now checking the function name:

print(greet.__name__)

Output:

wrapper

Instead of returning "greet", Python returns "wrapper" because the original metadata has been overridden by the wrapper function.
This creates problems for:

debugging
logging
API documentation
introspection
testing frameworks

To solve this problem, Python provides functools.wraps.

Using `functools.wraps`

from functools import wraps

def decorator(func):

   @wraps(func)
   def wrapper(*args, **kwargs):
       return func(*args, **kwargs)

   return wrapper

Using it again:

@decorator
def greet():
   """This function greets the user"""
   print("Hello")

Now:

print(greet.__name__)

Output:

greet

The @wraps(func) decorator copies the original function metadata into the wrapper function. This is considered a best practice when writing decorators in production applications.

Decorators with arguments

In many real-world scenarios, decorators need configuration values. This requires creating decorators that accept arguments.
A decorator with arguments introduces an additional level of nesting.
Example:

def repeat(n):

   def decorator(func):

       def wrapper(*args, **kwargs):

           for _ in range(n):
               func(*args, **kwargs)

       return wrapper

   return decorator

Using it:

@repeat(3)
def greet():
   print("Hello")

Calling:

greet()

Output:

Hello
Hello
Hello

Understanding the structure

This example contains three functions:

repeat()        → accepts decorator arguments
decorator()     → accepts the original function
wrapper()       → executes additional logic

The execution flow becomes:

greet = repeat(3)(greet)

This pattern is heavily used in:

retry mechanisms
caching systems
rate limiting
authorization frameworks
logging systems
timeout handling

For example, a retry decorator may accept the number of retries:

@retry(5)

A caching decorator may accept an expiration time:

@cache(expire=60)

Decorator arguments make decorators significantly more flexible and reusable.

Chaining Multiple Decorators

Python allows multiple decorators to be applied to the same function.

Example:

@decorator_one
@decorator_two
def func():
   pass

This is internally interpreted as:

func = decorator_one(decorator_two(func))

The execution order is important.

Python applies decorators from bottom to top:

decorator_two wraps the function first
decorator_one wraps the result next

Example of chained decorators

def decorator_one(func):

   def wrapper():
       print("Decorator One - Before")

       func()

       print("Decorator One - After")

   return wrapper


def decorator_two(func):

   def wrapper():
       print("Decorator Two - Before")

       func()

       print("Decorator Two - After")

   return wrapper

Applying both decorators:

@decorator_one
@decorator_two
def greet():
   print("Hello")

Calling:

greet()

Output:

Decorator One - Before
Decorator Two - Before
Hello
Decorator Two - After
Decorator One - After

Understanding the execution flow

The function call stack becomes:

decorator_one(
   decorator_two(
       greet
   )
)

This creates nested execution layers where each decorator adds behavior before and after the wrapped function. Decorator chaining is extensively used in frameworks. For example, a web route may simultaneously use:

authentication
caching
rate limiting
logging

Example:

@app.route("/dashboard")
@login_required
@cache_page(60)
def dashboard():
   return "Dashboard"

Each decorator contributes a separate layer of functionality while keeping the core business logic clean and isolated.

Conclusion

Python decorators provide a clean and powerful way to add extra functionality to functions without modifying the original code. They help reduce code duplication, improve reusability, and make applications easier to maintain.

From simple logging examples to advanced use cases in frameworks like Flask and Django, decorators play an important role in modern Python development. Understanding how decorators work helps in writing cleaner, more scalable, and more professional Python code.

NVIDIA B300 Blackwell Ultra: A Technical Deep Dive

DigitalOcean — Thu, 07 May 2026 23:53:39 +0000

The NVIDIA B300 (Blackwell Ultra) is NVIDIA's latest data center GPU, built for AI training and inference. In this deep dive, we break down the full architecture, from its dual-die design and 5th-generation tensor cores to NVFP4 precision and NVLink 5 scaling.

What we cover:
00:00 - Introduction
00:56 - Why the B300 exists
02:25 - B300 vs B200 vs H100 — the numbers
03:59 - Dual-reticle design & NV-HBI interconnect
05:04 - 5th-gen tensor cores & NVFP4
07:56 - 288GB HBM3e memory breakdown
09:04 - Multi-GPU & NVLink 5 architecture
10:38 - Performance & efficiency summary

Video Demo: How Does Model Compression Change AI Reasoning?

DigitalOcean — Thu, 30 Apr 2026 16:00:00 +0000

In this video, I benchmark Mistral-7B-Instruct-v0.2 on an NVIDIA H200 DigitalOcean GPU in three formats: FP16, INT8, and 4-bit AWQ — and test how precision impacts reasoning quality, speed, VRAM usage, and real serving density.

We’ll cover:
👉 What quantization actually does to model weights
👉 Where reasoning starts breaking down (FP16 → INT8 → 4-bit)
👉 Why memory savings don’t always reduce total GPU usage in vLLM
👉 Tokens/sec vs aggregate throughput
👉 When 4-bit wins — and when it doesn’t

If you're building AI systems and deciding between full precision and aggressive quantization, this is a practical infrastructure-level breakdown of the real tradeoffs.

Chapters:
00:00 Introduction
00:41 Understanding how quantization works
01:42 Why do you even need quantization
02:38 The experiment we ran
03:56 The observations we had
05:43 Overall learnings