DEV Community: DigitalOcean

How do I run untrusted LLM-generated code in an isolated environment?

DigitalOcean — Thu, 05 Nov 2026 18:01:52 +0000

Run untrusted LLM-generated code in a hardware-isolated sandbox with the network locked to an allowlist and only the credentials the task needs. DigitalOcean Managed Agents does this through Harness Runtime, which gives each session its own Firecracker microVM. The hypervisor, not container namespaces, enforces isolation. The service is in public preview.

Key takeaways:

Each session runs in its own microVM, so code in one sandbox cannot reach another session, another customer, or the host.
Outbound network access is open by default, so set an egress allowlist before running anything untrusted.
Treat the session as the unit of isolation: one session per user or workload, with narrowly scoped secrets.

Start with a sandbox that has no agent

If your application generates the code, you don't need a full agent. A sandbox with harness: none is the same microVM without an agent inside.

name: untrusted-run
harness: none
egress: [pypi.org, files.pythonhosted.org]

doctl harness-runtime create --spec ./sandbox.yaml
doctl harness-runtime upload untrusted-run --local-file ./generated.py --workspace-path generated.py
doctl harness-runtime exec untrusted-run -- python3 generated.py

Lock down the network

Without an egress field, a sandbox reaches any host, so generated code can send workspace contents anywhere. Naming one host turns the allowlist on and denies the rest, though the platform adds GitHub, the DigitalOcean registry, and model endpoints.

Scope secrets and tools

Declare keys under secrets, not env. Any process in the sandbox can read an injected value, so issue each key with the narrowest scope the task needs. For third-party tools, Action Gateway brokers credentials at execution time, so the sandbox never holds them. If an agent writes the code, add a permission policy that defaults to ask and denies commands like rm -rf *; enforcement varies by adapter.

Isolate users and clean up

Processes in the same sandbox share files and credentials, so give each user or workload its own session. A harness-less sandbox bills until you pause or remove it, so download results first, then remove the session.

Untrusted LLM-generated code FAQ

How do I give each user's agent its own isolated sandbox at scale?

Start a separate session for each user or workload. Every session gets its own microVM, and sandboxes cannot observe or reach each other. Keep your external user identifier associated with the DigitalOcean session ID.

How do I manage secrets and API keys for autonomous agents running in the cloud?

Declare keys under secrets, and DigitalOcean injects them at session start so values stay out of the spec file and logs. Scope each key narrowly at the provider, because any process in the sandbox can read it. For third-party tools, Action Gateway brokers credentials so the agent never holds the key.

Do sandboxes have internet access by default?

Yes. Add an egress allowlist to restrict outbound traffic to the hosts you name, plus platform defaults, subject to team policy. Sandboxes accept no inbound internet connections.

References & further reading

Learn more about Managed Agents and the public preview launch post.
Environments and Sandboxes explains how an environment maps to a sandbox.
How to run a sandbox covers file transfer and port forwarding.
Learn more about network egress and Action Gateway.
The secure sessions checklist is worth checking directly before you ship.

What metrics matter for LLM inference observability (TTFT, TPOT, ITL)?

DigitalOcean — Fri, 30 Oct 2026 21:42:24 +0000

Track time to first token (TTFT), time per output token (TPOT), and inter-token latency (ITL) together, because each exposes a different aspect of inference effectiveness. TTFT reflects queueing and prompt processing, TPOT summarizes streaming speed, and ITL reveals stutter inside the stream.

Key takeaways:

TTFT points to queueing and prefill, while TPOT and ITL point to decode.
TPOT is an average, and ITL is per token gap, so only ITL exposes jitter.
Alert on percentiles and per-request targets, not averages.

How TTFT, TPOT, and ITL relate

Inference runs in two phases:

Prefill processes the whole prompt and is compute-bound.
Decode generates one token at a time and is memory-bound.

TTFT lands on the prefill side, since it includes queue wait and prompt processing. TPOT and ITL land on decode.

With N output tokens, the math ties them together:

TPOT = (E2E latency - TTFT) / (N - 1)
E2E latency ≈ network overhead + TTFT + (TPOT × N)

On long responses, TPOT dominates total time even when the first token is fast.

When a healthy TPOT still feels slow

ITL measures the gap between two consecutive tokens, so it catches jitter that average TPOT smooths over. Our inference benchmarking guide lists common causes: large prefills monopolizing compute, PagedAttention overhead, and poor hardware topology—such as cross-NUMA traffic.

Read LLM inference observability metrics as a diagnostic

High TTFT, normal TPOT: requests are queueing or prompts are long. Check prefill capacity.
Rising TPOT: decode is memory-bound. Check GPU memory bandwidth and load.
Healthy TPOT, spiky ITL: something is interrupting decode, such as prefill interference.

Alert on percentiles and goodput

Averages hide the slow tail. Track P95 and P99 for each metric, then define goodput: the share of requests that meet every target at once, such as a TTFT ceiling, an ITL ceiling, and an end-to-end budget. Set thresholds per use case.

Check how your tool defines each metric

Definitions vary. NVIDIA's benchmarking docs, for example, treat ITL as the average gap between tokens, also called TPOT, and warn that tools differ on whether TTFT is included. Confirm what each tool calculates before comparing numbers. On the DigitalOcean Inference Engine, Serverless Inference metrics cover TTFT, end-to-end latency, error rates, and per-request cost with token counts. You can derive TPOT from those with the formula shared earlier.

LLM inference observability FAQ

What is disaggregated prefill and decode in LLM inference?

It runs prompt processing (prefill, which is compute-bound) and token generation (decode, which is memory-bound) on separate workers. The KV cache must move between them, and depending on the implementation, that transfer can delay TTFT or the second token’s ITL. On the DigitalOcean Inference Engine, Serverless Inference metrics report TTFT and end-to-end latency, so you can check whether an architecture change affects first-token time.

What is the difference between TPOT and ITL?

TPOT is the average time per token after the first token, calculated across the entire response. ITL, in the stricter sense, is the gap between two specific consecutive tokens, so it exposes stutter that a healthy TPOT hides. You can derive it from TTFT and end-to-end latency with TPOT = (E2E latency - TTFT) / (N - 1).

Why track P95 and P99 instead of averages?

Averages hide outliers. P95 and P99 show what your slowest requests experience, which is what you compare against latency targets and use to define goodput. Serverless Inference metrics on DigitalOcean cover TTFT, end-to-end latency, error rates, and per-request cost with token counts, so you can set those targets against both speed and spend.

References & further reading

Worth bookmarking: DigitalOcean's guide to LLM inference benchmarking, which covers the formulas above plus concurrency sweeps.
Learn more about Serverless Inference metrics and the DigitalOcean Inference Engine.
Worth checking directly before you compare tools: NVIDIA's LLM benchmarking metric definitions

AI tech stacks see increase in open source adoption

DigitalOcean — Tue, 27 Oct 2026 18:05:28 +0000

ICYMI AI: The latest AI trends, delivered to Dev.to

Tech Talk: The AI stack is more modular and multi-modal than ever

With a changing landscape of models, standards, and systems, no two AI tech stacks are alike. This open ecosystem is the emerging standard of architecture, something last week’s Open Intelligence Summit (OIS) highlighted with speakers from NVIDIA, Amplitude, Open Hands, and Nous Research.

According to internal DigitalOcean figures, 88% of our tokens run on open-weight models, a 27% increase from May 2026. This is accompanied by a 50% growth in token volume over the last six months.

It’s clear that loyalty to a single model is fading, and a new layer is emerging to manage churn: routers. Analysis from Vals AI frames routers as a new AI primitive. They choose models based on quality and cost per request, so applications don’t need to revisit model selection every time something new launches. They also provide more opportunities for developers to select models based on use case, rather than being locked into frontier models or a specific provider.

Open standards are doing the work that closed-loop systems and frontier models used to do, and the tooling is adapting:

DeepSeek opened Search and Fetch APIs that return live web content as Markdown for agents and RAG pipelines.
Perplexity open-sourced its agent sandbox.
Anthropic’s OpenTelemetry support means traces from different providers can share the same dashboards.

When models are interchangeable, differentiation moves to the layers you control: evaluation, observability, security, and cost. With the ability to fine-tune multiple layers, developers need concrete plans for AI stack management. Abstract your model calls behind a routing layer. Standardize on OpenTelemetry for agent traces. Build evals you can rerun against any model. And keep a cost dashboard open.

Our new Managed Agents and Agent Learning Services, announced at OIS, paired with our cache-aware Inference Router, allow you to build AI products and own your intelligence.

Industry news

AI adoption looks broad until you check what’s measurably working. a16z’s State of Markets II presentation found that 69% of S&P 500 companies have a live AI deployment and about 30% report a quantified result, but only around 2% track a metric for it, while the top 1% of companies by AI spend outspend the top 10% roughly eightfold. The gap is less about access to tools than about tying AI to a workflow you can measure.

ChatGPT is shifting from walls of text to answers you can interact with. OpenAI’s GPT-6 launch introduced “Intelligent UI,” which lets the model respond with charts, buttons, forms, and quick tools like savings calculators or bill splitters, all inside the conversation. OpenAI also says GPT-6 Instant starts answering web-search questions 44% sooner on average than GPT-5.6 Instant.

Community tutorials

Agents in Production, Part 1: Build an Overnight AI Support Agent using Managed Agents

An unattended support agent ran for 1 hour 48 minutes against a live board, triaged 99 tickets, stopped all 35 sensitive ones for a person, and sent zero replies, because the agent was never given a route to send. Learn how to scope board credentials, restrict egress to a single host, and add deterministic checks to drafts so the agent can't act outside its brief.

How to Safely Let AI Agents Call Production APIs using DigitalOcean Action Gateway

Across 1,300 episodes, 45 to 48 of every 100 runs without a gateway ended with at least one harmful change, and the agent re-sent a refund in 66 of 75 tries. See how idempotency keys, version checks, and approval tiers brought harmful changes to 0 of 100.

What real time-to-first-token looks like on DigitalOcean Serverless Inference

Across 380 calls to six models, five landed between 0.88 and 0.98 seconds median TTFT on short prompts, but DeepSeek V3.2 hit 78.3 and 186.9 seconds on two of 80 calls. Get the measurement method and code repository to run the same audit on your own models and prompt sizes.

Should You Use Spot or On-Demand GPUs for Model Training?

At a 60-minute cadence, a Spot GPU fits single-node jobs and multi-node jobs with elastic restart. Resuming a 98.3 GB checkpoint after a simulated reclaim took 706.5 seconds on MI355X and 436.6 seconds on B300. Walk through the break-even model and decision table, and plug in your own checkpoint cost and reclaim rate.

Deployment Isn't the Bottleneck: Inference Economics Is

The same $3-per-hour GPU can cost about $0.30 or about $3 per million tokens, depending on whether it runs at full or 10 percent utilization. Learn how cold starts, continuous batching, and KV-cache limits set cost per request, and run the arithmetic for your own model.

Video tutorials

Does context length affect inference cost linearly? We measured it

Every AI pricing page treats context length as linear: ten times the tokens, ten times the cost. This video shows what happens when you run one H200 GPU Droplet, serve Mistral 3 14B with vLLM, and run the same test from 2,000-token prompts up to 256,000.

Product updates

Now in Public Preview: DigitalOcean Insights

Insights gives you one place to see what your agents, applications, and infrastructure are actually doing, with metrics, logs, and traces built in and nothing to install or instrument. Follow an agent run from the model call through every tool it used and find exactly what broke, without switching tools or lining up timestamps yourself. It’s free to use during public preview.

Now Available: Inference Router Enhancements

Choosing the right model for every request shouldn’t require constant testing and manual routing. DigitalOcean’s enhanced Inference Router automatically selects models based on task performance, cost, and latency, helping you balance quality and efficiency as new models become available.

Now in Public Preview: Simulations

Simulations, a new capability of DigitalOcean Evaluations, lets you test a change through realistic multi-turn scenarios in an isolated sandbox on Harness Runtime, with the same configuration and Action Gateway tools your agent uses in production. Pull scenarios from Signals, generate them from your agent’s configuration, or describe one in plain language. An LLM judge scores every trajectory, and running a baseline against a candidate shows exactly what improved and what regressed.

Now in Public Preview: Signals

Signals watches production traffic on Managed Agents and Inference Engine and automatically highlights potential problems across user sentiment, tool execution, and infrastructure, with nothing extra to instrument. Flagged sessions become evaluation scenarios in one click. Signals is opt-in, designed to redact PII before processing, and keeps traces within your DigitalOcean Team boundary.

Where can I get free credits to test LLM inference

DigitalOcean — Fri, 23 Oct 2026 20:45:01 +0000

DigitalOcean gives new customers a free \$5 credit, with no credit card required, through its Managed Agents free trial. The credit covers inference on DigitalOcean-hosted models plus the agent runtime and tools around them. It's sized for testing, and Managed Agents is in Public Preview.

Key takeaways:

New DigitalOcean customers get \$5 in free credit, no credit card required, to test inference through Managed Agents.
Trial inference runs on DigitalOcean-hosted models such as Kimi K3, GLM 5.3, and DeepSeek V4 Flash. Frontier models like Claude and GPT bill at list price.
Credit drains as you use compute, storage, and tools, so pause idle sessions and watch your balance.

What the free credit covers

Managed Agents pairs Harness Runtime, which runs agent sessions in isolated Firecracker microVMs, with Action Gateway, which gives agents governed access to 16,000+ tools through one MCP endpoint. It integrates natively with the Inference Engine, so inference tokens, agent execution, and tool use come together in one place.

Refer to the Agent Droplets pricing page for more details about the trial offer, as well as details for paid Pro (\$50 per month) and Team (\$200 per month) plans.

Which models you can test

The trial's inference is built around models DigitalOcean hosts, including Kimi K3 and GLM 5.3. For scale, as of October 2026, the DigitalOcean-hosted GLM-5.3 Flash lists at \$0.15 per million input tokens and \$0.50 per million output tokens on the Inference pricing page. Frontier models like Claude and GPT are also available through DigitalOcean without a provider key of your own, billed at list price and outside the Agent Droplets plan discounts. If you bring your own provider key, that provider bills you directly.

DigitalOcean's Agent Droplets announcement estimates the free trial at roughly 4.5M tokens of inference.

How to start

Sign up as a new customer, open Managed Agents in the control panel, and start the trial. The Harness Runtime docs include a quickstart that launches an agent with one doctl command, and the launch post reports supported harnesses like OpenCode and Codex CLI.

What to watch while you test

Usage draws down your balance as agents run, and Harness Runtime has no per-session or per-product spend limit. Some Action Gateway tools, like Exa web search and web fetch, are prepaid and metered per call. Serverless inference is prepaid too, so access is suspended when credits and balances run out.

To save credits, pause sessions you aren't using. Pausing stops CPU and memory charges, though retained checkpoints still accrue storage fees. Current rates are on the Harness Runtime pricing page.

LLM inference free credits FAQ

Is there an inference provider with a free tier?

DigitalOcean offers new customers a free trial with \$5 in credit through Managed Agents, no credit card required. It's a starter credit for new customers rather than an ongoing free tier, and it covers inference on DigitalOcean-hosted models, runtime, storage, and tools.

Where can I run inference for an AI app and bill it all through one invoice?

Managed Agents integrates with the DigitalOcean Inference Engine, so inference on DigitalOcean-hosted models, agent compute, storage, and tool calls bill together. Models you reach with your own provider key are billed by that provider.

How do I cap inference billing to avoid surprise costs from runaway agents?

Harness Runtime has no per-session or per-product spend limit, and usage draws down your prepaid balance. Pausing a session stops CPU and memory charges, and auto-pause handles idle sessions, though retained checkpoints still accrue storage fees. Monitor usage closely on autonomous workflows.

References & further reading

Start with the Managed Agents product page for trial terms.
The launch post and the Agent Droplets announcement explain the architecture and the trial.
Worth bookmarking: the Agent Droplets pricing page for plan comparisons.
The Harness Runtime pricing page lists compute and storage rates. Worth checking directly before you build.
The Inference pricing docs have per-model token rates and prepaid billing rules.
Learn more about Harness Runtime and the Inference Engine.

Where to get H100 GPUs without long-term a commitment

DigitalOcean — Tue, 20 Oct 2026 15:24:05 +0000

You can get H100 GPUs without a contract from DigitalOcean GPU Droplets. Create one from the console or API, pay per second, and destroy it when the job ends. Commitments only pay off once your usage is steady.

Key takeaways:

On-demand H100 GPU Droplets pricing starts at \$4.41 per GPU per hour (as of October 2026), billed per second, with no term.
A 12-month reserved H100 rate is lower (\$3.26 per GPU per hour), but that is the commitment you are avoiding.
Powering off does not stop billing. Destroying the Droplet does.

What on-demand H100 access looks like

H100 GPU Droplets come with one GPU (80 GB of GPU memory, 20 vCPUs, 240 GiB of RAM) or eight (640 GB of GPU memory, 160 vCPUs, 1,920 GiB of RAM), at \$35.28 per hour for the eight-GPU size. They are available in New York (NYC2), Amsterdam (AMS3), and Toronto (TOR1). Billing is per second with a 60-second or \$0.01 minimum, whichever is higher.

When on-demand beats reserved

A 10-hour fine-tuning run on one H100 costs about \$44.10. A reserved plan only wins if you would keep that GPU busy roughly 74% of the year if you pay for the reserved GPU for the full 12 months (\$3.26 divided by \$4.41 is the break-even utilization). Prototypes, experiments, and bursty inference rarely clear that bar.

Keep costs predictable

The most common surprise is an idle GPU. Billing continues while a GPU Droplet is powered off because the hardware stays reserved, so destroy it when you finish. Set spend alerts to get an email when monthly spend crosses percentage thresholds of a monthly budget you set. Alerts notify you but don't cap spending.

If H100 capacity or budget is tight

The same on-demand catalog includes NVIDIA H200 at \$4.47 per GPU per hour and AMD Instinct MI300X at \$2.59. Spot GPU Droplets are cheaper but interruptible, with at least two hours' notice before reclaim (less in emergencies), and are currently listed for B300, MI350X, and MI355X rather than H100. If you only need model output rather than a server, the Inference Engine lets you pay for what you use.

H100 GPU FAQ

Which provider bills GPU usage by the second with no long-term commitment?

DigitalOcean GPU Droplets bill per second, with a minimum charge of 60 seconds or \$0.01, whichever is higher. On-demand plans have no contract, and billing runs from creation until you destroy the Droplet.

Do I still pay for a powered-off H100 GPU Droplet?

Yes. The GPU and other compute resources stay reserved while the Droplet is off, so billing continues until you destroy it.

Which AI platform supports both AMD and NVIDIA GPUs?

DigitalOcean GPU Droplets offer NVIDIA H100 and H200 alongside AMD Instinct MI300X on demand. You can create any of them from the same console or API.

References & further reading

Learn more about GPU Droplets pricing and reserved plans, and check it directly before you build, since rates change.
The Droplet pricing docs spell out per-second billing and the powered-off rule.
Worth bookmarking: GPU availability by region for which datacenters carry H100.
How to create GPU Droplets walks through image, plan, and datacenter choices.
Learn more about Spot vs on-demand GPU Droplets if interruptible capacity fits your workload.
For dedicated hardware instead of a virtual machine, see Bare Metal GPUs.

How do I cap inference billing to avoid surprise costs from idle GPUs or runaway agents?

DigitalOcean — Wed, 14 Oct 2026 16:31:01 +0000

You cap inference billing by layering controls, not by flipping one switch. On DigitalOcean, Serverless Inference is prepaid, so your balance is a real ceiling, and scoped keys, alerts, and cleanup jobs cover the rest. Spend alerts alone won't stop anything: they email you when spending crosses a threshold, but your usage keeps running.

Key takeaways:

Serverless Inference stops when funds run out, so fund a dedicated balance with only what you plan to spend.
Dedicated Inference and GPU Droplets bill by the hour until destroyed, so automate cleanup.
Agents need caps in your own code, because rate limits slow spending but don't cap it.

Use prepaid balances as a ceiling

Serverless usage draws from promotional credits, then the Inference and Agents balance, then your account prepayment balance. Only Serverless Inference and Managed Agents draw from the first, so it works as a separate AI budget. Auto-reload is on by default, so turn it off for a hard stop. A zero balance is also an outage, so reserve this for experiments and non-critical workloads.

Add alerts and scope your keys

Spend alerts email you at percentage thresholds of your monthly budget, within an hour of each threshold being crossed. Their sole purpose is notification. Give each service its own model access key, scoped to specific models and a VPC, so one leak or loop has a small blast radius. Regenerating a key takes effect immediately, which is how you want it to work during an incident.

Cap agents in code

Rate limits (up to 120 requests and 1.5M tokens per minute at the lowest serverless tier) limit speed, not total spend. Harness Runtime has no per-session or per-product spend limit, so an agent stuck in an unintended loop can quickly drain balances. Set a max output token count per call, a max step count per task, and a running spend counter that stops the run. You should also pause idle sessions, since paused sessions incur no compute charges.

Stop paying for idle GPUs

Dedicated Inference bills per GPU-hour, and GPU Droplets bill per second until destroyed. Powering off doesn't stop the meter. At \$30.32 per hour for 8x H100, an idle Dedicated Inference endpoint costs roughly \$728 a day.

To cap inference billing and avoid surprise costs, tag experimental GPU Droplets and destroy them nightly:

doctl compute droplet delete --tag-name ephemeral-gpu --force

Inference billing cap FAQ

Can I get usage alerts or hard spending caps on a managed AI platform before bills run away?

DigitalOcean spend alerts notify you at budget thresholds but don't limit usage. The closest hard cap is Serverless Inference's prepaid model, where access is suspended when credits and balances run out.

Does powering off a GPU Droplet stop billing?

No. The GPU stays reserved, so billing continues until you destroy the Droplet.

Which inference option doesn't penalize variable workloads?

Serverless Inference, which bills per token instead of reserving GPUs, is a solid choice for variable workloads. Dedicated Inference bills per GPU-hour, whether or not anyone calls it, so it suits steady traffic better than spiky traffic.

References & further reading

Spend alerts: worth setting up first.
Manage Serverless Inference prepayment: explains the order balances are drawn down.
Learn more about model access keys and inference limits.
Inference pricing: worth checking directly before you build.
Droplet pricing and billing
Harness Runtime pricing: worth bookmarking before you run agents at scale.

Jev: an AI model that can’t write a sentence

DigitalOcean — Fri, 09 Oct 2026 14:00:00 +0000

ICYMI AI: The latest AI trends, delivered to you on Dev.to

Tech Talk: TypeSafe’s Jev is a model that only returns probabilities

TypeSafe's Jev, released into early access on September 15, is a general-purpose classification model. It reads up to 32K tokens of input (such as a support ticket, a customer record, or a row of data). It answers any number of typed questions about it in a single parallel pass, and returns probabilities rather than prose.

Outputs are limited to three options: Choice, Score, and Noul (yes or no), so structured-output errors and bad tool calls drop to zero by construction. A valid answer isn't always the right one, and Jev doesn't claim otherwise.

What it does is tell you how sure it is. TypeSafe trains it for calibration, so its probabilities match real-world frequencies. When Jev says it's 80% confident, it's right about 80% of the time. That lets you set a threshold, automatically acting on high-confidence calls and routing the rest to a human or a larger model.

The pricing is what makes Jev worth a look. At \$0.042 per million input tokens with free output, TypeSafe estimates about \$0.0004 per decision, roughly 1/76th the cost of GPT-5.6 Terra on its benchmark. Within days, imitators appeared: Laya, Bespoke Nimble, and Kev, several of them open-weight fine-tunes of Qwen 3.5 you can run locally. OpenAI likewise followed with a Decisions API of its own.

For developers, the change is in how you structure the call. Instead of sending an LLM one broad question like "is this spam?" and parsing a paragraph back, you pass Jev the email once, along with a set of narrower typed questions, such as: Does the sender domain match the claimed company? Does it ask for credentials? How urgent is the language? Each comes back as a probability. Your own code then applies the thresholds and makes the call, so the routing policy lives in your repo, versioned and auditable, and the model only scores the inputs.

Jev is available today on Serverless Inference through the DigitalOcean Model Library, at Typesafe’s list price—no separate TypeSafe account needed. Pair it with the DigitalOcean Inference Router to send routing and classification traffic to Jev while keeping frontier models in reserve for the calls that need them.

Event calendar

Open Intelligence Summit

October 12-13th, San Francisco

The argument is no longer “Is open source AI cheaper?” It’s now “Who owns intelligence?” A movement is forming around open intelligence and generating the belief that the intelligence powering our AI products and companies should be more open, portable, and controllable.

Join DigitalOcean, NVIDIA, RadixArk, Inferact, and more for deep dives, demos, and discussions about the technologies that keep intelligence portable, interoperable, and in the hands of builders.

Industry news

Near-flagship intelligence at a fifth of the price was the headline at OpenAI DevDay, but the bigger shift is the stack forming around the model.

OpenAI announced GPT-6.1 Sol, which it says delivers near-Astra intelligence at a fifth of Astra's token prices, plus an Ultrafast tier with up to 6x faster generation in the API. It also added computer use to its Agents API and opened ChatGPT, which it puts at 1.2B weekly users, to developer-built plugins and partners like Devin, Notion, and Vercel.

Agent safety is moving below the application layer.

NVIDIA's Open Agent Safety Platform pairs OpenShell, an open-source runtime that traces agent actions and enforces policy, with Sentry, an out-of-band watchdog on BlueField-4 DPUs that NVIDIA says can quarantine an agent that leaves its boundaries within milliseconds. NVIDIA says more than 100 organizations, including Anthropic and CrowdStrike, are working with the technology, and DigitalOcean is among the companies supporting the platform.

New models available on DigitalOcean Inference Engine

We recently launched models from TypeSafeAI, DeepSeek, Z.ai, NVIDIA, OpenAI, and Anthropic, all with different strengths and use cases. Learn more about the models in our What’s New blog.

Start using them through the DigitalOcean API or directly in the Cloud Console.

DeepSeek-V4.1-Flash

DeepSeek-V4.1-Flash cuts the KV cache footprint to 1/4 of V4-Flash while outperforming DeepSeek's own V4-Pro flagship on coding and agentic benchmarks, addressing the cache-hit charges that often dominate agent costs. The 552B MoE model activates just 8B parameters on input, delivering frontier performance with infrastructure economics that improve as you scale.

GLM-5.3 and GLM-5.3-Flash from Z.ai

Z.ai's post-training translates to significant performance improvements for GLM-5.3, achieving a 50% lower cost per completed coding task than GLM-5.2 while jumping to 28.3 on Terminal-Bench 3.0. GLM-5.3-Flash adds native image and video input with the same agentic strength, giving teams both reasoning-heavy and efficient multimodal paths on a single platform.

Nemotron™ 3 Diarization from NVIDIA

NVIDIA's Nemotron™ 3 Diarization replaces the multi-stage speaker attribution pipeline (VAD, embeddings, clustering, post-processing) with a single 100M-parameter model, keeping audio within DigitalOcean's infrastructure rather than routing it to third-party APIs. It identifies up to 8 speakers (double the previous limit) with a 14.72% error rate.

GPT-6.1 Sol, GPT-6 Astra, Sol, and Luna from OpenAI

One week after GPT-6 Sol’s launch, OpenAI releases GPT-6.1 Sol, which offers near-Astra-level coding and substantial improvements in document understanding, computer use, and scientific work. OpenAI's three-tier GPT-6 family also debuts on DigitalOcean: Astra for computer use and coding, Sol for frontier reasoning at 90% lower cost than Opus 5, and Luna for high-volume tasks.

Claude Fable 5.1, Opus 5.5, and Sonnet 5.5 from Anthropic

Anthropic's latest point releases deliver focused gains: Fable 5.1 reaches 55.8% on Terminal-Bench 4.0 with 75% cheaper cache reads, while Opus 5.5 matches high-effort results at default effort using half the tokens and 20% lower per-token cost. Sonnet 5.5 runs 30% faster with equivalent cost savings, with a shift toward better token efficiency and lower verification burden.

Community tutorials

How naive failover between inference providers makes outages worse

In a six-hour fault-injection run on an H200 GPU Droplet, the standard retry-with-backoff policy pushed 3.54x normal load onto a degraded provider, while a retry budget plus circuit breaker held it to 1.02x, and naive retries left 3,743 zombie requests generating 959,232 output tokens for clients that had already disconnected. Learn how to classify inference errors before failing over, replace fixed retry counts with a budget, and verify whether your provider actually stops billing when you cancel a stream.

What vision-language models cost to serve at production scale

A single uploaded photo can add more than a thousand input tokens before the model writes a word, and at 500,000 requests a month, that gap is the difference between \$155 on serverless Nemotron Nano 12B VL and \$3,175.20 for a dedicated H100 running 24/7. Walk through the per-request cost math, the roughly 10.2 million-request breakeven, where dedicated wins, and six levers for cutting image-token spend.

Speculative decoding's hidden cost: the KV cache capacity you give up, and where throughput turns negative

On a 141 GB H200 serving Qwen3-14B with vLLM, a separate Qwen3-0.6B draft model consumed about 43.8 percent of baseline batch slots, and even n-gram speculation, predicted to cost nothing, measured a 2.8 percent capacity tax. See where each method's throughput crosses below baseline as concurrency rises, and get the config.json arithmetic to predict a draft model's KV cost before spending GPU hours.

Cache reliability is a pricing term: what I learned measuring DeepSeek-V4.1-Flash on DigitalOcean Serverless Inference

DeepSeek-V4.1-Flash has the deepest cache discount in the Serverless Inference catalog at \$0.006 per million cached tokens, yet across 844 API calls it hit the cache on only 78.6% of warm turns, and with a miss priced at 50x a hit, it cost 2.2x what GLM-5.3-Flash did for the same 15-turn, 200K-token sessions. Discover how DigitalOcean's 64-token-block prefix cache works, how to log your own hit rate from the usage object, and why reasoning_effort: max cost up to 1.88x more without improving accuracy.

Why DeepSeek-V4.1-Flash splits reading from writing

DeepSeek-V4.1-Flash's Causal Encoder-Decoder splits a 40-layer model into a 20-layer encoder that runs prefill at 8B active parameters and a full stack that decodes at 16B, shrinking the global KV cache to about 890 bytes per token, roughly a quarter of V4-Flash's. Learn why that asymmetry, not the 74.2 versus 62.7 DeepSWE score against V4-Pro, is what lowers the bill for read-heavy coding agents.

Product updates

Introducing DigitalOcean Managed Agents, now in Public Preview

AI agents are ready to take on more, and we built the infrastructure to match. DigitalOcean Managed Agents brings together Harness Runtime, a purpose-built runtime for running agent sessions at scale, and Action Gateway, governed access to the tools agents need to get real work done, all in one platform.

Deploy your preferred agent harness, Claude Code, Codex CLI, OpenCode, and more, or bring your own, and connect to 16,000+ tools across 500+ providers through a single managed endpoint. Sessions are ready in under a second, resume in about 300 milliseconds with full context intact, and you only pay for the compute you actually use.

Whether you're automating support workflows, refactoring code across repositories, or building the next agent-powered product, Managed Agents gives you the infrastructure to do it, without building it yourself.

DigitalOcean MicroVMs now in private preview

Isolated, Firecracker-based VMs that pause when idle and resume with state intact, built for agent workloads that run in bursts. The same layer that powers Managed Agents is now available directly to teams building their own agent platforms, sandboxes, and short-lived jobs.

What is disaggregated prefill and decode in LLM inference?

DigitalOcean — Tue, 06 Oct 2026 18:03:00 +0000

What is disaggregated prefill and decode in LLM inference?

Prefill is compute-bound, decode is memory-bound. Learn why colocating them hurts TTFT and ITL, and when disaggregated serving pays off.

Every LLM request runs through two phases with effectively opposite hardware profiles. Understanding the split explains most of the latency and cost behavior you see in production, and why disaggregated prefill and decode has become a standard serving pattern.

Key takeaways:

Prefill is compute-bound and sets TTFT; decode is memory-bandwidth-bound and sets ITL/TPOT.
Running both on the same GPUs creates prefill/decode interference, visible as TTFT spikes and ITL jitter.
Chunked prefill and continuous batching reduce interference; disaggregating the phases onto separate pools removes it, at the cost of a KV cache transfer.
Disaggregation is worth it for long prompts, high concurrency, and strict streaming SLOs. A managed option like DigitalOcean Serverless Inference gives you it without the setup; for small self-hosted deployments, a well-tuned single pool on a GPU Droplet running vLLM is often enough.

The two phases of LLM inference

Prefill processes the entire prompt in one forward pass and builds the KV cache, the attention state every later token reads from. It is compute-bound: arithmetic intensity is high and the GPU spends its time doing math, not waiting on memory. Prefill governs time to first token (TTFT), and its attention cost grows quadratically with prompt length.

Decode generates output one token at a time. Each step reloads the model weights and the growing KV cache from HBM to produce a single token, so it is memory-bandwidth-bound. Decode governs inter-token latency (ITL) and time per output token (TPOT). A GPU with higher memory bandwidth streams tokens faster even if its FLOPs are unchanged.

Why colocating prefill and decode causes latency problems

Most serving engines batch prefill and decode on the same GPUs. That is where the interference shows up: a long prompt's prefill can monopolize compute while dozens of in-flight decode requests stall, and users see ITL jitter or a TTFT spike. The two phases also want different batch sizes and parallelism strategies, but colocated they must share one configuration.

Chunked prefill and continuous batching soften this by interleaving prompt chunks with decode steps, which is why our inference trilemma guide recommends them for latency-sensitive deployments.

What disaggregation changes

Disaggregated serving, formalized in the DistServe paper, runs prefill and decode on separate GPU pools. A prefill worker computes the KV cache, ships it over a fast interconnect (RDMA, RoCE, or a transfer library like NIXL or LMCache) to a decode worker, and the decode pool streams tokens without prefill interference. Because each pool is sized and tuned independently, you can hit TTFT and ITL targets at the same time rather than trading one for the other. The cost is added complexity plus a KV transfer step; that transfer can land on second-token latency rather than TTFT depending on how orchestration is done.

This is the architecture DigitalOcean already runs in production: Serverless Inference uses disaggregated serving over a RoCE network, paired with prefix-aware routing so shared system prompts skip redundant prefill entirely.

References & further reading

DistServe: Disaggregating Prefill and Decoding for Goodput-optimized LLM Serving: the primary source for the pattern, worth reading before you build.
LLM inference benchmarking: measure what matters: a clear walkthrough of TTFT, TPOT, ITL, and how each maps to prefill or decode.
The LLM inference trilemma: practical tuning advice, including when to reach for chunked prefill.
Learn more about debugging p99 TTFT and deploying NVIDIA Dynamo on GPU Droplets, a framework built around disaggregated serving.
GPU Droplets: current hardware and regions, worth checking directly since availability changes.

How to route LLM requests by cost vs. latency

DigitalOcean — Fri, 02 Oct 2026 18:33:17 +0000

Routing LLM requests by cost and latency means sending each request to the cheapest or fastest model that still meets the quality bar, rather than hardcoding a single model for everything. Production traffic isn't uniform: routine lookups, complex troubleshooting, and background jobs have different cost and latency requirements, even within a single app.

Using an inference router like DigitalOcean Inference Router involves defining request categories, assigning each a selection policy (cost, latency, or a benchmarked best fit), and setting fallback models so an unavailable model doesn't break the request. Cost is the price per token (which can vary by up to 100x across a model catalog); latency is mainly the time to first token (TTFT). The two don't move together, so route per task rather than using a single blended score.

Key takeaways:

Routing is a per-workload policy, not a search for "the best model."
Cost and latency usually need separate, explicit priorities.
Fallback and cache-aware routing protect the savings that routing is meant to deliver.
The DigitalOcean Inference Router implements this as configuration, not custom infrastructure.

LLM routing strategies

Static rules: Hardcode model per request type. Simple, but brittle.
Policy-based routing: A model pool per task with a cost/latency/optimal policy. Most production systems land here.
Dynamic routing: A classifier infers the task and policy automatically.

Routing LLM requests: A quick decision framework

Workload	Priority	Policy
Real-time chat	Latency	Speed-optimized (TTFT)
Routine lookups	Cost	Cost-optimized
Complex reasoning	Quality	Benchmarked "optimal"
Batch/background	Cost	Cost-optimized

Fallback models keep requests completing when the preferred model is down or rate-limited.

Cache-aware routing matters too: switching models to save a fraction of a cent can break a cached prompt and cost more overall.

How the DigitalOcean Inference Router routes LLM requests

Inference Router tasks pair a model pool with a selection policy: preset tasks default to a benchmarked Optimal policy; custom tasks choose Cost Efficiency, Speed Optimization (TTFT), or Manual Ranking.

Fallback models handle unmatched traffic. X-Model-Affinity preserves cache reuse, and X-Routing-Max-Switch-Spend-Pct (default 20%) caps the cost of switching mid-session.

It's a drop-in change. Set "model": "router:your-router-name". Routing decisions typically resolve in about 200ms, billed at the serving model's standard rate with no separate router charge during public preview.

LLM routing FAQ

What is LLM routing?

LLM routing is the practice of directing each request to the model best suited to it—by cost, latency, or task fit—instead of sending every request through one model regardless of what it needs. The DigitalOcean Inference Router implements this as a managed feature, so teams get task-aware routing and fallback handling without building it themselves.

How do you model cost-per-token for LLM inference?

Multiply input tokens by the model's input rate and output tokens by its output rate, then sum the two, since the rates are usually very different. Because rates vary widely across a model catalog, this is best tracked per task rather than as one blended number. The DigitalOcean Inference Router reports token usage and cost-relevant metrics per model and per task in its Analyze view.

What metrics matter for LLM inference observability?

Time to first token (TTFT), time per output token (TPOT), and inter-token latency (ITL) are the core metrics for judging responsiveness. A latency-optimized routing policy should be measured against these, not just overall request time. The DigitalOcean Inference Router uses TTFT specifically as the basis for its Speed Optimization selection policy.

What causes cold start latency in GPU inference, and how do you avoid it?

Cold starts happen when a model has to load onto available GPU capacity before it can serve a request, rather than running on an already-warm instance. Pooled serverless capacity and routing policies that keep related traffic on one model reduce how often this happens. The DigitalOcean Inference Router uses model affinity to keep a session's requests on the same warm model, rather than triggering repeated cold starts across models.

Can I route by both cost and latency at the same time?

Yes—typically by defining separate tasks or policies for different request types rather than one blended score. For example, cost-first for background work and latency-first for real-time chat. The DigitalOcean Inference Router supports this directly. Each task in a router can have its own model pool and its own Cost Efficiency, Speed Optimization, or Manual Ranking policy.

References & further reading

How to use Inference Router: The full documentation on tasks, selection policies, fallback models, and the cache-aware router referenced throughout this piece. Worth reading directly before you configure a router for production traffic.
Inference pricing: Current per-model Serverless Inference rates, since routing decisions are only as good as the pricing data behind them.
How DigitalOcean built Inference Router: An inside look at the architecture decisions behind cache-aware routing and fallback handling, useful if you want the reasoning behind the mechanics.
DigitalOcean Serverless Inference: a deep dive: Background on the serverless layer that Inference Router sits in front of.
DigitalOcean's best LLM routers roundup: A good next read if you're comparing routing options across vendors rather than implementing a policy on a platform you've already chosen.
AI Inference Engine overview: shows where Inference Router fits alongside serverless, batch, and dedicated inference on DigitalOcean's broader platform.

Which inference provider has both serverless and dedicated GPU tiers on the same platform with one API?

DigitalOcean — Mon, 28 Sep 2026 17:17:40 +0000

DigitalOcean's Inference Engine puts serverless, dedicated, and batch inference behind a single endpoint, on the same GPU Droplets infrastructure, so a workload can move from pay-per-token prototyping to a reserved-GPU deployment without switching platforms or rewriting its integration. A handful of other inference providers share a single API across serverless and dedicated tiers, too, but they generally limit that unification to inference alone, rather than to the broader cloud the dedicated GPUs run on.

Why serverless-only or dedicated-only platforms create a migration tax

Most teams don't start with a clear answer to "How much GPU capacity do we need?". They prototype on a pay-per-token endpoint, and only once traffic gets steady and predictable does it make financial sense to reserve GPUs outright. Many inference providers treat serverless and dedicated as separate products, with distinct SDKs, auth, and billing dashboards, so moving from one to the other means rewriting integration code in the middle of a product's growth curve, which is exactly when a team has the least time to do it. A provider that puts both tiers behind a single API avoids that: you prototype on serverless and repoint the same client to a dedicated endpoint once volume justifies it, often by changing a model or endpoint parameter rather than an integration.

How the DigitalOcean Inference Engine unifies serverless, dedicated, and batch inference

The DigitalOcean Inference Engine runs three inference patterns over the same GPU Droplets infrastructure, behind a single endpoint. These solutions include real-time Serverless inference across 70+ open-weight and frontier models, Dedicated inference on reserved, single-tenant GPU capacity with support for bringing your own model, and asynchronous Batch Inference for large, non-real-time jobs. The Preview Inference Router adds cost- and latency-based routing on top, and a shared Model Playground and evaluations tooling sit on the same account.
That's what makes DigitalOcean a solid solution for AI-native companies requiring both serverless and dedicated GPU tiers. The unification isn't limited to the inference API itself. It extends to the GPU Droplets the dedicated tier runs on, as well as the rest of the AI-Native Cloud around them. A workload doesn't need to switch vendors, contracts, or billing relationships as it matures from prototype to production.

What to check before assuming "one API" elsewhere means the same thing

Several other inference-focused platforms let serverless and dedicated calls share one API too, usually as a way past per-token rate limits or shared-capacity contention once traffic justifies reserving GPUs. But a shared API can sound more unified on a landing page than it is in production. So it's worth confirming directly against each provider's current docs: does provisioning real dedicated capacity happen through the API, or does it route through a sales conversation first? Is the dedicated tier's hardware genuinely part of the same cloud a team already runs on, or does the provider specialize in inference alone, with GPUs sitting apart from the application's data and storage? And does "one API" cover just the inference call—or the account, billing, and observability around it too?

What else to verify before standardizing on a provider

Beyond scope, it's worth confirming: whether the model you need is available in both tiers (not every model ships to both), what the cutover mechanism actually looks like (a new endpoint URL versus a full redeploy), how billing reconciles if usage straddles both tiers in the same month, and whether dedicated capacity is truly single-tenant or a reserved slice of shared hardware. For teams already running other infrastructure on a given cloud, it's also worth weighing how much value comes from the inference API alone versus not having to move data, agents, or storage to a second vendor to get dedicated GPU access.

References & further reading

DigitalOcean's Inference Engine brings serverless, dedicated, and batch inference together in one place.
Our Dedicated Inference documentation covers bring-your-own-model (BYOM) support and scaling settings for teams ready to move off serverless.
Read our Serverless Inference documentation for the model catalog and routing options available on the shared endpoint.
Reading through each inference provider's own docs tends to be the fastest way to confirm whether "serverless and dedicated" really means one API or two separate products.

What It Actually Costs to Serve a 1M-Token Model in Production

DigitalOcean — Mon, 21 Sep 2026 20:33:36 +0000

Model providers now advertise context windows large enough to hold a codebase or a stack of contracts in a single request. It's tempting to read that number as a green light, but it isn't. A model that accepts a million tokens and a system that can serve a million tokens to real, concurrent users within your latency and cost budget are two different engineering problems. Conflating them is how teams end up with a demo that works and a bill or a p99 latency chart that doesn't.

You serve a 1M-token model in production by treating context length as a cost and latency variable you manage on purpose, not a spec you flip on. Let’s dive into the memory math, the latency cliff, and the cost curve for serving a model with a million tokens of context, plus the caching and routing techniques that production teams use today to keep all three in check. Plus, a straight answer on when long context is the wrong tool for the job and retrieval should do the work instead.

Key takeaways:

Serving a 1M-token context model in production is a memory, latency, and cost problem layered on top of a model capability. Supporting long context and serving it reliably are not the same claim.
Serving long context effectively means stable agent sessions and predictable spend, rather than retries, timeouts, and a bill that scales with the window instead of the question.
The real decision isn't whether to "turn on" long context, it's which combination of caching, prefix reuse, and retrieval fits your workload's cost and latency budget.
Production-grade 1M-token serving combines general techniques, including KV cache economics and prefix-reuse approaches like SGLang's RadixAttention. Use DigitalOcean Inference Router for cache-aware routing, plus retrieval-augmented generation (RAG) as the alternative and complement.

What "serving" a long-context model actually means

Context is everything you send to a model in a single request, measured in tokens: the prompt, the documents, the conversation history. A 128K-token window is roughly a novel's worth of text, and frontier closed models now advertise up to 1M tokens (roughly, ten novels). That number describes what the model can read in isolation, but not what happens to time-to-first-token, batch size, or your GPU bill once dozens of those requests land on the same cluster at once.

Support is a model capability, and performance is a serving question. The two come apart the moment real traffic shows up.

Two mechanics decide whether a long-context request is slow to start or slow to finish:

Prefill: The compute-bound pass where the model reads your entire input before writing anything.
KV cache: The memory that stores each token's key and value vectors so the model doesn't reread the input on every step.

Dig in to how prefill/decode disaggregation works: the two-phase split behind every long-context serving decision below.

Why its worth optimizing for long-context

Getting long-context serving right pays off in multiple ways.

Here's what's actually at stake once you're running this in production:

Stable agent and document workloads: A correctly cached long-context session makes it possible for a coding agent or a multi-turn document review to hold state across dozens of turns without resending the same 100K-token context on every message.
Costs you can forecast: Once you know whether traffic is cache-friendly or retrieval-shaped, it's easier to accurately predict spend.
No silent accuracy loss: Long-context serving done well accounts for the fact that models answer less reliably as input grows. This happens by routing precision lookups to retrieval instead of trusting long-context recall, and by testing accuracy at the context lengths you'll actually run rather than assuming a bigger model or window is automatically more reliable.
Latency you can put in an SLA: Understanding the prefill/decode split lets you set a time-to-first-token target you can actually hit, instead of promising an interactive response time a long-context request can't deliver.
Room to scale without a rebuild: Architectures built around caching and routing from the start absorb growth in context length and concurrent users without the rewrite that a "just resend everything" approach eventually forces.

What to evaluate before you commit to a long-context approach

Long-context serving isn't a single decision, but rather a set of infrastructure and architecture choices considered together:

Cache-awareness: Check whether the serving layer reuses the KV cache across requests that share a common prefix, rather than recomputing it each time. The DigitalOcean Inference Router, for example, pins a session to the same model so that a warm cache isn't discarded mid-task when switching to a different, cheaper model.
Batching under long-context load: Confirm the system maintains healthy batch sizes as the average context length grows, using techniques like chunked prefill and paged memory allocation, rather than letting a single long request starve everything queued behind it.
Isolation from noisy neighbors: Ask whether long-context and interactive traffic run in separate pools. Dedicated Inference on DigitalOcean exists specifically so a large document-processing job doesn't blow up latency for interactive users on the same infrastructure.
Pricing that reflects the real cost curve: Look for tiered, per-token pricing that acknowledges long-context requests cost more proportionally, plus a prompt-caching rate that's meaningfully cheaper than a full rewrite.
Observability into cache health, not just request counts: Make sure you can see KV-cache occupancy, eviction, and preemption counts—not just average latency—since that's what actually explains a p99 problem caused by long-context traffic.

Resend, cache, retrieve, or both: what each costs per session

The cleanest way to understand the cost of each approach is to price a full session rather than a single token.

These examples take a 10-turn agent conversation carrying a stable 128K-token context—a codebase or a document set—where each turn adds a short question and gets a short answer:

Strategy	Use Case	Mechanism	Cost per 10-turn session (128K context)
Resend the full context every turn	Short, one-off sessions where simplicity matters more than cost	Full context re-sent as input tokens on every turn	$3.93
Cache the context; reuse it	Stable, reused context — a codebase, a document set, a long agent session	Prefill cost paid once; later turns reuse the cached KV state	$0.95
Retrieve a relevant slice (RAG)	Large or changing corpora where each query touches a small fraction of the data	Only the relevant slice enters context; the rest of the corpus stays out	$0.33

Figures per DigitalOcean's long-context serving tradeoffs analysis, priced at Claude Sonnet's published rates ($3.00/M input tokens, $3.75/M to write the cache, $0.30/M to read it).

Caching cuts the bill roughly 4x against resending; retrieval cuts it roughly 12x. At thousands of sessions a day, that architecture choice matters more than any provider's per-token rate.

The four tradeoffs that decide whether long-context works in production

A 1M-token context window is a spec, not a guarantee that serving it will work. Memory, latency, throughput, and accuracy each break in their own way as context grows, and "supports 1M tokens" and "serves 1M tokens well" can be very different claims.

Memory: the KV cache doesn't scale the way people expect

For every token a model reads, it stores a key and a value vector so it doesn't have to reprocess the input on each new token. That cache grows in direct proportion to context length: the math works out to roughly 43GB for a single 128K-token request on a 70-billion-parameter model in BF16—close to the model's own memory footprint. Push to a million tokens, and the cache alone exceeds what a single high-end GPU holds, before the model's own weights are even loaded.

Latency: why time-to-first-token is the number that breaks

Attention compares every token against every other token, so compute grows with the square of input length: double the context, quadruple the work. That happens entirely during prefill, before a user sees a single word, which is why time-to-first-token ranges from milliseconds with short context to multiple seconds with long context. It also means one long request can hold a GPU for seconds, quietly degrading p95 and p99 latency for every short, unrelated request queued behind it.

Throughput: the bandwidth ceiling no amount of compute fixes

Generating each output token requires reading the entire KV cache from GPU memory, and the read speed is limited by hardware bandwidth. Double the context, double the cache, and tokens-per-second roughly halves, with a limit that has nothing to do with how much compute is sitting idle. A cluster can appear underutilized on a FLOPS dashboard and still serve long-context requests slowly because the bottleneck has shifted to the memory bus.

Accuracy: why a bigger window doesn't guarantee a better answer

Longer context doesn't reliably mean a better answer. A 2026 measurement study found that accuracy at long context lengths doesn't track model size — a smaller model sometimes outperformed a larger sibling — and that a wrong answer just becomes a retry, which turns an accuracy problem into a latency problem measured in total time-to-correct-answer. Production systems that route by expected accuracy at a given context length, not just by cost, cut that wasted retry time substantially.

How production teams are actually solving the long-context problem today

Successful teams aren't fighting the KV cache, they're making it reusable. SGLang's RadixAttention implements prefix caching alongside continuous batching, so a repeated prompt prefix (i.e. a system prompt, a shared reference document, a stable set of few-shot examples) gets computed once and reused across requests instead of recomputed on every call. That keeps throughput comparatively stable even as context length grows into the millions of tokens, which is exactly the workload where naive serving falls apart first.

You can run this pattern yourself on DigitalOcean GPU Droplets, or skip the setup work with the SGLang 1-Click Marketplace app, which handles the underlying infrastructure for you.

DigitalOcean's Inference Router adds cache-aware routing on top of that. It keeps a session pinned to the same model, so a warm KV cache doesn't get discarded mid-task by a switch to a different model chosen for cost reasons.

If you're self-hosting long-context serving on GPU Droplets rather than going through a managed router, three additional levers matter:

Chunked prefill, which breaks a large prefill pass into pieces so it doesn't stall the rest of the batch.
Tuning max batch size and max batched-token limits for your actual mix of context lengths.
Keeping long-context traffic in a separate pool from interactive traffic so one doesn't degrade the other.

When time-to-first-token degrades as context grows, DigitalOcean's inference-debugging guidance points to KV-cache pressure—not the scheduler—as the usual culprit. This is why watching KV-cache occupancy, eviction, and preemption counts diagnoses the problem faster than staring at a scheduler queue.

Do you even need long context? Choosing between caching, retrieval, or using both

If a model accepts a million tokens, it's fair to ask why you'd also use a vector database, a chunking pipeline, and a retrieval step. The answer is that the price of skipping retrieval includes the quadratic prefill cost, the multi-second time-to-first-token, and the tiered per-token pricing. They’re all paid on every request—even when the answer only needed to use three paragraphs of the input.

As a practical rule, long context plus caching wins when your working set is stable and reused. For example, many requests against the same document set, or a day-long agent session where the context is the state. RAG wins when your corpus is larger than any context window, changes frequently, or is mostly irrelevant to any single query. Precision lookups favor RAG even when the full corpus would technically fit, since retrieval keeps the model working at the shorter lengths where it's most reliable. Most production systems end up using both: retrieval to decide what enters the context, a long window to hold a generous amount of it, and caching wherever it repeats.

Long-context LLM serving FAQ

How do you serve long-context (1M token) models in production?

Serving a 1M-token model in production means treating context length as a cost and latency variable you manage on purpose, not a spec you flip on, since supporting long context and serving it reliably are not the same claim. It requires managing KV cache memory, the quadratic prefill cost, and the resulting time-to-first-token, then choosing the right mix of caching, prefix reuse, and retrieval for your workload. The DigitalOcean Inference Router applies cache-aware routing to this problem by pinning a session to the same model so a warm KV cache isn't discarded mid-task.

What is disaggregated prefill and decode in LLM inference?

Prefill is the compute-bound pass where the model reads your entire input before writing anything, while decode is the token-by-token generation that follows. Attention compares every token against every other token during prefill, so compute grows with the square of input length, meaning one long request can hold a GPU for seconds and degrade latency for every short request queued behind it. Chunked prefill breaks a large prefill pass into pieces so it doesn't stall the rest of the batch.

What metrics matter for LLM inference observability (TTFT, TPOT, ITL)?

Time-to-first-token is the number that breaks under long context, ranging from milliseconds with short context to multiple seconds with long context, because it's entirely determined by the prefill pass. Beyond latency averages, observability into KV-cache occupancy, eviction, and preemption counts is what actually explains a p99 problem caused by long-context traffic, not request counts alone. KV-cache pressure, not the scheduler, is the usual culprit when time-to-first-token degrades as context grows.

Which inference provider has the best caching to reduce token costs for long autonomous coding sessions?

Caching the context and reusing it lets a coding agent hold state across dozens of turns without resending the same 100K-token context every message. SGLang's RadixAttention implements this as prefix caching alongside continuous batching, computing a repeated prompt prefix once and reusing it across requests. DigitalOcean supports this pattern on GPU Droplets or through the SGLang 1-Click Marketplace app, and the Inference Router adds cache-aware routing so a warm cache isn't discarded by a mid-task model switch.

How do you model cost-per-token for LLM inference?

The cleanest way to model cost is pricing a full session rather than a single token: resending the full context every turn costs $3.93 for a 10-turn, 128K-context session, caching and reusing that context brings it to $0.95, and retrieving only the relevant slice (RAG) brings it to $0.33. Caching cuts the bill roughly 4x against resending, while retrieval cuts it roughly 12x, a difference that matters more at scale than any provider's per-token rate. DigitalOcean prices around this same curve, with tiered per-token rates that account for long-context requests costing more than proportionally, plus a prompt-caching rate meaningfully cheaper than a full rewrite.

Serve long-context models without managing the cache yourself

The DigitalOcean Inference Engine prices per token by model, with prompt-caching rates listed right next to the standard rate. That means you can work out what a long-context session costs before you run it, not after the invoice shows up. The Inference Router automatically applies cache-aware routing, pinning a session to the model whose KV cache is already warm rather than discarding it during a mid-task switch.

That gets you:

Automatic model-to-request matching, so cost- or latency-sensitive requests route to the right model without the need for writing the routing logic yourself.
Dedicated Inference to isolate long-context, latency-sensitive workloads from noisy-neighbor traffic.
Serverless-to-dedicated scaling with no migration step, so a workload that outgrows shared capacity doesn't require re-architecting.

Get started with DigitalOcean's Inference Engine →

Hippocratic AI cut prefill latency 2x on long-context clinical sessions running production inference on DigitalOcean's cache-aware routing.

https://www.youtube.com/watch?v=WeAAISnnNVA

These customer-reported results reflect Hippocratic AI’s methodology, workload, configuration, and measurement period and may not be representative of results in other environments. Results in customer environments may vary depending on configuration, implementation, and usage. Results and/or savings are not guaranteed.

Any references to third-party companies, trademarks, or logos in this document are for informational purposes only and do not imply any affiliation with, sponsorship by, or endorsement of those third parties.

AI token billing continues to cause sticker shock

DigitalOcean — Fri, 18 Sep 2026 16:25:40 +0000

Tech talk: are there inference providers that are transparent about token usage?

A top pain point for companies using AI is the discrepancy between expected and actual invoice rates. Particularly when it comes to tokens. The increase in AI usage and tokenmaxxing has created a paradox where a provider knows exactly how many tokens it generated on your behalf, yet you find out only after the fact. It's become a real cost problem when using reasoning models. Internal step-by-step reasoning counts as billed output, but it's invisible by design. The providers willing to open up their token accounting now are betting that transparency will become table stakes before their competitors catch up.

The 2025 study, “Predictive Auditing of Hidden Tokens in LLM APIs via Reasoning Length Estimation,” estimated that hidden reasoning tokens can account for over 90% of a model's total token spend on complex tasks. A short, simple-looking answer can hide 10,000+ reasoning tokens, but the only confirmation of their use is the bill.

This trend in billing surprises has rightfully drawn industry attention. Especially with token usage projected to grow 24x to 120 quadrillion tokens per month by 2030, according to research from Goldman Sachs. Developers have started noticing the gap between what usage dashboards imply and what invoices actually charge, and the industry's response is arriving on two tracks.

First, standards bodies are getting involved. The Linux Foundation's proposed Tokenomics Foundation wants open, vendor-neutral measurement standards so token accounting isn't just whatever a provider's internal dashboard says it is.

Second, providers like Groq and Cerebras are sharing flat per-token pricing. And the newly launched DigitalOcean Inference Router helps your team route tasks to the most cost-effective or optimal model, providing budget control based on your priorities for specific AI tasks.

Thanks to rapidly scaling AI adoption, developers know these costs aren’t a simple line item to ignore. When evaluating inference options, knowing exactly how your tokens are used and what that can mean for overall billing is essential.

Event calendar

Open Intelligence Summit by DigitalOcean

The argument is no longer “Is open source AI cheaper?” It’s now “Who owns intelligence?” A movement is forming around open intelligence and generating the belief that the intelligence powering our AI products and companies should be more open, portable, and controllable.

Join DigitalOcean, RadixArk, Inferact, and more for deep dives, demos, and discussions about the technologies that keep intelligence portable, interoperable, and in the hands of builders. The two-day event runs at The Midway in San Francisco from October 12-13th.

Industry news

Coding agents are earning real trust in production codebases

A study by researchers at Polytechnique Montréal, led by Canada Research Chair Foutse Khomh, analyzed 220,612 PRs across 539 Python repos and found Claude Code's PRs merge 84.3% of the time—well ahead of Codex (73.5%), Cursor (63.9%), Copilot (59.6%), and Devin (43.0%)—with bug rates comparable to or lower than human-written code.

OpenAI launches GPT-6 Astra

OpenAI introduced GPT-6 Astra, a new frontier model focused on advanced reasoning, computer use, coding, scientific research, and professional workflows. The model achieves major gains across benchmarks, including 99.9% on ARC-AGI-3 and 100% on ExploitBench, while also improving task efficiency and alignment.

Anthropic introduces Claude Fable 5.1 and Mythos 5.1

The new models target advanced coding, knowledge work, and scientific research, with Fable 5.1 offering stronger performance at lower cost and Mythos 5.1 providing more permissive safeguards for vetted cybersecurity and life-science users.

Grok Bot comes to Android

Cursor has launched the Grok Bot Android app, letting users assign tasks to AI Bots from their phones, continue conversations across devices, and monitor multiple Bots running in parallel on cloud computers. The app is currently in beta for eligible Cursor and SuperGrok plans.

Community tutorials

Why Your Best Model Is Two Models: Routing Between Kimi K3 and Claude

Hardcoded routing logic gets messy fast and using a small model like Haiku to classify requests means paying for two calls on every one. DigitalOcean built Plano-Orchestrator, a routing model that scored 87.84% accuracy against GPT-5.1's 86.93% and Claude Sonnet 4.5's 86.11%, deciding where a request goes in about 200 milliseconds. Walk through building a router that pools Kimi K3 and Claude, and see how to check with live traffic data whether it's actually working.

Resilient GPU Compute on DigitalOcean Kubernetes: Surviving Spot Interruptions

When DOKS reclaims Spot GPU capacity, it deletes the entire node pool at once, not one node at a time so the usual "autoscaler quietly relaunches it" assumption breaks completely, and nothing automatically brings a reclaimed pool back. Walk through building a pre-provisioned On-Demand fallback pool that costs nothing until a reclaim happens, and run the companion repo's automated interruption simulation to see the failover in action.

GLM-5.3-Flash is the cheapest model on DigitalOcean. It's also the most verbose.

GLM-5.3-Flash lists at one twelfth of Qwen3.8-Max's output price, but on unconfigured defaults, it burned 625 output tokens answering what port SSH uses because it silently defaults to maximum reasoning effort, cutting its real-world price advantage down to just 2.3x. See the measured cost breakdown across 2,700 API calls, and get the one-line reasoning_effort fix that made Flash 6.3x cheaper on the same prompts.

When pgvector Starts to Slow Down as Your Vector Table Grows

A pgvector's query latency doesn't creep up gradually as a table grows. It stays flat for a long time, then jumps hard the moment the HNSW graph outgrows available RAM and Postgres starts reading it from disk instead. Get the six-step method for finding your own breaking point, measuring recall, latency, and index build time at the table sizes you'll actually reach.

Does Context Length Affect Inference Cost Linearly? We Measured Why It Doesn't

Cost calculators price context length linearly, but measured throughput on a single H200 serving Ministral 3 14B fell from 19,089.8 tokens/sec at 2K context to 4,967.2 tokens/sec at 256K: a 3.84x collapse driven by concurrent request capacity dropping from 311 to just 2 as the fixed KV cache pool fills up. See the full cost curve and break-even utilization math showing exactly where a dedicated GPU stops being cheaper than a flat per-token serverless rate.

Product updates

DigitalOcean Kubernetes (DOKS) now supports Spot GPU Droplets

Running Spot GPUs on DOKS automates interruption handling with automatic cordoning, draining, and PodDisruptionBudget enforcement. Cluster autoscaling dynamically scales your GPU capacity based on workload demand. Kubernetes capabilities such as labels, taints, and node affinity allow you to configure on-demand node pools as a fallback when Spot capacity is reclaimed. Plus, reclaim events surface as native Kubernetes events. Try it today