DEV Community: DigitalOcean

AI coding goes from generation to verification

DigitalOcean — Fri, 06 Nov 2026 17:24:06 +0000

ICYMI AI: The latest AI trends, delivered to Dev.to

Tech Talk: Can (and should) AI verify its own generated code?

In the early days of AI-assisted code writing, having AI generate code meant providing suggestions you’d review. Yet emerging research demonstrates that newer AI systems are checking, testing, and improving their own work at scale.

Are you starting to wonder where your role fits in this evolving workflow?

Axiom Math directed swarms of agents for 24 days to produce roughly 43.1 million lines of Lean code, the first fully computer-checked proof of the Classification of Finite Simple Groups— a big step toward finding AI verification useful for evaluating AI-generated code. OpenAI and Anthropic each announced model-found results on the Exponential Time Hypothesis with Lean-verified proofs.

Now, the same coding loop that developers can run daily to write, check, and iterate can sustain weeks of unsupervised work and tackle more complex coding issues.

Production engineering is following the same recipe:

Meta described scaling LLM-generated Just-in-Time tests that catch bugs before code lands, with real work on false positives.
GitHub shares that Copilot Automations now learn from real outcomes.

For developers, the shift comes down to moving away from generating code and toward defining verifiers: tests, type systems, replay harnesses, and acceptance criteria that an agent can iterate against.

Even though Claude wrote 90% of Anthropic’s code in September, the process still required a heavy human hand in deciding what to build, risky code and architecture review, testing outputs, and owning outcomes. Using AI to verify its own generated code is seeing new advancements—but still needs a developer’s knowledge to double-check outputs and build the verification workflows to effectively produce usable results.

Industry news

The open-source engine behind a growing share of AI inference just hit production-ready.

With llm-d 1.0, now a CNCF project backed by Red Hat, IBM, Google Cloud, CoreWeave, and NVIDIA, the contributors cite demonstrations of 5 to 10 times lower costs for context-heavy agentic workloads and add multi-tenant quality of service and zero-downtime upgrades. DigitalOcean’s Salman Paracha, SVP of AI Engineering, shared how DigitalOcean relies on llm-d as foundational distributed inference software for Serverless Inference and Managed Agents, serving trillions of tokens.

Writing more code isn’t the same as shipping more software.

A Harvard working paper analyzing about 300 million engineering events at 718 firms found that after adopting fully agentic coding tools, lines of code rose 30% and pull requests rose 23%, yet resolved issues and epics showed no statistically significant gain. The likely culprit is review: the average time to review a pull request rose 49%, so the extra code piled up at the review stage.

AI assistants are moving from answering questions to using a computer for you.

Claude Duet, now in research preview, runs in its own cloud environment via a browser, integrates with 6,000+ connectors and plugins, and requires approval before payments or other sensitive steps. It joins Grok Bot, Dots, and Muse, and our side-by-side comparison shows how they differ on connectors, data use, and cost.

Customer corner

Higgsfield AI Evaluated 15+ Other Clouds, Then Sent Its Enterprise Traffic to DigitalOcean for Higher Uptime

Video production used to require serious budgets, professional crews, and weeks of post-production. Higgsfield AI is changing that by putting high-quality AI video creation in the hands of social media marketers, content creators, and brands.

To provide professional-grade video to its users for \$15/month, the company needed a reliable, cost-effective AI cloud provider. When neo-clouds couldn’t meet production workloads, Higgsfield AI turned to DigitalOcean and set up infrastructure that reliably supports its users’ video creation.

Community tutorials

DigitalOcean Serverless Inference latency: p50, p95 and p99 TTFT for GLM-5.3, DeepSeek V3.2, Gemma 4 and Llama 4 Maverick (3,456 requests, 48 hours)

We timed four models on DigitalOcean Serverless Inference for 48 hours. Across 3,456 requests, DeepSeek V3.2 started replying in 1.35 seconds at p50 but 13.7 seconds at p99, while GLM-5.3 streamed fastest at 59 tokens per second. Get the p50, p95, and p99 TTFT and tokens-per-second results for all four models, plus the published script and raw data to rerun the test yourself.

Agents in Production, Part 3: Let Your AI Support Agent Send Replies, Without Giving It the Credential

In two live runs covering 335 tickets, an approval rule on send_reply never fired because the agent only ever called action_invoke, and the agent marked 78 drafts as routine, which the support board then refused. Learn how to keep the credential out of the agent with Action Gateway's credential brokering, close the network with an egress allowlist, and enforce the send rule where it actually holds.

How Do I Monitor and Trace AI Agents Running in Production?

We ran a test feeding one bad tool result across multiple sessions. Every tool call succeeded, even though the answer was still wrong, and an automated debugger, given only a strict application log, found the bad call in 0 of 15 tries. See what to record instead: with every tool result and a check on each record, the debugger found it in 15 of 15, and the article includes a capture checklist and decision table you can copy.

Sandboxing LLM-Generated Code Needs a Different Threat Model

The isolation ladder from bare process to microVM was built for adversaries, but across 92 traced agent sessions, the median session that executed code sat idle 94.5% of its duration. Walk through a 1,500-row failure battery across five isolation tiers and a decision framework for which rung your workload belongs on.

Product news

Now Generally Available: Prompt Caching for DigitalOcean Serverless Inference

Cache affinity now makes your hit rate reliable rather than accidental, lifting flagship models to 85–95%+, so the up-to-90% discount on cached input and up to 80% faster time-to-first-token actually show up on your bill. A new dashboard tracks hit rate and savings per model, and prompt_cache_key lets you control how traffic is grouped or isolated.

How do I run untrusted LLM-generated code in an isolated environment?

DigitalOcean — Thu, 05 Nov 2026 18:01:52 +0000

Run untrusted LLM-generated code in a hardware-isolated sandbox with the network locked to an allowlist and only the credentials the task needs. DigitalOcean Managed Agents does this through Harness Runtime, which gives each session its own Firecracker microVM. The hypervisor, not container namespaces, enforces isolation. The service is in public preview.

Key takeaways:

Each session runs in its own microVM, so code in one sandbox cannot reach another session, another customer, or the host.
Outbound network access is open by default, so set an egress allowlist before running anything untrusted.
Treat the session as the unit of isolation: one session per user or workload, with narrowly scoped secrets.

Start with a sandbox that has no agent

If your application generates the code, you don't need a full agent. A sandbox with harness: none is the same microVM without an agent inside.

name: untrusted-run
harness: none
egress: [pypi.org, files.pythonhosted.org]

doctl harness-runtime create --spec ./sandbox.yaml
doctl harness-runtime upload untrusted-run --local-file ./generated.py --workspace-path generated.py
doctl harness-runtime exec untrusted-run -- python3 generated.py

Lock down the network

Without an egress field, a sandbox reaches any host, so generated code can send workspace contents anywhere. Naming one host turns the allowlist on and denies the rest, though the platform adds GitHub, the DigitalOcean registry, and model endpoints.

Scope secrets and tools

Declare keys under secrets, not env. Any process in the sandbox can read an injected value, so issue each key with the narrowest scope the task needs. For third-party tools, Action Gateway brokers credentials at execution time, so the sandbox never holds them. If an agent writes the code, add a permission policy that defaults to ask and denies commands like rm -rf *; enforcement varies by adapter.

Isolate users and clean up

Processes in the same sandbox share files and credentials, so give each user or workload its own session. A harness-less sandbox bills until you pause or remove it, so download results first, then remove the session.

Untrusted LLM-generated code FAQ

How do I give each user's agent its own isolated sandbox at scale?

Start a separate session for each user or workload. Every session gets its own microVM, and sandboxes cannot observe or reach each other. Keep your external user identifier associated with the DigitalOcean session ID.

How do I manage secrets and API keys for autonomous agents running in the cloud?

Declare keys under secrets, and DigitalOcean injects them at session start so values stay out of the spec file and logs. Scope each key narrowly at the provider, because any process in the sandbox can read it. For third-party tools, Action Gateway brokers credentials so the agent never holds the key.

Do sandboxes have internet access by default?

Yes. Add an egress allowlist to restrict outbound traffic to the hosts you name, plus platform defaults, subject to team policy. Sandboxes accept no inbound internet connections.

References & further reading

Learn more about Managed Agents and the public preview launch post.
Environments and Sandboxes explains how an environment maps to a sandbox.
How to run a sandbox covers file transfer and port forwarding.
Learn more about network egress and Action Gateway.
The secure sessions checklist is worth checking directly before you ship.

What metrics matter for LLM inference observability (TTFT, TPOT, ITL)?

DigitalOcean — Fri, 30 Oct 2026 21:42:24 +0000

Track time to first token (TTFT), time per output token (TPOT), and inter-token latency (ITL) together, because each exposes a different aspect of inference effectiveness. TTFT reflects queueing and prompt processing, TPOT summarizes streaming speed, and ITL reveals stutter inside the stream.

Key takeaways:

TTFT points to queueing and prefill, while TPOT and ITL point to decode.
TPOT is an average, and ITL is per token gap, so only ITL exposes jitter.
Alert on percentiles and per-request targets, not averages.

How TTFT, TPOT, and ITL relate

Inference runs in two phases:

Prefill processes the whole prompt and is compute-bound.
Decode generates one token at a time and is memory-bound.

TTFT lands on the prefill side, since it includes queue wait and prompt processing. TPOT and ITL land on decode.

With N output tokens, the math ties them together:

TPOT = (E2E latency - TTFT) / (N - 1)
E2E latency ≈ network overhead + TTFT + (TPOT × N)

On long responses, TPOT dominates total time even when the first token is fast.

When a healthy TPOT still feels slow

ITL measures the gap between two consecutive tokens, so it catches jitter that average TPOT smooths over. Our inference benchmarking guide lists common causes: large prefills monopolizing compute, PagedAttention overhead, and poor hardware topology—such as cross-NUMA traffic.

Read LLM inference observability metrics as a diagnostic

High TTFT, normal TPOT: requests are queueing or prompts are long. Check prefill capacity.
Rising TPOT: decode is memory-bound. Check GPU memory bandwidth and load.
Healthy TPOT, spiky ITL: something is interrupting decode, such as prefill interference.

Alert on percentiles and goodput

Averages hide the slow tail. Track P95 and P99 for each metric, then define goodput: the share of requests that meet every target at once, such as a TTFT ceiling, an ITL ceiling, and an end-to-end budget. Set thresholds per use case.

Check how your tool defines each metric

Definitions vary. NVIDIA's benchmarking docs, for example, treat ITL as the average gap between tokens, also called TPOT, and warn that tools differ on whether TTFT is included. Confirm what each tool calculates before comparing numbers. On the DigitalOcean Inference Engine, Serverless Inference metrics cover TTFT, end-to-end latency, error rates, and per-request cost with token counts. You can derive TPOT from those with the formula shared earlier.

LLM inference observability FAQ

What is disaggregated prefill and decode in LLM inference?

It runs prompt processing (prefill, which is compute-bound) and token generation (decode, which is memory-bound) on separate workers. The KV cache must move between them, and depending on the implementation, that transfer can delay TTFT or the second token’s ITL. On the DigitalOcean Inference Engine, Serverless Inference metrics report TTFT and end-to-end latency, so you can check whether an architecture change affects first-token time.

What is the difference between TPOT and ITL?

TPOT is the average time per token after the first token, calculated across the entire response. ITL, in the stricter sense, is the gap between two specific consecutive tokens, so it exposes stutter that a healthy TPOT hides. You can derive it from TTFT and end-to-end latency with TPOT = (E2E latency - TTFT) / (N - 1).

Why track P95 and P99 instead of averages?

Averages hide outliers. P95 and P99 show what your slowest requests experience, which is what you compare against latency targets and use to define goodput. Serverless Inference metrics on DigitalOcean cover TTFT, end-to-end latency, error rates, and per-request cost with token counts, so you can set those targets against both speed and spend.

References & further reading

Worth bookmarking: DigitalOcean's guide to LLM inference benchmarking, which covers the formulas above plus concurrency sweeps.
Learn more about Serverless Inference metrics and the DigitalOcean Inference Engine.
Worth checking directly before you compare tools: NVIDIA's LLM benchmarking metric definitions

AI tech stacks see increase in open source adoption

DigitalOcean — Tue, 27 Oct 2026 18:05:28 +0000

ICYMI AI: The latest AI trends, delivered to Dev.to

Tech Talk: The AI stack is more modular and multi-modal than ever

With a changing landscape of models, standards, and systems, no two AI tech stacks are alike. This open ecosystem is the emerging standard of architecture, something last week’s Open Intelligence Summit (OIS) highlighted with speakers from NVIDIA, Amplitude, Open Hands, and Nous Research.

According to internal DigitalOcean figures, 88% of our tokens run on open-weight models, a 27% increase from May 2026. This is accompanied by a 50% growth in token volume over the last six months.

It’s clear that loyalty to a single model is fading, and a new layer is emerging to manage churn: routers. Analysis from Vals AI frames routers as a new AI primitive. They choose models based on quality and cost per request, so applications don’t need to revisit model selection every time something new launches. They also provide more opportunities for developers to select models based on use case, rather than being locked into frontier models or a specific provider.

Open standards are doing the work that closed-loop systems and frontier models used to do, and the tooling is adapting:

DeepSeek opened Search and Fetch APIs that return live web content as Markdown for agents and RAG pipelines.
Perplexity open-sourced its agent sandbox.
Anthropic’s OpenTelemetry support means traces from different providers can share the same dashboards.

When models are interchangeable, differentiation moves to the layers you control: evaluation, observability, security, and cost. With the ability to fine-tune multiple layers, developers need concrete plans for AI stack management. Abstract your model calls behind a routing layer. Standardize on OpenTelemetry for agent traces. Build evals you can rerun against any model. And keep a cost dashboard open.

Our new Managed Agents and Agent Learning Services, announced at OIS, paired with our cache-aware Inference Router, allow you to build AI products and own your intelligence.

Industry news

AI adoption looks broad until you check what’s measurably working. a16z’s State of Markets II presentation found that 69% of S&P 500 companies have a live AI deployment and about 30% report a quantified result, but only around 2% track a metric for it, while the top 1% of companies by AI spend outspend the top 10% roughly eightfold. The gap is less about access to tools than about tying AI to a workflow you can measure.

ChatGPT is shifting from walls of text to answers you can interact with. OpenAI’s GPT-6 launch introduced “Intelligent UI,” which lets the model respond with charts, buttons, forms, and quick tools like savings calculators or bill splitters, all inside the conversation. OpenAI also says GPT-6 Instant starts answering web-search questions 44% sooner on average than GPT-5.6 Instant.

Community tutorials

Agents in Production, Part 1: Build an Overnight AI Support Agent using Managed Agents

An unattended support agent ran for 1 hour 48 minutes against a live board, triaged 99 tickets, stopped all 35 sensitive ones for a person, and sent zero replies, because the agent was never given a route to send. Learn how to scope board credentials, restrict egress to a single host, and add deterministic checks to drafts so the agent can't act outside its brief.

How to Safely Let AI Agents Call Production APIs using DigitalOcean Action Gateway

Across 1,300 episodes, 45 to 48 of every 100 runs without a gateway ended with at least one harmful change, and the agent re-sent a refund in 66 of 75 tries. See how idempotency keys, version checks, and approval tiers brought harmful changes to 0 of 100.

What real time-to-first-token looks like on DigitalOcean Serverless Inference

Across 380 calls to six models, five landed between 0.88 and 0.98 seconds median TTFT on short prompts, but DeepSeek V3.2 hit 78.3 and 186.9 seconds on two of 80 calls. Get the measurement method and code repository to run the same audit on your own models and prompt sizes.

Should You Use Spot or On-Demand GPUs for Model Training?

At a 60-minute cadence, a Spot GPU fits single-node jobs and multi-node jobs with elastic restart. Resuming a 98.3 GB checkpoint after a simulated reclaim took 706.5 seconds on MI355X and 436.6 seconds on B300. Walk through the break-even model and decision table, and plug in your own checkpoint cost and reclaim rate.

Deployment Isn't the Bottleneck: Inference Economics Is

The same $3-per-hour GPU can cost about $0.30 or about $3 per million tokens, depending on whether it runs at full or 10 percent utilization. Learn how cold starts, continuous batching, and KV-cache limits set cost per request, and run the arithmetic for your own model.

Video tutorials

Does context length affect inference cost linearly? We measured it

Every AI pricing page treats context length as linear: ten times the tokens, ten times the cost. This video shows what happens when you run one H200 GPU Droplet, serve Mistral 3 14B with vLLM, and run the same test from 2,000-token prompts up to 256,000.

Product updates

Now in Public Preview: DigitalOcean Insights

Insights gives you one place to see what your agents, applications, and infrastructure are actually doing, with metrics, logs, and traces built in and nothing to install or instrument. Follow an agent run from the model call through every tool it used and find exactly what broke, without switching tools or lining up timestamps yourself. It’s free to use during public preview.

Now Available: Inference Router Enhancements

Choosing the right model for every request shouldn’t require constant testing and manual routing. DigitalOcean’s enhanced Inference Router automatically selects models based on task performance, cost, and latency, helping you balance quality and efficiency as new models become available.

Now in Public Preview: Simulations

Simulations, a new capability of DigitalOcean Evaluations, lets you test a change through realistic multi-turn scenarios in an isolated sandbox on Harness Runtime, with the same configuration and Action Gateway tools your agent uses in production. Pull scenarios from Signals, generate them from your agent’s configuration, or describe one in plain language. An LLM judge scores every trajectory, and running a baseline against a candidate shows exactly what improved and what regressed.

Now in Public Preview: Signals

Signals watches production traffic on Managed Agents and Inference Engine and automatically highlights potential problems across user sentiment, tool execution, and infrastructure, with nothing extra to instrument. Flagged sessions become evaluation scenarios in one click. Signals is opt-in, designed to redact PII before processing, and keeps traces within your DigitalOcean Team boundary.

Where can I get free credits to test LLM inference

DigitalOcean — Fri, 23 Oct 2026 20:45:01 +0000

DigitalOcean gives new customers a free \$5 credit, with no credit card required, through its Managed Agents free trial. The credit covers inference on DigitalOcean-hosted models plus the agent runtime and tools around them. It's sized for testing, and Managed Agents is in Public Preview.

Key takeaways:

New DigitalOcean customers get \$5 in free credit, no credit card required, to test inference through Managed Agents.
Trial inference runs on DigitalOcean-hosted models such as Kimi K3, GLM 5.3, and DeepSeek V4 Flash. Frontier models like Claude and GPT bill at list price.
Credit drains as you use compute, storage, and tools, so pause idle sessions and watch your balance.

What the free credit covers

Managed Agents pairs Harness Runtime, which runs agent sessions in isolated Firecracker microVMs, with Action Gateway, which gives agents governed access to 16,000+ tools through one MCP endpoint. It integrates natively with the Inference Engine, so inference tokens, agent execution, and tool use come together in one place.

Refer to the Agent Droplets pricing page for more details about the trial offer, as well as details for paid Pro (\$50 per month) and Team (\$200 per month) plans.

Which models you can test

The trial's inference is built around models DigitalOcean hosts, including Kimi K3 and GLM 5.3. For scale, as of October 2026, the DigitalOcean-hosted GLM-5.3 Flash lists at \$0.15 per million input tokens and \$0.50 per million output tokens on the Inference pricing page. Frontier models like Claude and GPT are also available through DigitalOcean without a provider key of your own, billed at list price and outside the Agent Droplets plan discounts. If you bring your own provider key, that provider bills you directly.

DigitalOcean's Agent Droplets announcement estimates the free trial at roughly 4.5M tokens of inference.

How to start

Sign up as a new customer, open Managed Agents in the control panel, and start the trial. The Harness Runtime docs include a quickstart that launches an agent with one doctl command, and the launch post reports supported harnesses like OpenCode and Codex CLI.

What to watch while you test

Usage draws down your balance as agents run, and Harness Runtime has no per-session or per-product spend limit. Some Action Gateway tools, like Exa web search and web fetch, are prepaid and metered per call. Serverless inference is prepaid too, so access is suspended when credits and balances run out.

To save credits, pause sessions you aren't using. Pausing stops CPU and memory charges, though retained checkpoints still accrue storage fees. Current rates are on the Harness Runtime pricing page.

LLM inference free credits FAQ

Is there an inference provider with a free tier?

DigitalOcean offers new customers a free trial with \$5 in credit through Managed Agents, no credit card required. It's a starter credit for new customers rather than an ongoing free tier, and it covers inference on DigitalOcean-hosted models, runtime, storage, and tools.

Where can I run inference for an AI app and bill it all through one invoice?

Managed Agents integrates with the DigitalOcean Inference Engine, so inference on DigitalOcean-hosted models, agent compute, storage, and tool calls bill together. Models you reach with your own provider key are billed by that provider.

How do I cap inference billing to avoid surprise costs from runaway agents?

Harness Runtime has no per-session or per-product spend limit, and usage draws down your prepaid balance. Pausing a session stops CPU and memory charges, and auto-pause handles idle sessions, though retained checkpoints still accrue storage fees. Monitor usage closely on autonomous workflows.

References & further reading

Start with the Managed Agents product page for trial terms.
The launch post and the Agent Droplets announcement explain the architecture and the trial.
Worth bookmarking: the Agent Droplets pricing page for plan comparisons.
The Harness Runtime pricing page lists compute and storage rates. Worth checking directly before you build.
The Inference pricing docs have per-model token rates and prepaid billing rules.
Learn more about Harness Runtime and the Inference Engine.

Where to get H100 GPUs without long-term a commitment

DigitalOcean — Tue, 20 Oct 2026 15:24:05 +0000

You can get H100 GPUs without a contract from DigitalOcean GPU Droplets. Create one from the console or API, pay per second, and destroy it when the job ends. Commitments only pay off once your usage is steady.

Key takeaways:

On-demand H100 GPU Droplets pricing starts at \$4.41 per GPU per hour (as of October 2026), billed per second, with no term.
A 12-month reserved H100 rate is lower (\$3.26 per GPU per hour), but that is the commitment you are avoiding.
Powering off does not stop billing. Destroying the Droplet does.

What on-demand H100 access looks like

H100 GPU Droplets come with one GPU (80 GB of GPU memory, 20 vCPUs, 240 GiB of RAM) or eight (640 GB of GPU memory, 160 vCPUs, 1,920 GiB of RAM), at \$35.28 per hour for the eight-GPU size. They are available in New York (NYC2), Amsterdam (AMS3), and Toronto (TOR1). Billing is per second with a 60-second or \$0.01 minimum, whichever is higher.

When on-demand beats reserved

A 10-hour fine-tuning run on one H100 costs about \$44.10. A reserved plan only wins if you would keep that GPU busy roughly 74% of the year if you pay for the reserved GPU for the full 12 months (\$3.26 divided by \$4.41 is the break-even utilization). Prototypes, experiments, and bursty inference rarely clear that bar.

Keep costs predictable

The most common surprise is an idle GPU. Billing continues while a GPU Droplet is powered off because the hardware stays reserved, so destroy it when you finish. Set spend alerts to get an email when monthly spend crosses percentage thresholds of a monthly budget you set. Alerts notify you but don't cap spending.

If H100 capacity or budget is tight

The same on-demand catalog includes NVIDIA H200 at \$4.47 per GPU per hour and AMD Instinct MI300X at \$2.59. Spot GPU Droplets are cheaper but interruptible, with at least two hours' notice before reclaim (less in emergencies), and are currently listed for B300, MI350X, and MI355X rather than H100. If you only need model output rather than a server, the Inference Engine lets you pay for what you use.

H100 GPU FAQ

Which provider bills GPU usage by the second with no long-term commitment?

DigitalOcean GPU Droplets bill per second, with a minimum charge of 60 seconds or \$0.01, whichever is higher. On-demand plans have no contract, and billing runs from creation until you destroy the Droplet.

Do I still pay for a powered-off H100 GPU Droplet?

Yes. The GPU and other compute resources stay reserved while the Droplet is off, so billing continues until you destroy it.

Which AI platform supports both AMD and NVIDIA GPUs?

DigitalOcean GPU Droplets offer NVIDIA H100 and H200 alongside AMD Instinct MI300X on demand. You can create any of them from the same console or API.

References & further reading

Learn more about GPU Droplets pricing and reserved plans, and check it directly before you build, since rates change.
The Droplet pricing docs spell out per-second billing and the powered-off rule.
Worth bookmarking: GPU availability by region for which datacenters carry H100.
How to create GPU Droplets walks through image, plan, and datacenter choices.
Learn more about Spot vs on-demand GPU Droplets if interruptible capacity fits your workload.
For dedicated hardware instead of a virtual machine, see Bare Metal GPUs.

How do I cap inference billing to avoid surprise costs from idle GPUs or runaway agents?

DigitalOcean — Wed, 14 Oct 2026 16:31:01 +0000

You cap inference billing by layering controls, not by flipping one switch. On DigitalOcean, Serverless Inference is prepaid, so your balance is a real ceiling, and scoped keys, alerts, and cleanup jobs cover the rest. Spend alerts alone won't stop anything: they email you when spending crosses a threshold, but your usage keeps running.

Key takeaways:

Serverless Inference stops when funds run out, so fund a dedicated balance with only what you plan to spend.
Dedicated Inference and GPU Droplets bill by the hour until destroyed, so automate cleanup.
Agents need caps in your own code, because rate limits slow spending but don't cap it.

Use prepaid balances as a ceiling

Serverless usage draws from promotional credits, then the Inference and Agents balance, then your account prepayment balance. Only Serverless Inference and Managed Agents draw from the first, so it works as a separate AI budget. Auto-reload is on by default, so turn it off for a hard stop. A zero balance is also an outage, so reserve this for experiments and non-critical workloads.

Add alerts and scope your keys

Spend alerts email you at percentage thresholds of your monthly budget, within an hour of each threshold being crossed. Their sole purpose is notification. Give each service its own model access key, scoped to specific models and a VPC, so one leak or loop has a small blast radius. Regenerating a key takes effect immediately, which is how you want it to work during an incident.

Cap agents in code

Rate limits (up to 120 requests and 1.5M tokens per minute at the lowest serverless tier) limit speed, not total spend. Harness Runtime has no per-session or per-product spend limit, so an agent stuck in an unintended loop can quickly drain balances. Set a max output token count per call, a max step count per task, and a running spend counter that stops the run. You should also pause idle sessions, since paused sessions incur no compute charges.

Stop paying for idle GPUs

Dedicated Inference bills per GPU-hour, and GPU Droplets bill per second until destroyed. Powering off doesn't stop the meter. At \$30.32 per hour for 8x H100, an idle Dedicated Inference endpoint costs roughly \$728 a day.

To cap inference billing and avoid surprise costs, tag experimental GPU Droplets and destroy them nightly:

doctl compute droplet delete --tag-name ephemeral-gpu --force

Inference billing cap FAQ

Can I get usage alerts or hard spending caps on a managed AI platform before bills run away?

DigitalOcean spend alerts notify you at budget thresholds but don't limit usage. The closest hard cap is Serverless Inference's prepaid model, where access is suspended when credits and balances run out.

Does powering off a GPU Droplet stop billing?

No. The GPU stays reserved, so billing continues until you destroy the Droplet.

Which inference option doesn't penalize variable workloads?

Serverless Inference, which bills per token instead of reserving GPUs, is a solid choice for variable workloads. Dedicated Inference bills per GPU-hour, whether or not anyone calls it, so it suits steady traffic better than spiky traffic.

References & further reading

Spend alerts: worth setting up first.
Manage Serverless Inference prepayment: explains the order balances are drawn down.
Learn more about model access keys and inference limits.
Inference pricing: worth checking directly before you build.
Droplet pricing and billing
Harness Runtime pricing: worth bookmarking before you run agents at scale.

We moved a production backend from a phone

Vinay K — Tue, 13 Oct 2026 13:19:33 +0000

Published by Vinay Kumar, Chief Product and Technology Officer at DigitalOcean

Every era of the cloud has a unit of work. For agents, it’s a human who directs, a controller who runs the operation, and a worker who does the heavy lifting. Last week we watched all three move a production backend, from a phone.

I’ve spent fifteen years building cloud services: S3 and EBS in the early days of AWS, Oracle Cloud Infrastructure from inception, and now the agentic cloud at DigitalOcean. In every era, moving a backend meant the same thing: a project plan, a maintenance window, and a team working late into the night.

Last week, Rohit Bhattacharjee on our team moved an entire production backend from his phone. Database, APIs, file storage, and about 31,000 AI search embeddings. He never opened a laptop. A message while waiting for coffee. A decision between meetings. By the end of the week, it was done.

This wasn’t a demo. It was a real workload with a bill.

Three parties, three jobs

The worker was a DigitalOcean Managed Agent, an AI that does real work in the cloud, not just chat. It ran in our Harness Runtime: a private, isolated computer that spins up for each session, with a workspace that persists when the agent pauses. Through our Action Gateway, a single connection to 20,000+ tools, it could use GitHub, Supabase, a web browser, and the whole DigitalOcean API.

The agent never saw a password. The gateway applies credentials at the moment of use, so they never reach the model.

It provisioned the database, loaded all 40 access policies, and mapped every Supabase feature to its DigitalOcean counterpart. It wasn’t advising on a migration. It was performing one.

The controller was a personal agent running on his phone, the agent Rohit actually talked to. He ran the migration in plain chat, and the personal agent turned his messages into work: tasking the worker, tracking the run, and reporting back. When he asked it to check a claim, it did. The worker said 145,432 rows had been loaded, so the controller counted them. Validation was the side job. The main job was holding the whole operation together in one conversation, so a person never had to leave the flow of his day.

The worker and controller come from different model families. That’s the point of an open platform: you choose the right model for each job.

Rohit provided the credentials, his judgment, and a credit card. He reviewed a $180 proposed plan and saved almost 80% on a revision with just a few taps. Everything else, the two agents handled between themselves.

What moved

The workload was huffl.ai, an agentic canvas that puts docs, spreadsheets, whiteboard, presentations, and search in one place. Underneath: 31 tables, 145,432 rows, three storage buckets, 24 functions, and 30,650 embeddings, the numeric fingerprints that make search work.

Before moving anything, the worker read all of it and caught two problems that would have poisoned the migration. Two overloaded functions that looked like duplicates. A “vector count” that was really measuring string lengths.

The new home: managed Postgres ($15), one App Platform app running four services ($20), and one object store ($5). About $40 a month. The data moved with per-table parity checks, and every one of the 30,650 embeddings verified non-null.

Pay for the work, not the wait

An agent spends most of its session waiting: on a model, a tool, or a person. The runtime paused the worker whenever it waited and resumed it in about 300 milliseconds. While it slept, the meter stopped. We bill for active CPU, not provisioned capacity. The agent runtime for the entire run cost a few dollars.

What this changes

Moving a backend used to mean a team, a plan, a maintenance window, and a war room. Last week it meant chat bubbles. The human says what needs to happen, the controller runs the operation, the worker does the work, and every claim gets checked along the way.

The work didn’t get smaller. What one person can accomplish got bigger. See it in action: watch the video:

Try DigitalOcean Managed Agents today. Bring your preferred harness, connect your tools, and give your agents a complete environment with our Cuttlefish mobile application coming soon. Get started today.

What would you trust an agent to move for you, and what would you want checking its work?

Jev: an AI model that can’t write a sentence

DigitalOcean — Fri, 09 Oct 2026 14:00:00 +0000

ICYMI AI: The latest AI trends, delivered to you on Dev.to

Tech Talk: TypeSafe’s Jev is a model that only returns probabilities

TypeSafe's Jev, released into early access on September 15, is a general-purpose classification model. It reads up to 32K tokens of input (such as a support ticket, a customer record, or a row of data). It answers any number of typed questions about it in a single parallel pass, and returns probabilities rather than prose.

Outputs are limited to three options: Choice, Score, and Noul (yes or no), so structured-output errors and bad tool calls drop to zero by construction. A valid answer isn't always the right one, and Jev doesn't claim otherwise.

What it does is tell you how sure it is. TypeSafe trains it for calibration, so its probabilities match real-world frequencies. When Jev says it's 80% confident, it's right about 80% of the time. That lets you set a threshold, automatically acting on high-confidence calls and routing the rest to a human or a larger model.

The pricing is what makes Jev worth a look. At \$0.042 per million input tokens with free output, TypeSafe estimates about \$0.0004 per decision, roughly 1/76th the cost of GPT-5.6 Terra on its benchmark. Within days, imitators appeared: Laya, Bespoke Nimble, and Kev, several of them open-weight fine-tunes of Qwen 3.5 you can run locally. OpenAI likewise followed with a Decisions API of its own.

For developers, the change is in how you structure the call. Instead of sending an LLM one broad question like "is this spam?" and parsing a paragraph back, you pass Jev the email once, along with a set of narrower typed questions, such as: Does the sender domain match the claimed company? Does it ask for credentials? How urgent is the language? Each comes back as a probability. Your own code then applies the thresholds and makes the call, so the routing policy lives in your repo, versioned and auditable, and the model only scores the inputs.

Jev is available today on Serverless Inference through the DigitalOcean Model Library, at Typesafe’s list price—no separate TypeSafe account needed. Pair it with the DigitalOcean Inference Router to send routing and classification traffic to Jev while keeping frontier models in reserve for the calls that need them.

Event calendar

Open Intelligence Summit

October 12-13th, San Francisco

The argument is no longer “Is open source AI cheaper?” It’s now “Who owns intelligence?” A movement is forming around open intelligence and generating the belief that the intelligence powering our AI products and companies should be more open, portable, and controllable.

Join DigitalOcean, NVIDIA, RadixArk, Inferact, and more for deep dives, demos, and discussions about the technologies that keep intelligence portable, interoperable, and in the hands of builders.

Industry news

Near-flagship intelligence at a fifth of the price was the headline at OpenAI DevDay, but the bigger shift is the stack forming around the model.

OpenAI announced GPT-6.1 Sol, which it says delivers near-Astra intelligence at a fifth of Astra's token prices, plus an Ultrafast tier with up to 6x faster generation in the API. It also added computer use to its Agents API and opened ChatGPT, which it puts at 1.2B weekly users, to developer-built plugins and partners like Devin, Notion, and Vercel.

Agent safety is moving below the application layer.

NVIDIA's Open Agent Safety Platform pairs OpenShell, an open-source runtime that traces agent actions and enforces policy, with Sentry, an out-of-band watchdog on BlueField-4 DPUs that NVIDIA says can quarantine an agent that leaves its boundaries within milliseconds. NVIDIA says more than 100 organizations, including Anthropic and CrowdStrike, are working with the technology, and DigitalOcean is among the companies supporting the platform.

New models available on DigitalOcean Inference Engine

We recently launched models from TypeSafeAI, DeepSeek, Z.ai, NVIDIA, OpenAI, and Anthropic, all with different strengths and use cases. Learn more about the models in our What’s New blog.

Start using them through the DigitalOcean API or directly in the Cloud Console.

DeepSeek-V4.1-Flash

DeepSeek-V4.1-Flash cuts the KV cache footprint to 1/4 of V4-Flash while outperforming DeepSeek's own V4-Pro flagship on coding and agentic benchmarks, addressing the cache-hit charges that often dominate agent costs. The 552B MoE model activates just 8B parameters on input, delivering frontier performance with infrastructure economics that improve as you scale.

GLM-5.3 and GLM-5.3-Flash from Z.ai

Z.ai's post-training translates to significant performance improvements for GLM-5.3, achieving a 50% lower cost per completed coding task than GLM-5.2 while jumping to 28.3 on Terminal-Bench 3.0. GLM-5.3-Flash adds native image and video input with the same agentic strength, giving teams both reasoning-heavy and efficient multimodal paths on a single platform.

Nemotron™ 3 Diarization from NVIDIA

NVIDIA's Nemotron™ 3 Diarization replaces the multi-stage speaker attribution pipeline (VAD, embeddings, clustering, post-processing) with a single 100M-parameter model, keeping audio within DigitalOcean's infrastructure rather than routing it to third-party APIs. It identifies up to 8 speakers (double the previous limit) with a 14.72% error rate.

GPT-6.1 Sol, GPT-6 Astra, Sol, and Luna from OpenAI

One week after GPT-6 Sol’s launch, OpenAI releases GPT-6.1 Sol, which offers near-Astra-level coding and substantial improvements in document understanding, computer use, and scientific work. OpenAI's three-tier GPT-6 family also debuts on DigitalOcean: Astra for computer use and coding, Sol for frontier reasoning at 90% lower cost than Opus 5, and Luna for high-volume tasks.

Claude Fable 5.1, Opus 5.5, and Sonnet 5.5 from Anthropic

Anthropic's latest point releases deliver focused gains: Fable 5.1 reaches 55.8% on Terminal-Bench 4.0 with 75% cheaper cache reads, while Opus 5.5 matches high-effort results at default effort using half the tokens and 20% lower per-token cost. Sonnet 5.5 runs 30% faster with equivalent cost savings, with a shift toward better token efficiency and lower verification burden.

Community tutorials

How naive failover between inference providers makes outages worse

In a six-hour fault-injection run on an H200 GPU Droplet, the standard retry-with-backoff policy pushed 3.54x normal load onto a degraded provider, while a retry budget plus circuit breaker held it to 1.02x, and naive retries left 3,743 zombie requests generating 959,232 output tokens for clients that had already disconnected. Learn how to classify inference errors before failing over, replace fixed retry counts with a budget, and verify whether your provider actually stops billing when you cancel a stream.

What vision-language models cost to serve at production scale

A single uploaded photo can add more than a thousand input tokens before the model writes a word, and at 500,000 requests a month, that gap is the difference between \$155 on serverless Nemotron Nano 12B VL and \$3,175.20 for a dedicated H100 running 24/7. Walk through the per-request cost math, the roughly 10.2 million-request breakeven, where dedicated wins, and six levers for cutting image-token spend.

Speculative decoding's hidden cost: the KV cache capacity you give up, and where throughput turns negative

On a 141 GB H200 serving Qwen3-14B with vLLM, a separate Qwen3-0.6B draft model consumed about 43.8 percent of baseline batch slots, and even n-gram speculation, predicted to cost nothing, measured a 2.8 percent capacity tax. See where each method's throughput crosses below baseline as concurrency rises, and get the config.json arithmetic to predict a draft model's KV cost before spending GPU hours.

Cache reliability is a pricing term: what I learned measuring DeepSeek-V4.1-Flash on DigitalOcean Serverless Inference

DeepSeek-V4.1-Flash has the deepest cache discount in the Serverless Inference catalog at \$0.006 per million cached tokens, yet across 844 API calls it hit the cache on only 78.6% of warm turns, and with a miss priced at 50x a hit, it cost 2.2x what GLM-5.3-Flash did for the same 15-turn, 200K-token sessions. Discover how DigitalOcean's 64-token-block prefix cache works, how to log your own hit rate from the usage object, and why reasoning_effort: max cost up to 1.88x more without improving accuracy.

Why DeepSeek-V4.1-Flash splits reading from writing

DeepSeek-V4.1-Flash's Causal Encoder-Decoder splits a 40-layer model into a 20-layer encoder that runs prefill at 8B active parameters and a full stack that decodes at 16B, shrinking the global KV cache to about 890 bytes per token, roughly a quarter of V4-Flash's. Learn why that asymmetry, not the 74.2 versus 62.7 DeepSWE score against V4-Pro, is what lowers the bill for read-heavy coding agents.

Product updates

Introducing DigitalOcean Managed Agents, now in Public Preview

AI agents are ready to take on more, and we built the infrastructure to match. DigitalOcean Managed Agents brings together Harness Runtime, a purpose-built runtime for running agent sessions at scale, and Action Gateway, governed access to the tools agents need to get real work done, all in one platform.

Deploy your preferred agent harness, Claude Code, Codex CLI, OpenCode, and more, or bring your own, and connect to 16,000+ tools across 500+ providers through a single managed endpoint. Sessions are ready in under a second, resume in about 300 milliseconds with full context intact, and you only pay for the compute you actually use.

Whether you're automating support workflows, refactoring code across repositories, or building the next agent-powered product, Managed Agents gives you the infrastructure to do it, without building it yourself.

DigitalOcean MicroVMs now in private preview

Isolated, Firecracker-based VMs that pause when idle and resume with state intact, built for agent workloads that run in bursts. The same layer that powers Managed Agents is now available directly to teams building their own agent platforms, sandboxes, and short-lived jobs.

What is disaggregated prefill and decode in LLM inference?

DigitalOcean — Tue, 06 Oct 2026 18:03:00 +0000

What is disaggregated prefill and decode in LLM inference?

Prefill is compute-bound, decode is memory-bound. Learn why colocating them hurts TTFT and ITL, and when disaggregated serving pays off.

Every LLM request runs through two phases with effectively opposite hardware profiles. Understanding the split explains most of the latency and cost behavior you see in production, and why disaggregated prefill and decode has become a standard serving pattern.

Key takeaways:

Prefill is compute-bound and sets TTFT; decode is memory-bandwidth-bound and sets ITL/TPOT.
Running both on the same GPUs creates prefill/decode interference, visible as TTFT spikes and ITL jitter.
Chunked prefill and continuous batching reduce interference; disaggregating the phases onto separate pools removes it, at the cost of a KV cache transfer.
Disaggregation is worth it for long prompts, high concurrency, and strict streaming SLOs. A managed option like DigitalOcean Serverless Inference gives you it without the setup; for small self-hosted deployments, a well-tuned single pool on a GPU Droplet running vLLM is often enough.

The two phases of LLM inference

Prefill processes the entire prompt in one forward pass and builds the KV cache, the attention state every later token reads from. It is compute-bound: arithmetic intensity is high and the GPU spends its time doing math, not waiting on memory. Prefill governs time to first token (TTFT), and its attention cost grows quadratically with prompt length.

Decode generates output one token at a time. Each step reloads the model weights and the growing KV cache from HBM to produce a single token, so it is memory-bandwidth-bound. Decode governs inter-token latency (ITL) and time per output token (TPOT). A GPU with higher memory bandwidth streams tokens faster even if its FLOPs are unchanged.

Why colocating prefill and decode causes latency problems

Most serving engines batch prefill and decode on the same GPUs. That is where the interference shows up: a long prompt's prefill can monopolize compute while dozens of in-flight decode requests stall, and users see ITL jitter or a TTFT spike. The two phases also want different batch sizes and parallelism strategies, but colocated they must share one configuration.

Chunked prefill and continuous batching soften this by interleaving prompt chunks with decode steps, which is why our inference trilemma guide recommends them for latency-sensitive deployments.

What disaggregation changes

Disaggregated serving, formalized in the DistServe paper, runs prefill and decode on separate GPU pools. A prefill worker computes the KV cache, ships it over a fast interconnect (RDMA, RoCE, or a transfer library like NIXL or LMCache) to a decode worker, and the decode pool streams tokens without prefill interference. Because each pool is sized and tuned independently, you can hit TTFT and ITL targets at the same time rather than trading one for the other. The cost is added complexity plus a KV transfer step; that transfer can land on second-token latency rather than TTFT depending on how orchestration is done.

This is the architecture DigitalOcean already runs in production: Serverless Inference uses disaggregated serving over a RoCE network, paired with prefix-aware routing so shared system prompts skip redundant prefill entirely.

References & further reading

DistServe: Disaggregating Prefill and Decoding for Goodput-optimized LLM Serving: the primary source for the pattern, worth reading before you build.
LLM inference benchmarking: measure what matters: a clear walkthrough of TTFT, TPOT, ITL, and how each maps to prefill or decode.
The LLM inference trilemma: practical tuning advice, including when to reach for chunked prefill.
Learn more about debugging p99 TTFT and deploying NVIDIA Dynamo on GPU Droplets, a framework built around disaggregated serving.
GPU Droplets: current hardware and regions, worth checking directly since availability changes.

How to route LLM requests by cost vs. latency

DigitalOcean — Fri, 02 Oct 2026 18:33:17 +0000

Routing LLM requests by cost and latency means sending each request to the cheapest or fastest model that still meets the quality bar, rather than hardcoding a single model for everything. Production traffic isn't uniform: routine lookups, complex troubleshooting, and background jobs have different cost and latency requirements, even within a single app.

Using an inference router like DigitalOcean Inference Router involves defining request categories, assigning each a selection policy (cost, latency, or a benchmarked best fit), and setting fallback models so an unavailable model doesn't break the request. Cost is the price per token (which can vary by up to 100x across a model catalog); latency is mainly the time to first token (TTFT). The two don't move together, so route per task rather than using a single blended score.

Key takeaways:

Routing is a per-workload policy, not a search for "the best model."
Cost and latency usually need separate, explicit priorities.
Fallback and cache-aware routing protect the savings that routing is meant to deliver.
The DigitalOcean Inference Router implements this as configuration, not custom infrastructure.

LLM routing strategies

Static rules: Hardcode model per request type. Simple, but brittle.
Policy-based routing: A model pool per task with a cost/latency/optimal policy. Most production systems land here.
Dynamic routing: A classifier infers the task and policy automatically.

Routing LLM requests: A quick decision framework

Workload	Priority	Policy
Real-time chat	Latency	Speed-optimized (TTFT)
Routine lookups	Cost	Cost-optimized
Complex reasoning	Quality	Benchmarked "optimal"
Batch/background	Cost	Cost-optimized

Fallback models keep requests completing when the preferred model is down or rate-limited.

Cache-aware routing matters too: switching models to save a fraction of a cent can break a cached prompt and cost more overall.

How the DigitalOcean Inference Router routes LLM requests

Inference Router tasks pair a model pool with a selection policy: preset tasks default to a benchmarked Optimal policy; custom tasks choose Cost Efficiency, Speed Optimization (TTFT), or Manual Ranking.

Fallback models handle unmatched traffic. X-Model-Affinity preserves cache reuse, and X-Routing-Max-Switch-Spend-Pct (default 20%) caps the cost of switching mid-session.

It's a drop-in change. Set "model": "router:your-router-name". Routing decisions typically resolve in about 200ms, billed at the serving model's standard rate with no separate router charge during public preview.

LLM routing FAQ

What is LLM routing?

LLM routing is the practice of directing each request to the model best suited to it—by cost, latency, or task fit—instead of sending every request through one model regardless of what it needs. The DigitalOcean Inference Router implements this as a managed feature, so teams get task-aware routing and fallback handling without building it themselves.

How do you model cost-per-token for LLM inference?

Multiply input tokens by the model's input rate and output tokens by its output rate, then sum the two, since the rates are usually very different. Because rates vary widely across a model catalog, this is best tracked per task rather than as one blended number. The DigitalOcean Inference Router reports token usage and cost-relevant metrics per model and per task in its Analyze view.

What metrics matter for LLM inference observability?

Time to first token (TTFT), time per output token (TPOT), and inter-token latency (ITL) are the core metrics for judging responsiveness. A latency-optimized routing policy should be measured against these, not just overall request time. The DigitalOcean Inference Router uses TTFT specifically as the basis for its Speed Optimization selection policy.

What causes cold start latency in GPU inference, and how do you avoid it?

Cold starts happen when a model has to load onto available GPU capacity before it can serve a request, rather than running on an already-warm instance. Pooled serverless capacity and routing policies that keep related traffic on one model reduce how often this happens. The DigitalOcean Inference Router uses model affinity to keep a session's requests on the same warm model, rather than triggering repeated cold starts across models.

Can I route by both cost and latency at the same time?

Yes—typically by defining separate tasks or policies for different request types rather than one blended score. For example, cost-first for background work and latency-first for real-time chat. The DigitalOcean Inference Router supports this directly. Each task in a router can have its own model pool and its own Cost Efficiency, Speed Optimization, or Manual Ranking policy.

References & further reading

How to use Inference Router: The full documentation on tasks, selection policies, fallback models, and the cache-aware router referenced throughout this piece. Worth reading directly before you configure a router for production traffic.
Inference pricing: Current per-model Serverless Inference rates, since routing decisions are only as good as the pricing data behind them.
How DigitalOcean built Inference Router: An inside look at the architecture decisions behind cache-aware routing and fallback handling, useful if you want the reasoning behind the mechanics.
DigitalOcean Serverless Inference: a deep dive: Background on the serverless layer that Inference Router sits in front of.
DigitalOcean's best LLM routers roundup: A good next read if you're comparing routing options across vendors rather than implementing a policy on a platform you've already chosen.
AI Inference Engine overview: shows where Inference Router fits alongside serverless, batch, and dedicated inference on DigitalOcean's broader platform.

Which inference provider has both serverless and dedicated GPU tiers on the same platform with one API?

DigitalOcean — Mon, 28 Sep 2026 17:17:40 +0000

DigitalOcean's Inference Engine puts serverless, dedicated, and batch inference behind a single endpoint, on the same GPU Droplets infrastructure, so a workload can move from pay-per-token prototyping to a reserved-GPU deployment without switching platforms or rewriting its integration. A handful of other inference providers share a single API across serverless and dedicated tiers, too, but they generally limit that unification to inference alone, rather than to the broader cloud the dedicated GPUs run on.

Why serverless-only or dedicated-only platforms create a migration tax

Most teams don't start with a clear answer to "How much GPU capacity do we need?". They prototype on a pay-per-token endpoint, and only once traffic gets steady and predictable does it make financial sense to reserve GPUs outright. Many inference providers treat serverless and dedicated as separate products, with distinct SDKs, auth, and billing dashboards, so moving from one to the other means rewriting integration code in the middle of a product's growth curve, which is exactly when a team has the least time to do it. A provider that puts both tiers behind a single API avoids that: you prototype on serverless and repoint the same client to a dedicated endpoint once volume justifies it, often by changing a model or endpoint parameter rather than an integration.

How the DigitalOcean Inference Engine unifies serverless, dedicated, and batch inference

The DigitalOcean Inference Engine runs three inference patterns over the same GPU Droplets infrastructure, behind a single endpoint. These solutions include real-time Serverless inference across 70+ open-weight and frontier models, Dedicated inference on reserved, single-tenant GPU capacity with support for bringing your own model, and asynchronous Batch Inference for large, non-real-time jobs. The Preview Inference Router adds cost- and latency-based routing on top, and a shared Model Playground and evaluations tooling sit on the same account.
That's what makes DigitalOcean a solid solution for AI-native companies requiring both serverless and dedicated GPU tiers. The unification isn't limited to the inference API itself. It extends to the GPU Droplets the dedicated tier runs on, as well as the rest of the AI-Native Cloud around them. A workload doesn't need to switch vendors, contracts, or billing relationships as it matures from prototype to production.

What to check before assuming "one API" elsewhere means the same thing

Several other inference-focused platforms let serverless and dedicated calls share one API too, usually as a way past per-token rate limits or shared-capacity contention once traffic justifies reserving GPUs. But a shared API can sound more unified on a landing page than it is in production. So it's worth confirming directly against each provider's current docs: does provisioning real dedicated capacity happen through the API, or does it route through a sales conversation first? Is the dedicated tier's hardware genuinely part of the same cloud a team already runs on, or does the provider specialize in inference alone, with GPUs sitting apart from the application's data and storage? And does "one API" cover just the inference call—or the account, billing, and observability around it too?

What else to verify before standardizing on a provider

Beyond scope, it's worth confirming: whether the model you need is available in both tiers (not every model ships to both), what the cutover mechanism actually looks like (a new endpoint URL versus a full redeploy), how billing reconciles if usage straddles both tiers in the same month, and whether dedicated capacity is truly single-tenant or a reserved slice of shared hardware. For teams already running other infrastructure on a given cloud, it's also worth weighing how much value comes from the inference API alone versus not having to move data, agents, or storage to a second vendor to get dedicated GPU access.

References & further reading

DigitalOcean's Inference Engine brings serverless, dedicated, and batch inference together in one place.
Our Dedicated Inference documentation covers bring-your-own-model (BYOM) support and scaling settings for teams ready to move off serverless.
Read our Serverless Inference documentation for the model catalog and routing options available on the shared endpoint.
Reading through each inference provider's own docs tends to be the fastest way to confirm whether "serverless and dedicated" really means one API or two separate products.