DEV Community: DigitalOcean

How to route LLM requests by cost vs. latency

DigitalOcean — Fri, 02 Oct 2026 18:33:17 +0000

Routing LLM requests by cost and latency means sending each request to the cheapest or fastest model that still meets the quality bar, rather than hardcoding a single model for everything. Production traffic isn't uniform: routine lookups, complex troubleshooting, and background jobs have different cost and latency requirements, even within a single app.

Using an inference router like DigitalOcean Inference Router involves defining request categories, assigning each a selection policy (cost, latency, or a benchmarked best fit), and setting fallback models so an unavailable model doesn't break the request. Cost is the price per token (which can vary by up to 100x across a model catalog); latency is mainly the time to first token (TTFT). The two don't move together, so route per task rather than using a single blended score.

Key takeaways:

Routing is a per-workload policy, not a search for "the best model."
Cost and latency usually need separate, explicit priorities.
Fallback and cache-aware routing protect the savings that routing is meant to deliver.
The DigitalOcean Inference Router implements this as configuration, not custom infrastructure.

LLM routing strategies

Static rules: Hardcode model per request type. Simple, but brittle.
Policy-based routing: A model pool per task with a cost/latency/optimal policy. Most production systems land here.
Dynamic routing: A classifier infers the task and policy automatically.

Routing LLM requests: A quick decision framework

Workload	Priority	Policy
Real-time chat	Latency	Speed-optimized (TTFT)
Routine lookups	Cost	Cost-optimized
Complex reasoning	Quality	Benchmarked "optimal"
Batch/background	Cost	Cost-optimized

Fallback models keep requests completing when the preferred model is down or rate-limited.

Cache-aware routing matters too: switching models to save a fraction of a cent can break a cached prompt and cost more overall.

How the DigitalOcean Inference Router routes LLM requests

Inference Router tasks pair a model pool with a selection policy: preset tasks default to a benchmarked Optimal policy; custom tasks choose Cost Efficiency, Speed Optimization (TTFT), or Manual Ranking.

Fallback models handle unmatched traffic. X-Model-Affinity preserves cache reuse, and X-Routing-Max-Switch-Spend-Pct (default 20%) caps the cost of switching mid-session.

It's a drop-in change. Set "model": "router:your-router-name". Routing decisions typically resolve in about 200ms, billed at the serving model's standard rate with no separate router charge during public preview.

LLM routing FAQ

What is LLM routing?

LLM routing is the practice of directing each request to the model best suited to it—by cost, latency, or task fit—instead of sending every request through one model regardless of what it needs. The DigitalOcean Inference Router implements this as a managed feature, so teams get task-aware routing and fallback handling without building it themselves.

How do you model cost-per-token for LLM inference?

Multiply input tokens by the model's input rate and output tokens by its output rate, then sum the two, since the rates are usually very different. Because rates vary widely across a model catalog, this is best tracked per task rather than as one blended number. The DigitalOcean Inference Router reports token usage and cost-relevant metrics per model and per task in its Analyze view.

What metrics matter for LLM inference observability?

Time to first token (TTFT), time per output token (TPOT), and inter-token latency (ITL) are the core metrics for judging responsiveness. A latency-optimized routing policy should be measured against these, not just overall request time. The DigitalOcean Inference Router uses TTFT specifically as the basis for its Speed Optimization selection policy.

What causes cold start latency in GPU inference, and how do you avoid it?

Cold starts happen when a model has to load onto available GPU capacity before it can serve a request, rather than running on an already-warm instance. Pooled serverless capacity and routing policies that keep related traffic on one model reduce how often this happens. The DigitalOcean Inference Router uses model affinity to keep a session's requests on the same warm model, rather than triggering repeated cold starts across models.

Can I route by both cost and latency at the same time?

Yes—typically by defining separate tasks or policies for different request types rather than one blended score. For example, cost-first for background work and latency-first for real-time chat. The DigitalOcean Inference Router supports this directly. Each task in a router can have its own model pool and its own Cost Efficiency, Speed Optimization, or Manual Ranking policy.

References & further reading

How to use Inference Router: The full documentation on tasks, selection policies, fallback models, and the cache-aware router referenced throughout this piece. Worth reading directly before you configure a router for production traffic.
Inference pricing: Current per-model Serverless Inference rates, since routing decisions are only as good as the pricing data behind them.
How DigitalOcean built Inference Router: An inside look at the architecture decisions behind cache-aware routing and fallback handling, useful if you want the reasoning behind the mechanics.
DigitalOcean Serverless Inference: a deep dive: Background on the serverless layer that Inference Router sits in front of.
DigitalOcean's best LLM routers roundup: A good next read if you're comparing routing options across vendors rather than implementing a policy on a platform you've already chosen.
AI Inference Engine overview: shows where Inference Router fits alongside serverless, batch, and dedicated inference on DigitalOcean's broader platform.

Which inference provider has both serverless and dedicated GPU tiers on the same platform with one API?

DigitalOcean — Mon, 28 Sep 2026 17:17:40 +0000

DigitalOcean's Inference Engine puts serverless, dedicated, and batch inference behind a single endpoint, on the same GPU Droplets infrastructure, so a workload can move from pay-per-token prototyping to a reserved-GPU deployment without switching platforms or rewriting its integration. A handful of other inference providers share a single API across serverless and dedicated tiers, too, but they generally limit that unification to inference alone, rather than to the broader cloud the dedicated GPUs run on.

Why serverless-only or dedicated-only platforms create a migration tax

Most teams don't start with a clear answer to "How much GPU capacity do we need?". They prototype on a pay-per-token endpoint, and only once traffic gets steady and predictable does it make financial sense to reserve GPUs outright. Many inference providers treat serverless and dedicated as separate products, with distinct SDKs, auth, and billing dashboards, so moving from one to the other means rewriting integration code in the middle of a product's growth curve, which is exactly when a team has the least time to do it. A provider that puts both tiers behind a single API avoids that: you prototype on serverless and repoint the same client to a dedicated endpoint once volume justifies it, often by changing a model or endpoint parameter rather than an integration.

How the DigitalOcean Inference Engine unifies serverless, dedicated, and batch inference

The DigitalOcean Inference Engine runs three inference patterns over the same GPU Droplets infrastructure, behind a single endpoint. These solutions include real-time Serverless inference across 70+ open-weight and frontier models, Dedicated inference on reserved, single-tenant GPU capacity with support for bringing your own model, and asynchronous Batch Inference for large, non-real-time jobs. The Preview Inference Router adds cost- and latency-based routing on top, and a shared Model Playground and evaluations tooling sit on the same account.
That's what makes DigitalOcean a solid solution for AI-native companies requiring both serverless and dedicated GPU tiers. The unification isn't limited to the inference API itself. It extends to the GPU Droplets the dedicated tier runs on, as well as the rest of the AI-Native Cloud around them. A workload doesn't need to switch vendors, contracts, or billing relationships as it matures from prototype to production.

What to check before assuming "one API" elsewhere means the same thing

Several other inference-focused platforms let serverless and dedicated calls share one API too, usually as a way past per-token rate limits or shared-capacity contention once traffic justifies reserving GPUs. But a shared API can sound more unified on a landing page than it is in production. So it's worth confirming directly against each provider's current docs: does provisioning real dedicated capacity happen through the API, or does it route through a sales conversation first? Is the dedicated tier's hardware genuinely part of the same cloud a team already runs on, or does the provider specialize in inference alone, with GPUs sitting apart from the application's data and storage? And does "one API" cover just the inference call—or the account, billing, and observability around it too?

What else to verify before standardizing on a provider

Beyond scope, it's worth confirming: whether the model you need is available in both tiers (not every model ships to both), what the cutover mechanism actually looks like (a new endpoint URL versus a full redeploy), how billing reconciles if usage straddles both tiers in the same month, and whether dedicated capacity is truly single-tenant or a reserved slice of shared hardware. For teams already running other infrastructure on a given cloud, it's also worth weighing how much value comes from the inference API alone versus not having to move data, agents, or storage to a second vendor to get dedicated GPU access.

References & further reading

DigitalOcean's Inference Engine brings serverless, dedicated, and batch inference together in one place.
Our Dedicated Inference documentation covers bring-your-own-model (BYOM) support and scaling settings for teams ready to move off serverless.
Read our Serverless Inference documentation for the model catalog and routing options available on the shared endpoint.
Reading through each inference provider's own docs tends to be the fastest way to confirm whether "serverless and dedicated" really means one API or two separate products.

What It Actually Costs to Serve a 1M-Token Model in Production

DigitalOcean — Mon, 21 Sep 2026 20:33:36 +0000

Model providers now advertise context windows large enough to hold a codebase or a stack of contracts in a single request. It's tempting to read that number as a green light, but it isn't. A model that accepts a million tokens and a system that can serve a million tokens to real, concurrent users within your latency and cost budget are two different engineering problems. Conflating them is how teams end up with a demo that works and a bill or a p99 latency chart that doesn't.

You serve a 1M-token model in production by treating context length as a cost and latency variable you manage on purpose, not a spec you flip on. Let’s dive into the memory math, the latency cliff, and the cost curve for serving a model with a million tokens of context, plus the caching and routing techniques that production teams use today to keep all three in check. Plus, a straight answer on when long context is the wrong tool for the job and retrieval should do the work instead.

Key takeaways:

Serving a 1M-token context model in production is a memory, latency, and cost problem layered on top of a model capability. Supporting long context and serving it reliably are not the same claim.
Serving long context effectively means stable agent sessions and predictable spend, rather than retries, timeouts, and a bill that scales with the window instead of the question.
The real decision isn't whether to "turn on" long context, it's which combination of caching, prefix reuse, and retrieval fits your workload's cost and latency budget.
Production-grade 1M-token serving combines general techniques, including KV cache economics and prefix-reuse approaches like SGLang's RadixAttention. Use DigitalOcean Inference Router for cache-aware routing, plus retrieval-augmented generation (RAG) as the alternative and complement.

What "serving" a long-context model actually means

Context is everything you send to a model in a single request, measured in tokens: the prompt, the documents, the conversation history. A 128K-token window is roughly a novel's worth of text, and frontier closed models now advertise up to 1M tokens (roughly, ten novels). That number describes what the model can read in isolation, but not what happens to time-to-first-token, batch size, or your GPU bill once dozens of those requests land on the same cluster at once.

Support is a model capability, and performance is a serving question. The two come apart the moment real traffic shows up.

Two mechanics decide whether a long-context request is slow to start or slow to finish:

Prefill: The compute-bound pass where the model reads your entire input before writing anything.
KV cache: The memory that stores each token's key and value vectors so the model doesn't reread the input on every step.

Dig in to how prefill/decode disaggregation works: the two-phase split behind every long-context serving decision below.

Why its worth optimizing for long-context

Getting long-context serving right pays off in multiple ways.

Here's what's actually at stake once you're running this in production:

Stable agent and document workloads: A correctly cached long-context session makes it possible for a coding agent or a multi-turn document review to hold state across dozens of turns without resending the same 100K-token context on every message.
Costs you can forecast: Once you know whether traffic is cache-friendly or retrieval-shaped, it's easier to accurately predict spend.
No silent accuracy loss: Long-context serving done well accounts for the fact that models answer less reliably as input grows. This happens by routing precision lookups to retrieval instead of trusting long-context recall, and by testing accuracy at the context lengths you'll actually run rather than assuming a bigger model or window is automatically more reliable.
Latency you can put in an SLA: Understanding the prefill/decode split lets you set a time-to-first-token target you can actually hit, instead of promising an interactive response time a long-context request can't deliver.
Room to scale without a rebuild: Architectures built around caching and routing from the start absorb growth in context length and concurrent users without the rewrite that a "just resend everything" approach eventually forces.

What to evaluate before you commit to a long-context approach

Long-context serving isn't a single decision, but rather a set of infrastructure and architecture choices considered together:

Cache-awareness: Check whether the serving layer reuses the KV cache across requests that share a common prefix, rather than recomputing it each time. The DigitalOcean Inference Router, for example, pins a session to the same model so that a warm cache isn't discarded mid-task when switching to a different, cheaper model.
Batching under long-context load: Confirm the system maintains healthy batch sizes as the average context length grows, using techniques like chunked prefill and paged memory allocation, rather than letting a single long request starve everything queued behind it.
Isolation from noisy neighbors: Ask whether long-context and interactive traffic run in separate pools. Dedicated Inference on DigitalOcean exists specifically so a large document-processing job doesn't blow up latency for interactive users on the same infrastructure.
Pricing that reflects the real cost curve: Look for tiered, per-token pricing that acknowledges long-context requests cost more proportionally, plus a prompt-caching rate that's meaningfully cheaper than a full rewrite.
Observability into cache health, not just request counts: Make sure you can see KV-cache occupancy, eviction, and preemption counts—not just average latency—since that's what actually explains a p99 problem caused by long-context traffic.

Resend, cache, retrieve, or both: what each costs per session

The cleanest way to understand the cost of each approach is to price a full session rather than a single token.

These examples take a 10-turn agent conversation carrying a stable 128K-token context—a codebase or a document set—where each turn adds a short question and gets a short answer:

Strategy	Use Case	Mechanism	Cost per 10-turn session (128K context)
Resend the full context every turn	Short, one-off sessions where simplicity matters more than cost	Full context re-sent as input tokens on every turn	$3.93
Cache the context; reuse it	Stable, reused context — a codebase, a document set, a long agent session	Prefill cost paid once; later turns reuse the cached KV state	$0.95
Retrieve a relevant slice (RAG)	Large or changing corpora where each query touches a small fraction of the data	Only the relevant slice enters context; the rest of the corpus stays out	$0.33

Figures per DigitalOcean's long-context serving tradeoffs analysis, priced at Claude Sonnet's published rates ($3.00/M input tokens, $3.75/M to write the cache, $0.30/M to read it).

Caching cuts the bill roughly 4x against resending; retrieval cuts it roughly 12x. At thousands of sessions a day, that architecture choice matters more than any provider's per-token rate.

The four tradeoffs that decide whether long-context works in production

A 1M-token context window is a spec, not a guarantee that serving it will work. Memory, latency, throughput, and accuracy each break in their own way as context grows, and "supports 1M tokens" and "serves 1M tokens well" can be very different claims.

Memory: the KV cache doesn't scale the way people expect

For every token a model reads, it stores a key and a value vector so it doesn't have to reprocess the input on each new token. That cache grows in direct proportion to context length: the math works out to roughly 43GB for a single 128K-token request on a 70-billion-parameter model in BF16—close to the model's own memory footprint. Push to a million tokens, and the cache alone exceeds what a single high-end GPU holds, before the model's own weights are even loaded.

Latency: why time-to-first-token is the number that breaks

Attention compares every token against every other token, so compute grows with the square of input length: double the context, quadruple the work. That happens entirely during prefill, before a user sees a single word, which is why time-to-first-token ranges from milliseconds with short context to multiple seconds with long context. It also means one long request can hold a GPU for seconds, quietly degrading p95 and p99 latency for every short, unrelated request queued behind it.

Throughput: the bandwidth ceiling no amount of compute fixes

Generating each output token requires reading the entire KV cache from GPU memory, and the read speed is limited by hardware bandwidth. Double the context, double the cache, and tokens-per-second roughly halves, with a limit that has nothing to do with how much compute is sitting idle. A cluster can appear underutilized on a FLOPS dashboard and still serve long-context requests slowly because the bottleneck has shifted to the memory bus.

Accuracy: why a bigger window doesn't guarantee a better answer

Longer context doesn't reliably mean a better answer. A 2026 measurement study found that accuracy at long context lengths doesn't track model size — a smaller model sometimes outperformed a larger sibling — and that a wrong answer just becomes a retry, which turns an accuracy problem into a latency problem measured in total time-to-correct-answer. Production systems that route by expected accuracy at a given context length, not just by cost, cut that wasted retry time substantially.

How production teams are actually solving the long-context problem today

Successful teams aren't fighting the KV cache, they're making it reusable. SGLang's RadixAttention implements prefix caching alongside continuous batching, so a repeated prompt prefix (i.e. a system prompt, a shared reference document, a stable set of few-shot examples) gets computed once and reused across requests instead of recomputed on every call. That keeps throughput comparatively stable even as context length grows into the millions of tokens, which is exactly the workload where naive serving falls apart first.

You can run this pattern yourself on DigitalOcean GPU Droplets, or skip the setup work with the SGLang 1-Click Marketplace app, which handles the underlying infrastructure for you.

DigitalOcean's Inference Router adds cache-aware routing on top of that. It keeps a session pinned to the same model, so a warm KV cache doesn't get discarded mid-task by a switch to a different model chosen for cost reasons.

If you're self-hosting long-context serving on GPU Droplets rather than going through a managed router, three additional levers matter:

Chunked prefill, which breaks a large prefill pass into pieces so it doesn't stall the rest of the batch.
Tuning max batch size and max batched-token limits for your actual mix of context lengths.
Keeping long-context traffic in a separate pool from interactive traffic so one doesn't degrade the other.

When time-to-first-token degrades as context grows, DigitalOcean's inference-debugging guidance points to KV-cache pressure—not the scheduler—as the usual culprit. This is why watching KV-cache occupancy, eviction, and preemption counts diagnoses the problem faster than staring at a scheduler queue.

Do you even need long context? Choosing between caching, retrieval, or using both

If a model accepts a million tokens, it's fair to ask why you'd also use a vector database, a chunking pipeline, and a retrieval step. The answer is that the price of skipping retrieval includes the quadratic prefill cost, the multi-second time-to-first-token, and the tiered per-token pricing. They’re all paid on every request—even when the answer only needed to use three paragraphs of the input.

As a practical rule, long context plus caching wins when your working set is stable and reused. For example, many requests against the same document set, or a day-long agent session where the context is the state. RAG wins when your corpus is larger than any context window, changes frequently, or is mostly irrelevant to any single query. Precision lookups favor RAG even when the full corpus would technically fit, since retrieval keeps the model working at the shorter lengths where it's most reliable. Most production systems end up using both: retrieval to decide what enters the context, a long window to hold a generous amount of it, and caching wherever it repeats.

Long-context LLM serving FAQ

How do you serve long-context (1M token) models in production?

Serving a 1M-token model in production means treating context length as a cost and latency variable you manage on purpose, not a spec you flip on, since supporting long context and serving it reliably are not the same claim. It requires managing KV cache memory, the quadratic prefill cost, and the resulting time-to-first-token, then choosing the right mix of caching, prefix reuse, and retrieval for your workload. The DigitalOcean Inference Router applies cache-aware routing to this problem by pinning a session to the same model so a warm KV cache isn't discarded mid-task.

What is disaggregated prefill and decode in LLM inference?

Prefill is the compute-bound pass where the model reads your entire input before writing anything, while decode is the token-by-token generation that follows. Attention compares every token against every other token during prefill, so compute grows with the square of input length, meaning one long request can hold a GPU for seconds and degrade latency for every short request queued behind it. Chunked prefill breaks a large prefill pass into pieces so it doesn't stall the rest of the batch.

What metrics matter for LLM inference observability (TTFT, TPOT, ITL)?

Time-to-first-token is the number that breaks under long context, ranging from milliseconds with short context to multiple seconds with long context, because it's entirely determined by the prefill pass. Beyond latency averages, observability into KV-cache occupancy, eviction, and preemption counts is what actually explains a p99 problem caused by long-context traffic, not request counts alone. KV-cache pressure, not the scheduler, is the usual culprit when time-to-first-token degrades as context grows.

Which inference provider has the best caching to reduce token costs for long autonomous coding sessions?

Caching the context and reusing it lets a coding agent hold state across dozens of turns without resending the same 100K-token context every message. SGLang's RadixAttention implements this as prefix caching alongside continuous batching, computing a repeated prompt prefix once and reusing it across requests. DigitalOcean supports this pattern on GPU Droplets or through the SGLang 1-Click Marketplace app, and the Inference Router adds cache-aware routing so a warm cache isn't discarded by a mid-task model switch.

How do you model cost-per-token for LLM inference?

The cleanest way to model cost is pricing a full session rather than a single token: resending the full context every turn costs $3.93 for a 10-turn, 128K-context session, caching and reusing that context brings it to $0.95, and retrieving only the relevant slice (RAG) brings it to $0.33. Caching cuts the bill roughly 4x against resending, while retrieval cuts it roughly 12x, a difference that matters more at scale than any provider's per-token rate. DigitalOcean prices around this same curve, with tiered per-token rates that account for long-context requests costing more than proportionally, plus a prompt-caching rate meaningfully cheaper than a full rewrite.

Serve long-context models without managing the cache yourself

The DigitalOcean Inference Engine prices per token by model, with prompt-caching rates listed right next to the standard rate. That means you can work out what a long-context session costs before you run it, not after the invoice shows up. The Inference Router automatically applies cache-aware routing, pinning a session to the model whose KV cache is already warm rather than discarding it during a mid-task switch.

That gets you:

Automatic model-to-request matching, so cost- or latency-sensitive requests route to the right model without the need for writing the routing logic yourself.
Dedicated Inference to isolate long-context, latency-sensitive workloads from noisy-neighbor traffic.
Serverless-to-dedicated scaling with no migration step, so a workload that outgrows shared capacity doesn't require re-architecting.

Get started with DigitalOcean's Inference Engine →

Hippocratic AI cut prefill latency 2x on long-context clinical sessions running production inference on DigitalOcean's cache-aware routing.

https://www.youtube.com/watch?v=WeAAISnnNVA

These customer-reported results reflect Hippocratic AI’s methodology, workload, configuration, and measurement period and may not be representative of results in other environments. Results in customer environments may vary depending on configuration, implementation, and usage. Results and/or savings are not guaranteed.

Any references to third-party companies, trademarks, or logos in this document are for informational purposes only and do not imply any affiliation with, sponsorship by, or endorsement of those third parties.

AI token billing continues to cause sticker shock

DigitalOcean — Fri, 18 Sep 2026 16:25:40 +0000

Tech talk: are there inference providers that are transparent about token usage?

A top pain point for companies using AI is the discrepancy between expected and actual invoice rates. Particularly when it comes to tokens. The increase in AI usage and tokenmaxxing has created a paradox where a provider knows exactly how many tokens it generated on your behalf, yet you find out only after the fact. It's become a real cost problem when using reasoning models. Internal step-by-step reasoning counts as billed output, but it's invisible by design. The providers willing to open up their token accounting now are betting that transparency will become table stakes before their competitors catch up.

The 2025 study, “Predictive Auditing of Hidden Tokens in LLM APIs via Reasoning Length Estimation,” estimated that hidden reasoning tokens can account for over 90% of a model's total token spend on complex tasks. A short, simple-looking answer can hide 10,000+ reasoning tokens, but the only confirmation of their use is the bill.

This trend in billing surprises has rightfully drawn industry attention. Especially with token usage projected to grow 24x to 120 quadrillion tokens per month by 2030, according to research from Goldman Sachs. Developers have started noticing the gap between what usage dashboards imply and what invoices actually charge, and the industry's response is arriving on two tracks.

First, standards bodies are getting involved. The Linux Foundation's proposed Tokenomics Foundation wants open, vendor-neutral measurement standards so token accounting isn't just whatever a provider's internal dashboard says it is.

Second, providers like Groq and Cerebras are sharing flat per-token pricing. And the newly launched DigitalOcean Inference Router helps your team route tasks to the most cost-effective or optimal model, providing budget control based on your priorities for specific AI tasks.

Thanks to rapidly scaling AI adoption, developers know these costs aren’t a simple line item to ignore. When evaluating inference options, knowing exactly how your tokens are used and what that can mean for overall billing is essential.

Event calendar

Open Intelligence Summit by DigitalOcean

The argument is no longer “Is open source AI cheaper?” It’s now “Who owns intelligence?” A movement is forming around open intelligence and generating the belief that the intelligence powering our AI products and companies should be more open, portable, and controllable.

Join DigitalOcean, RadixArk, Inferact, and more for deep dives, demos, and discussions about the technologies that keep intelligence portable, interoperable, and in the hands of builders. The two-day event runs at The Midway in San Francisco from October 12-13th.

Industry news

Coding agents are earning real trust in production codebases

A study by researchers at Polytechnique Montréal, led by Canada Research Chair Foutse Khomh, analyzed 220,612 PRs across 539 Python repos and found Claude Code's PRs merge 84.3% of the time—well ahead of Codex (73.5%), Cursor (63.9%), Copilot (59.6%), and Devin (43.0%)—with bug rates comparable to or lower than human-written code.

OpenAI launches GPT-6 Astra

OpenAI introduced GPT-6 Astra, a new frontier model focused on advanced reasoning, computer use, coding, scientific research, and professional workflows. The model achieves major gains across benchmarks, including 99.9% on ARC-AGI-3 and 100% on ExploitBench, while also improving task efficiency and alignment.

Anthropic introduces Claude Fable 5.1 and Mythos 5.1

The new models target advanced coding, knowledge work, and scientific research, with Fable 5.1 offering stronger performance at lower cost and Mythos 5.1 providing more permissive safeguards for vetted cybersecurity and life-science users.

Grok Bot comes to Android

Cursor has launched the Grok Bot Android app, letting users assign tasks to AI Bots from their phones, continue conversations across devices, and monitor multiple Bots running in parallel on cloud computers. The app is currently in beta for eligible Cursor and SuperGrok plans.

Community tutorials

Why Your Best Model Is Two Models: Routing Between Kimi K3 and Claude

Hardcoded routing logic gets messy fast and using a small model like Haiku to classify requests means paying for two calls on every one. DigitalOcean built Plano-Orchestrator, a routing model that scored 87.84% accuracy against GPT-5.1's 86.93% and Claude Sonnet 4.5's 86.11%, deciding where a request goes in about 200 milliseconds. Walk through building a router that pools Kimi K3 and Claude, and see how to check with live traffic data whether it's actually working.

Resilient GPU Compute on DigitalOcean Kubernetes: Surviving Spot Interruptions

When DOKS reclaims Spot GPU capacity, it deletes the entire node pool at once, not one node at a time so the usual "autoscaler quietly relaunches it" assumption breaks completely, and nothing automatically brings a reclaimed pool back. Walk through building a pre-provisioned On-Demand fallback pool that costs nothing until a reclaim happens, and run the companion repo's automated interruption simulation to see the failover in action.

GLM-5.3-Flash is the cheapest model on DigitalOcean. It's also the most verbose.

GLM-5.3-Flash lists at one twelfth of Qwen3.8-Max's output price, but on unconfigured defaults, it burned 625 output tokens answering what port SSH uses because it silently defaults to maximum reasoning effort, cutting its real-world price advantage down to just 2.3x. See the measured cost breakdown across 2,700 API calls, and get the one-line reasoning_effort fix that made Flash 6.3x cheaper on the same prompts.

When pgvector Starts to Slow Down as Your Vector Table Grows

A pgvector's query latency doesn't creep up gradually as a table grows. It stays flat for a long time, then jumps hard the moment the HNSW graph outgrows available RAM and Postgres starts reading it from disk instead. Get the six-step method for finding your own breaking point, measuring recall, latency, and index build time at the table sizes you'll actually reach.

Does Context Length Affect Inference Cost Linearly? We Measured Why It Doesn't

Cost calculators price context length linearly, but measured throughput on a single H200 serving Ministral 3 14B fell from 19,089.8 tokens/sec at 2K context to 4,967.2 tokens/sec at 256K: a 3.84x collapse driven by concurrent request capacity dropping from 311 to just 2 as the fixed KV cache pool fills up. See the full cost curve and break-even utilization math showing exactly where a dedicated GPU stops being cheaper than a flat per-token serverless rate.

Product updates

DigitalOcean Kubernetes (DOKS) now supports Spot GPU Droplets

Running Spot GPUs on DOKS automates interruption handling with automatic cordoning, draining, and PodDisruptionBudget enforcement. Cluster autoscaling dynamically scales your GPU capacity based on workload demand. Kubernetes capabilities such as labels, taints, and node affinity allow you to configure on-demand node pools as a fallback when Spot capacity is reclaimed. Plus, reclaim events surface as native Kubernetes events. Try it today

When is Serverless Inference Cheaper than Your Self Hosted GPU? I Benchmarked gpt-oss-120b on Both

Yash Sharma — Tue, 23 Jun 2026 14:57:47 +0000

If you run LLM inference in production, you eventually will ask yourself, should you rent a GPU and run the model yourself, or do you use a serverless API and pay per token? Everyone has an opinion. Far fewer people show you the actual numbers that decide it.

So I ran both. I put the same model, gpt-oss-120b, on two setups, self-hosted with vLLM on a single AMD MI300X GPU Droplet, and on DigitalOcean's Serverless Inference. Then I measured the cold start, the warm latency, and the cost, and worked out exactly where one becomes the better choice than the other.

In short, self-hosted GPU is faster and more consistent once it's warm, but it carries a real cold start and you pay for it around the clock. Serverless hides the cold start and costs almost nothing at low volume, but you pay per token. Which one wins comes down to your traffic shape and how much your model actually outputs. Here are the numbers.

How long is the cold start on a self-hosted GPU?

When you run a model yourself, the GPU doesn't hold the model permanently. The weights have to be loaded from disk into the GPU's memory and the inference engine has to initialize before it can answer a single request. That startup delay, the gap between "process launched" and "first token out," is the cold start. You pay it every time you start the server fresh, a new deploy, a restart after a crash, or a new node coming up to handle load.

My setup here was a single AMD MI300X GPU Droplet running gpt-oss-120b with vLLM, with the weights already cached on disk. I started vLLM from cold and timed how long it took before the first token came back.

It took about 61 seconds. That wasn't a one-off, either, across three restarts it landed between 60.8 and 61.4 seconds every time.

One thing worth being precise about, because it's the most common misread, that 61 seconds is not the time to download the model. The weights were already saved on disk, so this is the cost you pay on a restart or redeploy, not a one-time setup. The startup logs show where the time actually goes:

Phase	Time
Load weights from disk into VRAM (~68 GB)	~24 s
`torch.compile`	~4 s
CUDA graph capture	~11 s
Engine init, KV cache, warmup	~21 s

So the 61 seconds includes compilation and warmup, the entire engine bring-up, right up to serving a request. What it excludes is the one-time download of the weights from Hugging Face, which you pay once and never again. (These phases come from one representative run and there's some overhead between stages, so they don't sum to exactly 61, but that's where the time lives.)

This also answers the obvious follow-up, what happens when you scale up? If a new replica boots from an image with the weights baked in, or mounts a shared volume that already has them, it pays this ~61-second load, not a download. A brand-new node with nothing staged would also have to pull the weights first, but well-run setups specifically avoid that, because you don't want every scale event re-downloading 68 GB. So 61 seconds is the realistic number for a properly configured restart or scale-up

Warm latency and throughput

Once the model is loaded, it's a different machine. Warm, the self-hosted MI300X returned the first token in about 322 ms and sustained roughly 154 tokens per second, and it was remarkably stable, across twenty requests, the spread was about two milliseconds.

Memory is worth a note, because the raw number looks alarming. The card showed about 173 GB of VRAM in use. But the weights themselves are only about 68 GB. vLLM reserves most of the rest up front as KV-cache headroom (roughly 100 GB of it) so it can serve many requests at once. A 120B model doesn't "need" 173 GB; the engine just claims the room ahead of time.

So the self-hosted trade-off is clean: once it's warm, it's fast, consistent, and entirely yours, but every cold start costs a full minute for our model, and you pay for the GPU whether or not anyone is using it.

Does serverless inference have a cold start?

Next I ran the identical test against Serverless Inference. Calling it is straightforward, you create a model access key and hit the OpenAI-compatible endpoint, so the client code is the same and only the base URL changes. I left the endpoint idle first, then measured first-token latency the same way.

The first token came back in about 546 ms, with a wider spread, anywhere from 446 ms to roughly 1.3 seconds across twenty runs. But there was no spin-up. I ran it twenty times after sitting idle and never caught a cold-start hit.

Two honest caveats on those numbers. First, the serverless requests travel over the network to DigitalOcean's endpoint, while the GPU test ran locally on the droplet, so some of that extra latency is network distance, not the model being slower. Here's the warm comparison side by side:

Metric	Self-hosted MI300X	Serverless Inference
Median time to first token	~322 ms	~546 ms
Spread (20 runs)	~2 ms	446 ms – 1.3 s
Throughput	~154 tok/s	(per-token billed)
Cold start	~61 s	none observed

So why no cold start on serverless? It didn't delete the cold start, it absorbed it. DigitalOcean pools GPU capacity across customers, so the model stayed warm without any effort from me, and I never paid the 61-second hit I took on my own box. The difference is the billing model: serverless isn't charged by the hour, it's charged per token.

To be fair, serverless isn't immune to cold starts. If you hit it during a genuinely quiet stretch, you can still catch one. The standard mitigations are sending periodic warm-up requests to keep a worker hot, or designing async-first so a slow first response doesn't matter. In this test I didn't need any of that, it just stayed warm.

When is serverless inference cheaper than your own GPU?

This is where the decision actually lives, and it comes down to arithmetic. The GPU is a flat cost, about $1.88 an hour for a single on-demand MI300X, the same whether it serves one request or a million. Serverless is usage-based, gpt-oss-120b is priced at $0.10 per million input tokens and $0.70 per million output tokens, so it costs almost nothing when you're quiet and climbs as you get busier.

The break-even point is your hourly GPU cost divided by your per-request cost:

break-even requests/hour = GPU $/hr ÷ [(input_tokens × $0.10/1M) + (output_tokens × $0.70/1M)]

The catch is that the per-request cost depends entirely on how much your model outputs, and that moves the crossover more than you'd expect. I measured it at three response lengths, on the same GPU at the same prices, changing only the output length:

Response type	Output tokens	Crossover
Short (classification / extraction)	~30	~18 requests/sec
Medium (paragraph answer)	~220	~3 requests/sec
Long (code / detailed explanation)	~1,200	<1 request/sec

That's roughly a 25x swing from identical hardware, with nothing changing but response length. Output tokens are the expensive side of the bill, so the chattier your app, the sooner owning a GPU pays for itself.

In plain terms, if your app sends short, snappy responses, serverless stays cheaper until you're well over a dozen requests per second, nonstop. If it writes long answers, the GPU starts winning below one request per second. (For comparison, Dedicated Inference, DigitalOcean's managed always-on endpoint, is billed by the GPU-hour like the Droplet but without you managing the it, so its economics sit closer to the self-hosted side of this table than the serverless side.) Drop your own response lengths and GPU rate into the formula and you'll find your exact line.

When to use serverless inference, and when not to

No "it depends." Here's the actual call.

For most teams, serverless is the right default. Bursty or spiky traffic, real idle stretches, a dev tool, an internal feature, a side project, anything async where nobody is staring at a spinner on the first request. In all of those, the cold start runs on someone else's pooled capacity, not yours, and you pay nothing while you're quiet. For that kind of traffic, it's almost perfect.

Run the GPU yourself when traffic is steady and high-volume, or when you have a latency SLA you can't miss. At that point your traffic rarely stops, so you're not benefiting from serverless's idle savings anyway, and you're using the GPU enough that flat hourly beats per-token. You keep it warm, so the cold start stops mattering. That's not a knock on serverless, it's just the wrong tool for that job.

If you're in between, start on serverless. Watch your token spend, and move to a dedicated GPU the day you cross the line for your response lengths. Don't buy a GPU to solve a problem you don't have yet.

Run it yourself

Everything here is reproducible. You can spin up an AMD GPU Droplet and run gpt-oss-120b on vLLM, hit the same model on Serverless Inference with a model access key, and check the serverless metrics and pricing pages against your own workload.

Don't take my crossover, run your own. And if you measure a cold start on your own setup, I'd genuinely like to see how the spread looks across different models and hardware.

The Hidden Cost of Complex AI Platforms: Why Developer Experience Matters

Shaoni Mukherjee — Thu, 28 May 2026 16:00:00 +0000

Key Takeaways

Developer experience is a real cost, not a soft metric: Time lost in setup, debugging, and switching tools directly slows down how fast teams can build and iterate.
Most friction comes from fragmented workflows: When model hosting, compute, and deployment live in different places, even simple tasks become multi-step processes.
Time-to-First-Value (TTFV) is a critical signal: The longer it takes to get a working output, the more likely teams are to lose momentum or abandon ideas early.
Scaling introduces a hidden breaking point: Moving from a simple API to dedicated infrastructure often forces teams to relearn workflows and rebuild systems.
This is a systems problem, not a feature gap: Many platforms weren’t designed end-to-end, which leads to disconnected experiences as teams grow.
The fastest teams aren’t just using better models: They’re working in environments where they can build, test, and scale without constant reconfiguration.

The cloud AI platform ecosystem today looks more powerful than ever, with access to powerful GPUs like NVIDIA H100 and H200, massive libraries of pre-trained models, and full pipelines for fine-tuning and inference.

I recently tried deploying a simple inference endpoint for a model. Ideally, it should have taken a few minutes:

provision compute
load the model
send a request

Instead, it took closer to two hours before I got a successful response.

Not because the model was difficult to run, but because of everything around it:

Figuring out where to start
No clear documentation
Generating and configuring the right credentials
Troubleshooting why the instance wasn’t accessible
Installing dependencies that weren’t preconfigured
Retrying after unclear or failed setup steps

None of these steps was particularly complex on its own. But together, they created enough friction to delay even a basic task.

This pattern shows up often when working with AI platforms today.

Most discussions focus on visible costs like:

Compute pricing
Storage usage
API costs

But in practice, the higher cost is harder to measure.

It’s the time spent navigating setup, resolving infrastructure issues, and figuring out how different parts of a platform fit together before any real work begins.

The real cost of building AI systems

When teams evaluate AI platforms, the focus usually stays on obvious metrics like compute pricing or model performance. But the actual cost of building AI systems runs much deeper. It shows up in how long it takes to get started, how mentally demanding the platform is, and how much time is lost dealing with infrastructure instead of building products.

One of the most overlooked factors is Time-to-First-Value (TTFV), the time it takes to go from signing up on a platform to getting your first meaningful output.

But when TTFV stretches into hours or even days due to setup issues, unclear steps, or complex configuration, it creates friction right from the start. Developers lose patience, delay experimentation, or abandon the platform altogether. Over time, this directly impacts developer retention and slows down innovation, because fewer ideas make it past the initial stage.

Fragmentation: When one platform feels like many

Imagine when a developer tries to log in and finds out multiple logins to separate platforms, which feels not only confusing but also hard to understand. When a single platform feels like multiple disconnected products stitched together.

On the surface, everything may exist under one umbrella. But once you start using it, the experience tells a different story.

Split product surfaces

On platforms like Nebius, you have AI Cloud and Token Factory, which require separate logins; this infrastructure feels like two separate worlds.

You might provision compute in one place, manage models in another, and handle access or tokens somewhere else entirely. Each part works on its own, but they don’t always feel connected.

For example, a developer might:

Set up a GPU instance in one interface
Switch to another section to access models
Move again to configure authentication or tokens

Even though it’s technically one platform, it doesn’t feel like a single, cohesive system. This lack of cohesion forces developers to constantly piece together workflows on their own.

Confusing navigation

Fragmentation often leads to a simple but frustrating question:
“Where do I even start?”

When features are spread across different sections or products, developers are left guessing:

Which interface should I use first?
Where do I run my model?
Where do I manage credentials or access?

Instead of a clear starting point, the experience becomes exploratory—and not in a good way.

A common situation is having to jump between different portals just to complete a basic setup. For instance, setting up access in one place and then realizing you need to log into a completely different interface to actually use it.

Broken flow

This fragmentation becomes even more apparent when workflows are interrupted.

Developers may encounter:

Separate logins for different parts of the platform
Different dashboards that don’t share context
Disconnected user experiences that don’t carry over progress

What fragmentation looks like

A typical workflow, for example, building and deploying an agent, might look simple:

But instead of happening in a single, continuous flow, each step exists in a different part of the platform.

Compute is managed in one dashboard
Model configuration happens in another section
Workflows are defined in a separate interface
Logs and monitoring are located somewhere else
Access and credentials are handled independently

Each step works on its own.

The hidden cost

Fragmentation usually doesn’t hurt in the beginning. When a single developer is experimenting, it’s still manageable to move between different sections of a platform and piece things together. The problem starts when the team grows, and the workflow becomes more complex. This typically happens when:

1) Multiple components like models, agents, and data sources are involved,

2) More than one developer is working on the system, and

3) Faster iteration and debugging become important.

At this stage, constantly switching between interfaces, tools, and dashboards slows everything down because there is no single place to see or manage the full workflow. This issue exists because most platforms are not built as a unified system from the start.

Fragmentation is not about missing features, but it is about how those features are connected to make it feel like a single system.

The anti-developer experience

A common pattern across many AI platforms is asking developers to commit before they’ve had a chance to see real value.

In some cases, you’re required to add billing details even before running your first model. In others, the free credits are so limited that you can barely complete a meaningful experiment. You might start testing an idea, only to run out of credits halfway through, without fully understanding whether it works.

This creates psychological friction.

Instead of freely exploring, developers become cautious. They hesitate to try new models, avoid running multiple experiments, and constantly think about cost rather than creativity. The experience shifts from curiosity to calculation.

But better-designed platforms take a different approach.

They give developers enough room to explore properly, sometimes even offering generous free credits, so you can actually spin up resources, run models, and experiment without immediate pressure. You can try things, make mistakes, and learn before worrying about billing.

Because once developers see something work, they’re far more likely to continue building.

The scaling cliff nobody talks about

Inference-as-a-service feels effortless in the beginning. You send a request to an API, get a response, and move on. There is no need to think about infrastructure, scaling, or deployment. This makes it incredibly effective during the early stages, where the focus is on building quickly, experimenting, and testing ideas without friction.

In this phase, everything works because the system is still small.

1) The number of requests is low,

2) Latency is not critical, and

3) Occasional failures are acceptable.

The platform handles everything behind the scenes, allowing developers to focus entirely on the product.

The problem starts when the system begins to grow.

As usage increases, the same setup is now operating under very different conditions. More users mean more requests, often happening at the same time. Latency is no longer just a technical detail; it becomes part of the user experience. Failures are no longer minor inconveniences; they directly impact reliability.

This is where cracks begin to appear.

A common scaling cliff in inference

A typical early setup looks like:

A hosted model endpoint
Pay-per-request pricing
No infrastructure management
Acceptable latency (often in the 300–500 ms range)

At low to moderate usage, this model works well. Teams can ship quickly, iterate rapidly, and avoid thinking about GPUs or deployment complexity.

The problem is not at the start, but it emerges when usage becomes predictable and sustained.

Where things start breaking

As request volume grows (for example, into the range of thousands of requests per day), a consistent pattern of issues begins to appear:

1. Latency variability increases

Cold starts become more frequent
P95 latency spikes unpredictably
Limited ability to tune performance

2. Cost efficiency degrades

Pay-per-request pricing scales linearly with usage
No optimization for steady workloads
The same workload becomes disproportionately expensive

3. Lack of capacity guarantees

No predictable throughput
No visibility into resource allocation
No way to reserve or prioritize compute

At this stage, the limitation is not a missing feature but a mismatch between the pricing and deployment model and the workload.

The forced transition

The natural next step is moving to dedicated infrastructure.

In practice, this transition introduces significant complexity:

Selecting GPU types without clear workload mapping
Configuring deployment environments manually
Implementing autoscaling policies
Managing routing, load balancing, and failure handling
Rebuilding abstractions that were previously handled by the platform

What begins as a simple API integration evolves into a full infrastructure problem.

The real cost

Teams are forced to shift from:

Product iteration → infrastructure management
Application logic → deployment tuning
Fast experimentation → operational maintenance

This shift directly impacts development velocity.

In many cases, the bottleneck is no longer model performance or GPU access, but the effort required to operate the system reliably at scale.

Why this matters

Inference is often presented as two separate modes:

Serverless APIs for getting started
Dedicated infrastructure for scaling

However, the transition between these modes is fragmented.

This creates a gap where teams:

Overpay for convenience longer than they should
Delay scaling due to operational complexity
Or prematurely invest in infrastructure

The issue is not the availability of tools.
It is the lack of a smooth, continuous path between them.

This is a structural problem in the current inference ecosystem — and one that directly impacts how quickly teams can move from prototype to production.

Why it feels like a cliff

This shift feels difficult not just because there is more to do, but because the change is abrupt.

Teams go from a world where everything is abstracted behind a simple API to one where they are responsible for compute, scaling, and reliability. There is no gradual transition between these two states.

There is no middle layer that offers both simplicity and control.

That is why it feels like a cliff instead of a smooth progression.

Why this happens

This gap exists because platforms are built with different starting points. Inference-focused platforms are designed for simplicity and fast onboarding, so they abstract away infrastructure details. Compute-focused platforms, on the other hand, are built for flexibility and performance, which means they require deeper involvement from the developer.

Over time, both types of platforms try to expand their capabilities. Inference platforms add more control, and compute platforms add higher-level abstractions. But these additions are layered on top rather than designed as a unified system.

As a result, the transition between simplicity and control is not seamless.

The real impact

This shift usually happens at a critical moment, when the product is gaining traction and needs to scale reliably.

Instead of focusing on improving the product, teams find themselves dealing with infrastructure, performance issues, and system stability. The pace of development slows down, not because the problem is harder, but because the platform now requires significantly more effort to manage.

It is what happens when they begin to work at scale, and the platform that once made things easy is no longer enough.

What good AI platforms actually look like

After all the friction, the starting problem, platform debugging, understanding the documentation, and platform fragmentation, it is easy to think the problem is missing features, but it's not.

Most platforms already have the same core capabilities. What actually matters is how much effort it takes to go from an idea to something that works and keep it working as it grows.

Scenario 1: Building an AI Agent in an Integrated Workflow
Consider building a simple AI agent or chatbot on an integrated platform where models, Knowledge bases, embedding models, and workflows are available in one place.

A simpler platform will make this process pretty straightforward:

Select the model
Define the agent logic
Add appropriate knowledge base
Add a data source to your knowledge base
Run a test input
Make your agent publicly available

And that's it. What stands out in this setup is not the number of features, but how the flow behaves.

You don’t need to switch between multiple interfaces to connect components. The model, workflow, and execution are visible in the same place. When you make a change, it reflects immediately without requiring additional setup or restarts.

If something fails, the issue is tied directly to the step where it happened. You don’t have to search across different dashboards to understand what went wrong.

The experience feels continuous.

You start with an idea, implement it, and see the result without getting pulled into infrastructure or configuration issues.

This is what a unified workflow looks like in practice, not just having all the pieces, but having them work together in a way that reduces effort at every step.

Scenario 2: Consider a setup where a team moves from a basic API-based workflow to dedicated inference in order to handle real user traffic more reliably.

The goal is simple:

Deploy a model with dedicated capacity
Send requests through a stable endpoint
Maintain consistent response times

What changes in this setup is not the workflow itself, but how predictable it becomes.

Once the model is deployed on dedicated infrastructure, requests are no longer competing for shared resources. Response times become more consistent, even as usage increases. Instead of worrying about rate limits or sudden slowdowns, the system behaves in a way that is easier to reason about.

At the same time, the transition does not require rebuilding everything from scratch. The way requests are sent and responses are handled remains familiar. The difference is that there is more control over how the system performs under load.

If something needs to be adjusted, such as scaling capacity or tuning performance, it can be done without changing the core application logic.

This is where dedicated inference makes a difference in practice, not by adding complexity, but by making the system more stable as it grows.

You don’t switch contexts to get basic work done In a well-designed platform, deploying a model, testing it, and monitoring it all happen in one place. You’re not jumping between dashboards, CLI tools, and cloud consoles just to complete a single workflow.
Time-to-First-Value (TTFV) stays consistently low It shouldn’t take hours to figure out how to get a model running. A good platform makes the “first successful response” happen quickly — not just in ideal conditions, but even when you’re unfamiliar with the setup. If you’re spending time debugging environment issues instead of validating outputs, that’s a design failure, not a user error.
The path from prototype to scale doesn’t change shape One of the biggest failure points in current platforms is that the workflow breaks when you scale. A well-designed system keeps the same mental model — the way you deploy and interact with a model at a small scale should still work when traffic increases. You shouldn’t need to relearn everything just to handle more requests.
Infrastructure decisions are abstracted until they actually matter You shouldn’t need to think about GPU types, networking, or provisioning just to test an idea. Good platforms delay these decisions without hiding them completely — they only surface when you have a real reason to care, like optimizing latency or cost.
Failure modes are visible and easy to debug When something breaks, it’s obvious where and why. You’re not digging through multiple systems trying to trace a failed request. Logs, errors, and performance signals are tied directly to the workflow you’re already using, so debugging doesn’t become a separate project.

Conclusion

The hardest part of building AI systems today isn’t getting access to models or GPUs, but it’s everything that happens around them.

It’s the time lost moving between tools.

It’s the friction of stitching together workflows that were never designed to work as one.
It’s the moment when something that worked at a small scale suddenly forces a complete rewrite.

And most of this doesn’t show up in benchmarks or pricing comparisons. It shows up in delays, workarounds, and abandoned ideas.

The teams that will win on inference aren’t the ones with the most compute. They’re the ones that can move from idea to working system and then to scale without having to change how they build along the way.

The real question isn’t which platform has the best features.

It’s this:
How many times does your workflow break before you get to something that actually works?

References

How to Deploy Hermes' Self-Improving AI Agent

haimantika mitra — Tue, 26 May 2026 16:00:00 +0000

Key takeaways

You host Hermes on a DigitalOcean Droplet (Ubuntu 24.04, at least 2 vCPUs and 4 GB RAM), install with the official script, and run hermes setup for your LLM provider.
The Telegram bot uses hermes gateway setup and a persistent gateway service so Hermes stays reachable after you close SSH.
Skills are Markdown files under ~/.hermes/skills/. MCP servers go in Hermes config and expose tools such as grocery search and cart actions.
HTTP MCP with OAuth on a headless server uses the URL Hermes prints plus an SSH tunnel from your laptop browser to the Droplet loopback port.
The grocery walkthrough uses Swiggy Instamart where available; you swap in another MCP URL for your region or stack.

2026 is the year of agents doing the task for us rather than just guiding us to. We saw the rise of OpenClaw and now we have Hermes agent. It is an open-source, self-improving AI agent built by Nous Research. It has a built-in learning loop that creates skills from experience, improves them during use, nudges itself to persist knowledge, and builds a deepening model of who you are across sessions.

In this tutorial, you will deploy Hermes on a DigitalOcean Droplet, connect it to Telegram, and extend it with a custom skill. As a practical example, you will see how to build a grocery tracking agent that monitors daily consumption, alerts you when stock is low, and places orders automatically through a grocery delivery service.

The grocery example uses Swiggy Instamart, which is available in India. But the approach works with any MCP-compatible service, such as a local grocery API, a task manager, a calendar, or a home automation system. The pattern is the same no matter what you connect.

Prerequisites

Before you begin, you will need:

A DigitalOcean account. If you do not have one, sign up here.
A DigitalOcean Droplet running Ubuntu 24.04 with at least 2 vCPUs and 4 GB RAM.
A Telegram account.
API key from an LLM provider. Hermes supports Anthropic, OpenAI, OpenRouter, and others.

What are we building

Hermes is model-agnostic and platform-agnostic. It can connect to any tool that supports the Model Context Protocol (MCP), and it can send and receive messages on Telegram, WhatsApp, Discord, and Slack. You can find more information on their documentation.

The architecture in this tutorial looks like this:

Let’s get started building:

Step 1. Create the Droplet

On the creation page, select:

Region: The one closest to you
Image: Ubuntu 24.04 LTS
Plan: Basic, with at least 2 vCPUs and 4 GB RAM (the s-2vcpu-4gb size)
Authentication: SSH Key

If you need help adding an SSH key, follow the DigitalOcean SSH key guide.

Once the Droplet is created, SSH into it:

ssh root@YOUR_DROPLET_IP

Update the system before installing anything:

apt update && apt upgrade -y

Step 2. Install Hermes Agent

Hermes provides an install script that handles all dependencies including Python, uv, and the hermes binary itself.

curl -fsSL https://raw.githubusercontent.com/NousResearch/hermes-agent/main/scripts/install.sh | bash

Once it finishes, reload your shell so the hermes command is available:

source ~/.bashrc

Run the setup wizard:

hermes setup

The wizard will ask you to choose an LLM provider and enter your API key. Select your provider, paste your key, and Hermes will save it to ~/.hermes/.env.

Verify the installation:

hermes --version

Step 3. Connect Hermes to Telegram

Hermes connects to Telegram through a bot. Run the gateway setup:

hermes gateway setup

When it asks which platform to use, select Telegram. Hermes will walk you through creating a bot with Telegram's BotFather. Follow the steps in your terminal.

Once setup is complete, start the gateway:

hermes gateway start

To keep it running after you disconnect from the Droplet, enable it as a system service:

systemctl enable hermes-gateway
systemctl start hermes-gateway

Open Telegram, find the bot you just created, and send it a message like "hello." If it responds, Hermes is connected and ready.

You can now talk to Hermes from anywhere on your phone. Ask it to check the weather, set a reminder, search the web, or run a command on your Droplet. It handles all of this out of the box.

Step 4. Understand skills and MCP servers

Before building the automation example, let's understand two core Hermes concepts:

Skills - These are Markdown files that teach Hermes how to handle specific tasks. You write a skill file describing what the task is, what triggers it, and what steps to follow. Hermes reads all skill files in ~/.hermes/skills/ at startup and uses them to handle relevant requests.

MCP Servers - It gives Hermes access to external services. MCP (Model Context Protocol) is an open standard that lets AI agents communicate with APIs in a structured way. If a service publishes an MCP server, Hermes can search its products, manage carts, read calendars, create tasks, and more. You add MCP servers to your Hermes config and Hermes uses them automatically when relevant.

Together, skills and MCP servers let you build automations that are specific to your life and the tools you use.

Step 5. Build real-world automation: Grocery tracking

Personally, I am very bad at keeping a tab of groceries and often find myself out of essential stock just when I am about to cook. So I wanted to automate this process for me. I eat similar food everyday and also measure my calories. I used all of this information to automate grocery shopping for me.

The goal of Hermes was to track my daily grocery consumption, alert me on Telegram when something is running low, and place an order through a grocery delivery service when I say yes.

This example uses the Swiggy Instamart MCP server, which is available in India. If you are in a different country, you can swap it out for any MCP-compatible grocery or delivery service. The skill logic stays exactly the same.

Creating the skill file

Create a directory for the skill:

mkdir -p ~/.hermes/skills/grocery

Create the skill file:

nano ~/.hermes/skills/grocery/grocery_tracker.md

Paste the following and edit the groceries: section to match your actual diet and quantities:

# Grocery Auto-Order Skill

### My Daily Grocery List

groceries:
  - name: "Milk"
    unit: "ml"
    daily_consumption: 500
    reorder_quantity: 2000
    alert_threshold_days: 3
    search_query: "fresh milk 1 litre"

  - name: "Eggs"
    unit: "pieces"
    daily_consumption: 2
    reorder_quantity: 12
    alert_threshold_days: 3
    search_query: "eggs 12 pack"

  - name: "Oats"
    unit: "gm"
    daily_consumption: 60
    reorder_quantity: 1000
    alert_threshold_days: 7
    search_query: "rolled oats 1kg"

  # Add your own items following the same format

### Instructions for Hermes

#### Daily check
Read ~/.hermes/grocery_inventory.json. Subtract each item's
daily_consumption from its current quantity. For any item where
quantity / daily_consumption is less than or equal to
alert_threshold_days, send a Telegram alert listing the low
items and ask if the user wants to place an order.

#### On YES
Search each low item using its search_query at the user's saved
delivery address. Add reorder_quantity of each to cart. Send a
cart summary via Telegram with prices and total. Wait for CONFIRM.

#### On CONFIRM
Place the order. Update grocery_inventory.json with the restocked
quantities. Send a delivery confirmation message.

Save with Ctrl+O, then Enter, then exit with Ctrl+X.

Creating the inventory file

The inventory file tracks how much of each item you have at home right now.

nano ~/.hermes/grocery_inventory.json

{
  "last_updated": "2026-05-07",
  "delivery_address": "YOUR FULL ADDRESS HERE",
  "snooze_until": null,
  "items": {
    "Milk": { "quantity": 1000, "unit": "ml" },
    "Eggs": { "quantity": 6, "unit": "pieces" },
    "Oats": { "quantity": 300, "unit": "gm" }
  }
}

Replace the quantities with what you actually have at home. Save and exit.

Connect to an MCP server

Add the grocery service MCP to your Hermes config:

hermes config edit

Scroll to the bottom of the file and add your MCP server. For Swiggy Instamart:

mcp_servers:
  swiggy-instamart:
    url: "https://mcp.swiggy.com/im"
    auth: oauth

For any other MCP-compatible service, replace the name and URL with the values from that service's documentation.

Save and exit, then verify it appears:

hermes mcp list

Step 6. Authenticate with your MCP server

Most MCP servers require OAuth authentication. Because your Droplet is headless, the OAuth callback needs to reach your browser through an SSH tunnel.

Run the login command on your Droplet:

hermes mcp login swiggy-instamart

Hermes will print a URL containing a port number in the redirect URI, like http://127.0.0.1:45123/callback. Note that port number.

On your local machine, open a new terminal and run the SSH tunnel using that port:

ssh -i ~/.ssh/id_ed25519 -L 45123:127.0.0.1:45123 root@YOUR_DROPLET_IP

Keep that terminal open, then immediately open the URL from your Droplet in your browser and complete the login. You have about 30 seconds before the token expires, so have both terminals ready before you start.

When authentication succeeds, your Droplet terminal will show:

✓ Authenticated — tools available

Telling Hermes Your Current Stock

Start Hermes and initialize the inventory:

hermes

Type:

Set up my grocery tracker. Ask me for current stock levels for each item in the grocery_tracker skill.

Hermes will ask about each item one by one. Answer with what you currently have at home and it will update grocery_inventory.json automatically. Once you have given the update, you will receive a message similar to this:

Setting up the daily alert

Still inside Hermes, set up the cron job:

Add a daily cron at 8am: check my grocery inventory using the grocery_tracker skill, subtract daily consumption, and send me a Telegram message if anything is running low.

Hermes will schedule this and confirm. From now on, every morning it checks your stock and messages you on Telegram if you need to reorder. This is how it looks:

Step 7. Test the full flow

Send a test message to your Telegram bot:

Check my groceries and tell me what's running low.

You should receive a Telegram message like this:

Reply YES. Hermes searches your connected grocery service, builds a cart, and sends a summary.

Once everything is set up, you can manage your agent entirely from Telegram:

Message	What Hermes does
`Check my groceries`	Shows all items and days of stock remaining
`I bought 12 eggs`	Updates egg stock in the inventory file
`Order groceries now`	Skips the check and goes straight to ordering
`What's running low?`	Lists items near their alert threshold
`SKIP`	Snoozes today's alert until tomorrow

Conclusion

You now have a self-hosted AI agent running on DigitalOcean that works for you around the clock. Hermes connects to Telegram and more messaging platforms so you can reach it from anywhere, and skills plus MCP servers let you extend it to handle almost anything.

The grocery automation you built in this tutorial is a starting point. The pattern is reusable: write a skill file that describes the task, connect an MCP server that gives Hermes access to the right service, and set a cron job to trigger it automatically. The grocery automation is one example. The same pattern works for any repetitive task in your life. A few ideas to get you started:

Bill reminders- Create a skill that tracks recurring payments, calculates due dates, and alerts you three days before each bill is due.

Health tracking- Log your workouts or meals via Telegram and have Hermes summarize your week every Sunday.

Home automation- Connect Hermes to a smart home MCP server and have it adjust lights, thermostats, or appliances based on a schedule or your location.

April 2026 DigitalOcean Tutorials: Inference Optimization and AI Infrastructure

DigitalOcean — Fri, 22 May 2026 21:55:02 +0000

Most AI teams hit the same walls once they move past prototyping. The RAG pipeline that worked flawlessly in a demo starts hallucinating under real traffic. Inference costs climb without clear optimization levers. GPU resources sit underutilized while workloads spike elsewhere.

Most of the time, the root cause traces back to architecture decisions that weren't pressure-tested for production. This month's DigitalOcean tutorials focus on diagnosing and fixing those failure points across the AI infrastructure stack.

Why RAG Systems Fail in Production

Why do seemingly solid RAG demos collapse under real-world conditions? This article traces failures back to retrieval quality, latency tradeoffs, and embedding drift. You’ll get a clear picture of how upstream decisions—such as chunking strategy and ranking—directly affect downstream LLM outputs. If your team is building production pipelines, evaluation, monitoring, and retrieval engineering matter just as much as model choice.

Dedicated vs. Serverless Inference as You Scale

The choice between serverless and dedicated inference isn't a one-time decision but an evolution driven by how your workload changes over time. Early on, serverless makes sense because traffic is unpredictable and iteration speed matters more than performance optimization. As usage stabilizes, the cracks show up—latency variability frustrates users and per-request pricing gets expensive for always-on systems. Walk-throughs of Modal and Together.ai show where that transition point hits and why delaying it costs you.

Fine-Tuned LLMs on Serverless Architecture

Parameter-efficient methods like LoRA let platforms serve hundreds of fine-tuned model variants from a single GPU by layering small adapter weights on top of a shared frozen base model. This makes serverless, pay-per-token inference possible for custom models without dedicated GPU deployments. The tradeoff is cold starts: idle adapters get evicted from VRAM and need to be reloaded, adding a few hundred milliseconds of latency to the first token. You’ll learn how to minimize that with keep-alive requests, adapter rank tuning, and smarter layer targeting.

The Silent Versioning Problem in AI Inference

This one is a cautionary tale about what happens when the model behind your endpoint changes and nobody tells you. The serving stack is full of moving parts that can shift independently of the model name, and the result is silent regressions that break prompt tuning and invalidate your evaluations before you even know something moved. It includes a practical buyer's checklist for pressing inference platforms on snapshot pinning, retention commitments, and how they handle disclosure when something in the stack changes.

The Hidden Bottlenecks in LLM Inference and How to Fix Them

Faster GPUs are not the answer if the rest of your serving stack can't keep up. Spoiler: the bottlenecks are GPU underutilization from rigid batching, memory bandwidth constraints during decode, KV cache fragmentation, and CPU-side overhead from tokenization and prompt assembly. Click through for a deeper look at each one and practical fixes.

We Built a Private-Document AI App to Test Platform Security. Here Is What We Could Actually Verify

AI security should always be treated as a first-class concern, not an afterthought. This tutorial puts that to the test by building a private-document chatbot and running the same workflow across six inference platforms: DigitalOcean, Baseten, Nebius, Fireworks AI, Modal, and Together AI. Each platform is evaluated on access controls, data retention defaults, network isolation, audit logging, and shared responsibility clarity. It doubles as a practical framework for figuring out what you can actually verify before sensitive data is in flight.

Post-Inference Storage and Querying with MongoDB

Many inference tutorials stop at the model response. This one keeps going. You'll build a FastAPI app that sends images through a vision model, stores the structured predictions in MongoDB, and then exposes endpoints that let you filter by detected labels and confidence scores or run aggregation pipelines across your full dataset. It's a practical blueprint for turning raw model output into something queryable and operational.

How to Build a Multi-Agent AI System with Docker and DigitalOcean

Instead of routing everything through a single model, multi-agent systems let you split a workflow across specialized agents that each handle a different part of the problem and pass results between them. The tradeoff is coordination complexity. This walkthrough covers how to containerize each agent with Docker, manage communication between them, and deploy the full system on DigitalOcean. You'll come away with a working deployment pattern you can adapt to your own orchestration needs.

Building an AI-Powered GPU Fleet Optimizer with the DigitalOcean AI Platform ADK

A single idle GPU Droplet left running overnight can add hundreds of dollars to your monthly bill, and standard CPU monitoring won't catch it because it can't see whether the GPU is actually doing work. This tutorial builds an AI-powered agent using the DigitalOcean AI Platform ADK that scrapes NVIDIA DCGM metrics like VRAM usage, engine utilization, and power draw across your fleet in real time. It compares those metrics against configurable thresholds to flag idle resources before they inflate your cloud spend. The repo is designed to be forked and customized to your own workloads, including adding tools that let the agent take action like powering off idle nodes.

Tutorial: This AI Now Tells You if a Meeting Could Be an Email

Andrew Dugan — Thu, 21 May 2026 16:00:00 +0000

Key Takeaways

DigitalOcean's Inference Router semantically routes prompts to the most appropriate model based on custom instructions. The setup process is 'point-and-click', with no hardcoded "if/else" logic required.
The router is built directly into the inference pipeline. Users can make inference requests normally, and the router automatically handles the workflow.
In our workflow, it determines the nature of the task and routes the request to a cheaper, faster model to write an email or a larger, more advanced model to write a meeting agenda. This architecture can scale beyond meetings and can be used for support tickets, code reviews, legal documents, and more.

Think back to the last time you received a calendar invite with no agenda, 12 attendees, and a title that says "Quick Sync". We've all either held or attended meetings that "could have been an email" at some point, but what if there was a way to have a gentle nudge built straight into your workflow that only leads us into a meeting when the task requires it. Instead of defaulting to a meeting, one could describe the details of the task that needs to be addressed, and immediately either an email is written for you to send out or a meeting agenda is written ready to attach to your calendar invites. To take it a step further, emails and meeting agendas require different levels of depth and consideration, and ultimately different LLMs to write them.

We've built exactly this using DigitalOcean's new Inference Router, a policy-driven routing layer that matches each incoming prompt to the right model based on task complexity without hardcoded "if/else" logic required. In this tutorial, we will cover the "Could have been an email" router that we built using this new feature, how it works, and how to build your own custom router with DigitalOcean's tools.

How the router works with DigitalOcean

Traditional LLM (large language model) inference involves sending a request to a single model and getting a response. The better or worse the model, the better or worse the response. LLM routers are a layer in between you and a group of models that takes your request, identifies the best model for the request, and has that specific model handle it. Routers can be customized to choose models based on speed, price, specific task, or any other optimization you are looking for. It allows teams to set up a single endpoint for a wide range of needs while getting the best possible price and speed for each request.

In our case, we built a router with two tasks. The first task we made is write_email. It is backed by a cheap, fast model (Llama 3.3 Instruct 70B) for writing a simple email. The second task is write_meeting_agenda. It is backed by a frontier model (Anthropic Claude Opus 4.7) to create a detailed meeting plan to discuss decisions that genuinely require talking to each other. In the request, you describe what you need done, the topic, the stakeholders, and any agenda items, and the router reads that description, matches it against the task definitions, and routes it to whichever model fits. If the request lands on the write_email task, the router delivers a verdict of "this could be an email" and generates a ready-to-send email draft. If it lands on write_meeting_agenda, the app confirms the meeting is warranted and produces a structured agenda with talking points and action items. The routing decision itself is the verdict. No additional classification logic is needed.

Step 1. Build the router

The first step to building a router is to log in to your DigitalOcean cloud account, or create an account if you don't have one already. Navigate to the router page and select "Create Router".

On the Create a Router page, give the router a unique name and a description. That description is not just metadata. It serves as a routing prompt, giving the router overall context so it can identify the most appropriate task for each incoming request. From there you define the tasks that make up the router's logic. Each task combines a name, a description, and a model pool with a selection policy. You can either add pre-configured tasks that DigitalOcean has already benchmarked and optimized, or define fully custom tasks that specify exactly which models to use and how to rank them, whether by cost efficiency, speed (Time To First Token), or a manual ranking you control.

Once your tasks are in place, the last piece is specifying fallback models. Fallback models catch any request that does not cleanly match one of your configured tasks, and they are tried in the priority order you set. This gives the router a safety net so that even if the incoming prompt is ambiguous or outside the scope of your named tasks, a response is still generated rather than failing silently. For our email/meeting router, that means a borderline "is this a meeting or an email?" input never goes unanswered.

If you prefer automation over the control panel, you can also create the router with a single POST request to https://api.digitalocean.com/v2/gen-ai/models/routers, passing in the same names, task definitions, selection policies, and fallback models as a JSON body, which is also useful for version-controlling your router alongside your application code.

Step 2. Build the app

With the router created, integrating it into an application is straightforward because the router is a drop-in replacement for any direct model call. You use the same Chat Completions endpoint (https://inference.do-ai.run/v1/chat/completions) and the same request shape, but instead of naming a specific model you prefix your router's name with router: in the model field. For this app, the field would look like "model": "router:meeting-or-email". Authentication works the same way. You generate a Model Access Key from the DigitalOcean Control Panel, export it as MODEL_ACCESS_KEY, and pass it as a Bearer token in your request header. The user's meeting description, agenda, and attendee list become the message content, and the router takes it from there.

[label meeting_or_email.py]
import requests

def meeting_or_email(user_input):
    url = "https://inference.do-ai.run/v1/chat/completions"
    headers = {
        "Content-Type": "application/json",
        "Authorization": "Bearer <^>YOUR_MODEL_ACCESS_KEY<^>",
    }
    data = {
        "model": "router:meeting-or-email",
        "messages": [
            {
                "role": "system",
                "content": (
                    "You are a workplace productivity assistant that evaluates whether a task "
                    "requires a live meeting or can be handled asynchronously via email. "
                    "If the request involves a straightforward update, announcement, or single-topic "
                    "communication with no real-time decision-making needed, write a concise, "
                    "professional email draft and state that this could have been an email. "
                    "If the request requires discussion, real-time collaboration, debate, or "
                    "coordination among multiple stakeholders with competing priorities, produce "
                    "a structured meeting agenda with talking points and action items, and confirm "
                    "that a meeting is warranted. Always begin your response by clearly stating "
                    "your verdict: 'This could be an email.' or 'This warrants a meeting.'"
                ),
            },
            {
                "role": "user",
                "content": user_input,
            }
        ]
    }

    response = requests.post(url, headers=headers, json=data)
    response_body = response.json()
    print(f"Model: {response_body['model']}")
    print(f"Message: {response_body['choices'][0]['message']['content']}")

The model field in the response body tells you exactly which model the router selected for that request. Requests the router judged as routine land on the cheaper, faster model, while requests it judged as genuinely complex land on the frontier model. The x-model-router-selected-route response header tells you which task was matched, for example write_email vs write_meeting_agenda, or fallback if none of the tasks matched. The app does not need any if/else logic to decide what kind of meeting it is. It reads the header the router already populated and maps it to a verdict message for the user.

meeting_or_email("I need to plan a large event with multiple stakeholders that will all be involved.")

[secondary_label Output]
Model: anthropic-claude-opus-4.7
Message: This warrants a meeting.

Coordinating a large event with multiple stakeholders involves competing priorities, real-time negotiation of responsibilities, and collaborative decision-making that simply cannot be handled efficiently via email threads. Below is a structured agenda to make the meeting productive.

---

## Event Planning Kickoff Meeting
...

You can see above that with a large project the task is routed to Opus 4.7. With a smaller task that just warrants an email, below, the task is routed to Llama3.3.

meeting_or_email("I have some metrics I want to share with my team.")

[secondary_label Output]
Model: llama3.3-70b-instruct
Message: This could be an email. 

Here's a draft email you could send to your team:

Subject: Update on Key Metrics

Dear Team,
...

Step 3. Deploy to DigitalOcean App Platform

Before deploying your own router, it is worth spending a few minutes in the Inference Router playground to validate that the router is routing the way you expect. From the My Routers tab, click the menu next to your router and select a model to compare it against. The Playground opens in a split view where you can type a meeting description and see both the router's response and the comparison model's response side by side. Each result shows the cost difference, end-to-end latency, the specific model the router selected, and the task that was matched for that query. This is a useful check to confirm that your task descriptions are correctly discriminating between routine syncs and complex-coordination requests before any real traffic hits the router.

Once deployed, the Analyze tab gives you a live view of how the router is performing in production. You can see aggregate metrics across all your routers or drill into a specific one, including total requests, total token usage, model match rate, and fallback rate. Model match rate is the percentage of requests matched to a configured task, and fallback rate is the percentage that fell through to the fallback models instead. For accuracy evaluation, the Router Evaluation tool in the Playground tab lets you upload a labeled dataset and run an LLM-as-a-Judge evaluation that scores responses on completeness, correctness, token usage, and latency. Together these two views give you what you need to iterate on your task descriptions and model pools after launch as you accumulate real meeting data.

Conclusion

The meeting app we built is a thin wrapper around a genuinely powerful idea. You do not have to choose which model handles a request, you just have to describe the conditions under which each model makes sense and let the router enforce those conditions at runtime. The router does not just save money on tokens. It changes how you think about designing for complexity. Instead of building one prompt that works adequately for everything, you build narrow, well-described task buckets and let semantic matching handle the dispatch.

The broader lesson here extends well beyond meetings and emails. The same pattern applies anywhere you have a mix of requests hitting a single endpoint. This could include a customer support queue where most tickets are simple FAQs but a few require nuanced reasoning, a code review pipeline where style fixes and architecture feedback warrant very different models, or a legal document classifier where boilerplate and novel clauses should not cost the same to process. Once you have written a router description and a pair of task definitions, you have infrastructure that scales horizontally without adding branching logic to your application code. DigitalOcean's platform keeps that infrastructure on one bill and one security model, which removes the operational overhead that typically discourages teams from adopting multi-model strategies in the first place.

Tutorial: Build a Cost-Aware AI Support Triage API

James Skelton — Tue, 19 May 2026 23:10:07 +0000

Key takeaways

AI applications use a single endpoint to handle multiple complex tasks: classification, urgency scoring, customer-facing drafting, and long-form summarization.
This does not account for varying cost, latency, and quality requirements.
Building a FastAPI and using serverless inference infrastructure makes it possible to address these requirements through effective routing.

Most AI applications start with a single model hard-coded into the app. That works well for a prototype, but it breaks down the moment a single endpoint has to handle multiple complex task categories: classification, urgency scoring, customer-facing drafting, and long-form summarization all benefit from different model choices. Those tasks do not share the same cost, latency, or quality requirements.

Support triage is the cleanest example of this. A user types "how do I reset my password?" and you spend the same per-token rate as you do on a multi-paragraph escalation from an enterprise customer with logs pasted in. You can branch on ticket type in your app code and pick a different model per branch, but now your model selection logic lives inside your handler, your fallback strategy is a try/except, and every pricing change means a redeploy. The consequences include a 70B model classifying one-word tickets, no fallback when that model is slow, and a redeploy every time pricing shifts.

In this tutorial, we'll use serverless inference via DigitalOcean's Inference router to easily and quickly build a FastAPI support triage endpoint that solves all these problems at once. By the end, you'll route classification, urgency scoring, customer replies, and escalation summaries to the right model for each job — automatically, with built-in fallback, and without a single model name in your application code. You'll have a production-ready API that's 71% cheaper than running everything on a frontier model.

What you're building

Let's construct a single endpoint, POST /triage, that takes a ticket payload and returns:

Classification: the issue category (billing, bug, how-to, account, etc.)
Urgency + sentiment: a severity score and a read on customer mood
Drafted reply: a short, customer-facing response
Escalation summary: a structured brief for a human agent, generated only when the ticket is complex enough to need one

The architecture moves from this:

App → hardcoded model (one model handles every task)

to this:

App → Serverless inference via Inference Router → best-fit model per task

The router is what makes the second diagram possible without your app knowing anything about which models exist.

Serverless Inference and DigitalOcean's Inference Router

The Inference Router lets you define tasks and model pools, then routes incoming prompts to the best-fit model based on those task definitions and selection policies. A task is a named job with a description: "classify_ticket, for example. A model pool is the set of candidate models the router can choose from for that task, governed by a selection policy: lowest cost, lowest latency, a manually set ranking, or a fallback order. You configure all of this once at the router level, and your app calls the router instead of any specific model."

Serverless inference lets you send API requests to models without having to create an AI agent or worry about managing infrastructure. This allow you to get started quickly without managing any components behind an inference endpoint.

The API surface is OpenAI-compatible. The base URL is https://inference.do-ai.run/v1/, and a single model access key covers both foundation models and routers.

Project setup

In order to continue, you need Python 3.10+, a DigitalOcean account with Serverless Inference enabled, and a model access key. We have already configured the full project in this repository for your convenience, but follow along in this next section to build out your own version of the API and learn why we made specific choices for the API.

The project layout is intentionally small:

support-triage/
├── main.py
├── sample_tickets.json
├── requirements.txt
└── .env

main.py holds the application code, requirements.txt the required packages, sample_tickets.json is a sample for testing the router, and .env holds the required secrets, keys, and URL base values.

To get started, clone the repo onto your machine and install everything by pasting the following into your terminal:

git clone https://github.com/Jameshskelton/triage_app
cd triage_app
python3 -m venv venv_triage
source venv_triage/bin/activate
pip install -r requirements.txt

The OpenAI SDK works as-is for DigitalOcean's Serverless Inference: you just point base_url and api_key at DigitalOcean instead of OpenAI.

Step 1: The baseline - direct model calls

Before we touch the router, let's build the version most developers would write first: one model, hardcoded, doing all four jobs. The next few step sections outline the work we did to build the application demo. If you would like to just test the final version, check out our repository where we stored this project.

To get started, we created main.py:

import os
import re
from openai import OpenAI
from fastapi import FastAPI
from pydantic import BaseModel
from dotenv import load_dotenv

load_dotenv()

client = OpenAI(
    base_url=os.environ["DO_INFERENCE_BASE_URL"],
    api_key=os.environ["DO_MODEL_ACCESS_KEY"],
)

MODEL = "llama3.3-70b-instruct"  # one model for everything

app = FastAPI()

class Ticket(BaseModel):
    subject: str
    body: str

def call_model(system: str, user: str) -> str:
    resp = client.chat.completions.create(
        model=MODEL,
        messages=[
            {"role": "system", "content": system},
            {"role": "user", "content": user},
        ],
    )
    return resp.choices[0].message.content.strip()

@app.post("/triage")
def triage(ticket: Ticket):
    text = f"Subject: {ticket.subject}\n\n{ticket.body}"

    category = call_model(
        "Classify this support ticket into one of: billing, bug, how-to, account, other. Reply with one word.",
        text,
    )
    urgency = call_model(
        "Score urgency from 1 (low) to 5 (critical) and note sentiment. Reply as 'score: N, sentiment: X'.",
        text,
    )
    reply = call_model(
        "Write a short, professional reply to this customer. Maximum 4 sentences.",
        text,
    )
    summary = call_model(
        "Summarize this ticket for a human agent. Include the problem, what's been tried, and recommended next steps.",
        text,
    )

    return {
        "category": category,
        "urgency": urgency,
        "reply": reply,
        "escalation_summary": summary,
    }

If we have set up our .env file correctly with the right API keys and values, we can run it using the code below:

uvicorn main:app --reload

Let’s test it with two tickets (one trivial, one complex), and audit the results.

Example input 1:

curl -X POST localhost:8000/triage -H "Content-Type: application/json" -d '{
  "subject": "Password reset",
  "body": "How do I reset my password?"
}'

Example output 1:

{
  "category": "account",
  "urgency": "score: 1, sentiment: neutral",
  "reply": "You can reset your password by selecting the Forgot password link on the sign-in page and following the email instructions. If you do not receive the reset email, check your spam folder or contact support for help.",
  "escalation_summary": "The customer is asking how to reset their password. No signs of account compromise, outage, or escalation risk. Recommended next step: provide standard password reset instructions."
}

Example input 2:

curl -X POST localhost:8000/triage -H "Content-Type: application/json" -d '{
  "subject": "Production outage on enterprise account",
  "body": "Our team has been unable to access the dashboard since 09:14 UTC. We have ~200 internal users blocked. Attached are logs showing 502s from the API gateway..."
}'

This gives us something like the corresponding example output 2:

{
  "category": "bug",
  "urgency": "score: 5, sentiment: frustrated",
  "reply": "Thank you for reporting this. We understand that a production dashboard outage affecting around 200 users is urgent, and we are escalating this to our engineering team immediately. Please continue to share any relevant logs or timestamps while we investigate.",
  "escalation_summary": "Enterprise customer reports a production dashboard outage beginning at 09:14 UTC. Approximately 200 internal users are blocked. Logs indicate 502 responses from the API gateway. Recommended next steps: escalate to engineering, inspect gateway and upstream service health, correlate errors around 09:14 UTC, and provide the customer with frequent status updates."
}

Both responses are useful. That is exactly why this baseline is tempting.

But look at what just happened: the same 70B model handled everything. The model classified "How do I reset my password?" into a simple category, scored urgency, drafted a short reply, and wrote an escalation summary that the ticket did not really need. Then it handled the enterprise outage, where the larger model actually makes sense.

That is the problem. The trivial ticket and the production outage have very different cost, latency, and quality requirements, but the app treats them the same. You are paying overkill rates for simple work, there is no fallback if the model is slow or unavailable, and any model-selection change means editing application code and redeploying. Let's fix that.

Step 2: Configure the Inference Router

In the DigitalOcean control panel, navigate to the Inference Router using the left-hand sidebar. Then, create a new Inference Router. Name your Router appropriately, and give it a descriptive description of what it will do. For example, we named ours triage-router, and described it as “Demo Triage API for DO tutorial”.

The router then needs its four tasks, each with a description and a model pool with a selection policy. Each of these is outlined below. If you want to copy them to recreate this experiment, copy and paste the values within to the Router tasks individually. This will make probabilistically similar results to what we have.

Task name	Description (fed to the router)	Model pool strategy
classify_ticket	Categorize short support messages into issue types (billing, bug, how-to, account).	Lowest cost
urgency_detection	Detect severity, sentiment, and escalation risk in a single pass.	Lowest latency
draft_customer_reply	Generate a short, professional customer-facing reply.	Manual ranking
escalate_complex_issue	Summarize complex tickets into structured briefs for a human agent.	Manual ranking

When we are creating the description, selecting the router prioritization policy, and selecting the model, we need to consider the exact task we want completed to optimize our results. Here are a few things worth noting as you configure these:

Task descriptions matter. The router uses them to match incoming requests to the right task. Be specific about what the task does, what kind of input it expects, and the format of the output.
Put at least two models in every pool. A pool of one is a single point of failure. Even your "lowest cost" pool should have a fallback in case the primary is unavailable.
The selection policy is enforced inside the pool, not across pools. "Lowest cost" means "the cheapest model in this pool that's currently healthy," not "the cheapest model on the platform."

Once the router is saved, you'll get a router ID. That's what your app will call.

Step 3: Refactor the app to use the router

Now the satisfying part. Replace the hardcoded MODEL constant with the router ID, and pass the task name through the request. Below is an example of what you could do to make it work, though not exactly what we did in our final release.

ROUTER = "your-router-id"  # from the DigitalOcean control panel

def parse_urgency(urgency_text: str) -> int:
    """Extract the integer score from 'score: N, sentiment: X'. Defaults to 3 if unparseable."""
    match = re.search(r"score:\s*(\d)", urgency_text, re.IGNORECASE)
    return int(match.group(1)) if match else 3

def call_router(task: str, system: str, user: str) -> dict:
    resp = client.chat.completions.create(
        model=ROUTER,
        messages=[
            {"role": "system", "content": system},
            {"role": "user", "content": user},
        ],
        extra_body={"task": task},  # router uses this to pick the pool
    )
    return {
        "content": resp.choices[0].message.content.strip(),
        "served_by": resp.model,  # the model the router actually picked
    }

@app.post("/triage")
def triage(ticket: Ticket):
    text = f"Subject: {ticket.subject}\n\n{ticket.body}"

    category = call_router(
        "classify_ticket",
        "Classify this support ticket into one of: billing, bug, how-to, account, other. Reply with one word.",
        text,
    )
    urgency = call_router(
        "urgency_detection",
        "Score urgency from 1 (low) to 5 (critical) and note sentiment. Reply as 'score: N, sentiment: X'.",
        text,
    )
    reply = call_router(
        "draft_customer_reply",
        "Write a short, professional reply to this customer. Maximum 4 sentences.",
        text,
    )

    # Only escalate when urgency warrants a human brief
    urgency_score = parse_urgency(urgency["content"])
    summary = None
    if urgency_score >= 4:
        summary = call_router(
            "escalate_complex_issue",
            "Summarize this ticket for a human agent. Include the problem, what's been tried, and recommended next steps.",
            text,
        )

    return {
        "category": category["content"],
        "urgency": urgency["content"],
        "urgency_score": urgency_score,
        "reply": reply["content"],
        "escalation_summary": summary["content"] if summary else None,
        "routing": {
            "classify_ticket": category["served_by"],
            "urgency_detection": urgency["served_by"],
            "draft_customer_reply": reply["served_by"],
            "escalate_complex_issue": summary["served_by"] if summary else None,
        },
    }

That's the whole change. It’s already done for you in the GitHub version, so there’s no need to manually do it yourself.

With this, there are no model names anywhere in the app. The router decides which model handles each task, using the policies you configured. If you want to swap the underlying model for draft_customer_reply next month, you do it in the router, not in this file.

The app triages one ticket by breaking it into smaller AI jobs instead of asking one model to do everything at once. When you call POST /triage, main.py builds the ticket text, then sends separate router calls for:

classify_ticket: decides the ticket category, like billing, bug, how-to, account, or other.
urgency_detection: scores severity from 1 to 5 and detects sentiment; the code uses the score to decide whether to escalate.
draft_customer_reply: writes a short customer-facing response.
escalate_complex_issue: Tickets scoring 4 or 5 on urgency trigger the escalation summary; lower scores skip it entirely, which is where most of the cost savings live.

The key thing: the app always calls your DigitalOcean router ID from .env as the model, and the router decides which underlying model should handle each prompt.

Step 4: Run mixed tickets through the router

With the router wired in, let's test it. The interesting behavior shows up when you feed the endpoint a mix of simple and complex examples. Here's a small batch of simple to complex examples in sample_tickets.json:

[
  {"subject": "Password reset", "body": "How do I reset my password?"},
  {"subject": "Invoice question", "body": "Why was I charged twice on invoice INV-3382?"},
  {"subject": "This is ridiculous", "body": "Third time this week your dashboard has gone down during our standup. We're seriously evaluating alternatives."},
  {"subject": "Dashboard weird", "body": "the dashboard is weird since yesterday"},
  {"subject": "Production outage", "body": "Our team has been unable to access the dashboard since 09:14 UTC. ~200 internal users blocked. Logs attached show 502s from the API gateway, traced to..."},
  {"subject": "Feature request + complaint", "body": "Can you add bulk export? Also the existing export is too slow and crashes on >10k rows."},
  {"subject": "API auth", "body": "Getting 401s after rotating my key. Following the docs at /auth/rotate but the new key returns invalid."}
]

In order to test them in sequence, we have provided run_batch.py to facilitate this test. You can run it yourself with the following command:

python3 run_batch.py sample_tickets.json --json

Loop through them and you'll see the routing do its job. The one-line "how do I reset my password?" hits the lowest-cost pool for classification and a small, fast model for urgency. The angry churn-risk message gets flagged high-urgency quickly, but the drafted reply comes from the higher-quality pool because that response is going to a real customer. The production outage gets routed to the higher-quality pool for the escalation summary, because that summary is what a human engineer is going to read at 09:15 UTC.

Because call_router surfaces resp.model as served_by, every response now tells you exactly which model handled each task. Here's what the production outage ticket returns:

{
  "category": "bug",
  "urgency": "score: 5, sentiment: frustrated",
  "urgency_score": 5,
  "reply": "Thank you for reporting this...",
  "escalation_summary": "Enterprise customer reports a production dashboard outage...",
  "routing": {
    "classify_ticket": "openai-gpt-5-nano",
    "urgency_detection": "anthropic-claude-haiku-4.5",
    "draft_customer_reply": "anthropic-claude-sonnet-4.6",
    "escalate_complex_issue": "anthropic-claude-opus-4.7"
  }
}

One request, four different models, zero model names in your application code. The cheap classifier handled the one-word category decision, Haiku scored urgency in a single fast pass, Sonnet drafted the customer-facing reply, and Opus produced the brief your on-call engineer reads. Run the password-reset ticket and the routing.escalate_complex_issue field comes back as null — the urgency score didn't clear the threshold, and that null is real money saved.

What this actually saves you

Let's put numbers on it. Assume an average ticket is 300 input tokens, with output tokens varying by task (40 for classification, 30 for urgency, 150 for a reply, 250 for an escalation summary). In our 7-ticket sample, 2-3 score high enough to escalate; we use 20% as a steady-state estimate.

Using DigitalOcean's published serverless inference rates:

Task	Model	Per-ticket cost
classify_ticket	GPT-5 Nano	$0.000031
urgency_detection	Claude Haiku 4.5	$0.000450
draft_customer_reply	Claude Sonnet 4.6	$0.003150
escalate_complex_issue (fires ~20% of tickets)	Claude Opus 4.7	$0.007750

At 100,000 tickets/month, three strategies compared:

Strategy	Monthly cost
Hardcoded Llama 3.3 70B for everything	$109
Router (cost-aware)	$518
Hardcoded Claude Opus 4.7 for everything	$1,775

The honest result: the router isn't the cheapest option. Hardcoded Llama 70B is. But Llama 70B writing your enterprise outage reply is the cost. You're only saving money by treating a churn-risk ticket the same as a password reset.

The fair comparison is against the realistic alternative: once you decide Llama's customer-facing replies aren't good enough, the choice is Opus-for-everything or the router. The router is 71% cheaper than all-Opus while only routing the expensive Opus 4.7 model to the tickets that actually need it.

Run this math on your own ticket mix before committing. The ratio of trivial-to-complex tickets is the biggest lever: a queue that's 80% password resets saves far more than one that's 80% escalations.

Production checklist

Before you put this in front of real tickets:

Log task type, latency, token usage, and selected model on every call. You can't tune what you can't see, and the router's value is invisible without per-task metrics.
Build a small eval set per task. Maybe 20 tickets per task with known-good outputs. Run it before changing pool composition. The whole point of the router is that you can swap models without code changes, but you still want to know whether the swap was an improvement.
Keep at least one fallback in every pool. A pool of one defeats half the reason to use a router.
Use direct model calls for controlled benchmarks. When you're measuring a specific model's behavior, you don't want the router making your benchmark non-deterministic.
Revisit routing rules quarterly. Model pricing and quality shift. The pool that was "lowest cost" six months ago might not be today.
Treat task descriptions as production config. Version them, review changes, don't edit them in the UI without a record.

Closing thoughts

The app you ended up with isn't bigger than the one you started with: it's actually smaller, because the model selection logic moved out of the code and into the router. The router is doing the work that used to be a match statement: matching tasks to models, falling back when something's unavailable, and giving you a single place to change strategy. Serverless inference via DigitalOcean's Inference Router enables your app more flexibility and efficiency without any of the hassle of a hardcoded setup.

From here, a few natural next steps: stream the draft_customer_reply task back to the client so agents can start reading before generation finishes; wire the escalation summaries into your real ticketing system; or stand up a second router for an unrelated workflow and reuse the same access key.

The full sample code is available in the companion repo, and the router configuration takes about five minutes in the DigitalOcean control panel.

Python Decorators: From Basics to Real-World Use Cases

DigitalOcean — Tue, 12 May 2026 21:04:07 +0000

This article was originally written by Shaoni Mukherjee (AI Technical Writer)

Key takeaways

Python decorators allow additional functionality to be added to functions without changing the original function code.
Decorators help reduce repeated code and improve code reusability.
The @decorator_name syntax is a cleaner way of wrapping functions.
Decorators are commonly used for logging, authentication, caching, validation, and performance monitoring.
*args and **kwargs make decorators flexible enough to work with different function arguments.
functools.wraps helps preserve the original function metadata and should be considered a best practice.
Multiple decorators can be chained together to add multiple layers of functionality.
Frameworks like Flask and Django rely heavily on decorators for routing, authentication, and request handling.
Decorators should be kept simple and focused to maintain readability and easier debugging.
Understanding decorators is important for writing cleaner and more maintainable Python applications.

Introduction

While building real-world Python applications, a common challenge is the repetition of certain logic codes, such as logging, authentication, validation, time, or performance monitoring across multiple functions. For instance, API endpoints often require user authentication checks, and performance-critical functions may need execution time tracking.

Adding the same logic code within each function often leads to cluttered code, reduced readability, and increased maintenance effort. Decorators address this problem by creating the separation of such cross-cutting concerns into reusable components that can be applied to functions in a clean and consistent manner. In frameworks like Flask, the @app.route("/") decorator links a URL to a function without requiring explicit routing logic, while in Django, decorators such as @login_required enforce access control by restricting views to authenticated users. This approach promotes modularity, improves code clarity, and simplifies the overall structure of applications.

What are Python decorators?

Decorators are basically a wrapper around a function to modify it for better use. The function remains the same, but the decorator adds an extra something to the function.

The core idea

Say you have a simple function:

def greet():
    print("Hello, world!")

Now imagine you want to print a line before and after every function you write, without modifying each one. A decorator lets you do exactly that:

def my_decorator(func):
    def wrapper():
        print("--- Before ---")
        func()           # calls the original function
        print("--- After ---")
    return wrapper

@my_decorator
def greet():
    print("Hello, world!")

greet()

Output:

--- Before ---
Hello, world!
--- After ---

The @my_decorator line is just shorthand for greet = my_decorator(greet). Python replaces your function with the wrapped version automatically.
To understand the concept better, let us take a real-world example of timing a function:

import time

def timer(func):
    def wrapper(*args, **kwargs):        # *args lets it work with ANY function
        start = time.time()
        result = func(*args, **kwargs)
        end = time.time()
        print(f"{func.__name__} took {end - start:.4f} seconds")
        return result
    return wrapper

@timer
def slow_task():
    time.sleep(1)
    print("Task done!")

slow_task()

Output:

Task done!
slow_task took 1.0012 seconds

Why decorators matter (especially in real projects)

They're everywhere in Python. Common use cases include:

@staticmethod / @classmethod — built into Python for class methods
@app.route('/home') — Flask/Django use them to define web routes
@login_required — Django uses this to protect pages behind authentication
Logging, caching, retrying failed requests — all cleanly handled with decorators

A decorator takes a function, adds behavior around it, and returns a new function without touching the original code.

How decorators work internally

To understand decorators better, we will first need to understand a few core Python concepts:

Foundation: Functions are objects in Python

In Python, functions aren't special, but they're just objects like integers or strings.

def say_hello():
    print("Hello!")

# Pass a function as an argument
def run_it(func):
    func()

run_it(say_hello)   # prints: Hello!

# Assign a function to a variable
my_func = say_hello
my_func()           # prints: Hello!

# Return a function from another function
def get_greeter():
    def say_hi():
        print("Hi!")
    return say_hi   # returning the function, not calling it

greeter = get_greeter()
greeter()           # prints: Hi!

This is the entire foundation that decorators are built on.

Why are decorators needed?

Imagine there are many functions in a project, and each function needs logging.

Without decorators:

def add(a, b):
    print("Function started")
    result = a + b
    print("Function ended")
    return result

def multiply(a, b):
    print("Function started")
    result = a * b
    print("Function ended")
    return result

Problem:

Repeated code
Hard to maintain in large projects
If logging changes, every function must be updated

Decorators solve this problem by reusing common functionality.

With decorators:

Using decorators, the repeated code ("Function started" and "Function ended") can be moved into a single reusable decorator.
Instead of writing the same lines inside every function, the decorator handles it automatically.

Step 1: Create the decorator

def log_function(func):

    def wrapper(a, b):
        print("Function started")

        result = func(a, b)

        print("Function ended")

        return result

    return wrapper

Step 2: Apply the Decorator

@log_function
def add(a, b):
    return a + b


@log_function
def multiply(a, b):
    return a * b

Calling the Functions

print(add(2, 3))
print(multiply(4, 5))

Output:

Function started
Function ended
5

Function started
Function ended
20

What changed?

The functions now only contain their main logic:

return a + b

and

return a * b

The extra behavior (logging) is handled by the decorator separately.

Visual understanding

When this runs:

add(2, 3)

Python internally does this:

add = log_function(add)

So the actual flow becomes:

wrapper()
    ├── print("Function started")
    ├── call original add()
    ├── print("Function ended")
    └── return result

Better Version Using *args and **kwargs

The previous decorator only works for functions with two arguments.
A more reusable decorator looks like this:

def log_function(func):

    def wrapper(*args, **kwargs):
        print("Function started")

        result = func(*args, **kwargs)

        print("Function ended")

        return result

    return wrapper

Now it works with:

any number of arguments
positional arguments
keyword arguments

Why this is powerful

Imagine 100 functions needing logging.

Without decorators:

repeated code everywhere

With decorators:

write logging once
reuse everywhere

This is one of the biggest reasons decorators are widely used in real-world Python projects and frameworks like:

Common practical examples of Python decorators

A few of the most common practical examples are listed here, from solo projects to production systems.

1. Timing and performance measurement

Useful when profiling slow functions or benchmarking code.

import time
from functools import wraps

def timer(func):
    @wraps(func)
    def wrapper(*args, **kwargs):
        start = time.perf_counter()
        result = func(*args, **kwargs)
        end = time.perf_counter()
        print(f"{func.__name__} ran in {end - start:.4f}s")
        return result
    return wrapper

@timer
def process_data(n):
    total = sum(range(n))
    return total

process_data(1_000_000)
# process_data ran in 0.0312s

perf_counter() is preferred over time.time() for short measurements, and it's higher resolution and is not affected by system clock adjustments.

2. Logging

Instead of adding print statements everywhere, a logging decorator handles it in one place.

import logging
from functools import wraps

logging.basicConfig(level=logging.INFO)

def log_calls(func):
    @wraps(func)
    def wrapper(*args, **kwargs):
        logging.info(f"Calling {func.__name__} | args={args} kwargs={kwargs}")
        result = func(*args, **kwargs)
        logging.info(f"{func.__name__} returned {result}")
        return result
    return wrapper

@log_calls
def multiply(a, b):
    return a * b

multiply(4, 5)
# INFO: Calling multiply | args=(4, 5) kwargs={}
# INFO: multiply returned 20

In production, you'd swap logging.info for a structured logger like structlog or a cloud logging sink.

3. Retry on failure

Critical for network calls, API requests, or anything that can fail transiently.

import time
from functools import wraps

def retry(times=3, delay=1):
    def decorator(func):
        @wraps(func)
        def wrapper(*args, **kwargs):
            for attempt in range(1, times + 1):
                try:
                    return func(*args, **kwargs)
                except Exception as e:
                    print(f"Attempt {attempt} failed: {e}")
                    if attempt < times:
                        time.sleep(delay)
            raise Exception(f"{func.__name__} failed after {times} attempts")
        return wrapper
    return decorator

@retry(times=3, delay=2)
def fetch_data(url):
    import requests
    response = requests.get(url, timeout=5)
    response.raise_for_status()
    return response.json()

fetch_data("https://api.example.com/data")
# Attempt 1 failed: Connection timeout
# Attempt 2 failed: Connection timeout
# Attempt 3 failed: Connection timeout
# Exception: fetch_data failed after 3 attempts

Notice this is a decorator factory — retry(times=3) returns the actual decorator. This is how you pass arguments to decorators.

4. Caching memoization

Avoids recomputing expensive results by storing previous outputs.

from functools import wraps

def memoize(func):
    cache = {}
    @wraps(func)
    def wrapper(*args):
        if args not in cache:
            cache[args] = func(*args)
            print(f"Cache miss — computing for {args}")
        else:
            print(f"Cache hit for {args}")
        return cache[args]
    return wrapper

@memoize
def fibonacci(n):
    if n <= 1:
        return n
    return fibonacci(n - 1) + fibonacci(n - 2)

fibonacci(6)
# Cache miss — computing for (6,)
# Cache miss — computing for (5,)
# ...
fibonacci(6)
# Cache hit for (6,)   ← instantly returns stored result

Python actually ships a production-grade version of this built in:

from functools import lru_cache

@lru_cache(maxsize=128)
def fibonacci(n):
    if n <= 1:
        return n
    return fibonacci(n - 1) + fibonacci(n - 2)

lru_cache (Least Recently Used) is thread-safe and evicts old entries when the cache is full — use it over a hand-rolled version in real projects.

5. Access control authorization

A staple in web frameworks like Flask and Django.

from functools import wraps

def require_role(role):
    def decorator(func):
        @wraps(func)
        def wrapper(user, *args, **kwargs):
            if user.get("role") != role:
                raise PermissionError(f"Access denied. Required role: {role}")
            return func(user, *args, **kwargs)
        return wrapper
    return decorator

@require_role("admin")
def delete_user(user, user_id):
    print(f"Deleting user {user_id}")

admin = {"name": "Shaoni", "role": "admin"}
guest = {"name": "Guest", "role": "viewer"}

delete_user(admin, 42)    # Deleting user 42
delete_user(guest, 42)    # PermissionError: Access denied. Required role: admin

Django's @login_required and @permission_required follow this exact pattern internally.

6. Input validation

Validate arguments before they even reach your function's logic.

from functools import wraps

def validate_positive(*arg_positions):
    def decorator(func):
        @wraps(func)
        def wrapper(*args, **kwargs):
            for i in arg_positions:
                if args[i] <= 0:
                    raise ValueError(
                        f"Argument at position {i} must be positive, got {args[i]}"
                    )
            return func(*args, **kwargs)
        return wrapper
    return decorator

@validate_positive(0, 1)
def calculate_area(width, height):
    return width * height

calculate_area(5, 10)    # 50
calculate_area(-3, 10)   # ValueError: Argument at position 0 must be positive

7. Rate Limiting

Preventing a function from being called too frequently is very common in API clients.

import time
from functools import wraps

def rate_limit(calls_per_second=1):
    min_interval = 1.0 / calls_per_second
    last_called = [0.0]   # mutable container to hold state in closure

    def decorator(func):
        @wraps(func)
        def wrapper(*args, **kwargs):
            elapsed = time.time() - last_called[0]
            wait = min_interval - elapsed
            if wait > 0:
                print(f"Rate limit: waiting {wait:.2f}s")
                time.sleep(wait)
            last_called[0] = time.time()
            return func(*args, **kwargs)
        return wrapper
    return decorator

@rate_limit(calls_per_second=2)
def call_api(endpoint):
    print(f"Calling {endpoint}")

call_api("/users")
call_api("/posts")    # Rate limit: waiting 0.49s
call_api("/comments") # Rate limit: waiting 0.49s

Quick reference

Decorator	Use Case	Real-world Equivalent
`@timer`	Measure execution time	Profiling, benchmarking
`@log_calls`	Audit function calls	Observability, debugging
`@retry`	Handle transient failures	API clients, DB connections
`@lru_cache`	Cache expensive results	ML inference, DB queries
`@require_role`	Guard endpoints by role	Django, Flask auth
`@validate_positive`	Sanitize inputs early	Data pipelines, APIs
`@rate_limit`	Throttle call frequency	External API clients

Real-world use cases in frameworks

Decorators are heavily used in modern Python frameworks because they provide a clean and reusable way to add functionality to applications without modifying the core business logic.
Frameworks such as Flask and Django use decorators for:

Routing
Authentication
Authorization
Caching
Request validation
Restricting HTTP methods
Logging

These decorators make applications cleaner, easier to maintain, and more readable.

Flask routing decorator

One of the most common examples of decorators appears in Flask routing.
Using Flask:

from flask import Flask

app = Flask(__name__)

@app.route("/")
def home():
   return "Homepage"

Here:

@app.route("/")

is a decorator.
It tells Flask:
“When a user visits /, execute the home() function.”

Flask authentication decorator

Decorators are also commonly used for authentication.
Example:

@app.route("/dashboard")
@login_required
def dashboard():
   return "Dashboard"

Here:

@login_required

checks whether the user is logged in before allowing access to the dashboard.

Why this is useful

Without decorators, authentication checks would need to be repeated inside every protected function.
Example without decorator:

def dashboard():
   if not logged_in:
       return "Please log in"
   return "Dashboard"

Using decorators:

avoids repeated code
keeps route definitions clean
centralizes authentication logic

This becomes extremely useful in large applications with many protected routes.

Django authentication decorator

Django also uses decorators extensively.
Example:

from django.contrib.auth.decorators import login_required
@login_required
def dashboard(request):
   return HttpResponse("Welcome")

The @login_required decorator ensures:

only authenticated users can access the view
unauthorized users are redirected to the login page

Benefits

Reusable security checks
Cleaner view functions
Better maintainability
Centralized authentication handling

Django HTTP method restriction

Django provides decorators to restrict HTTP request methods.

Example:

from django.views.decorators.http import require_POST
@require_POST
def submit(request):
   return HttpResponse("Submitted")

The decorator:

@require_POST

ensures the function only accepts POST requests.
If a GET request is sent, Django automatically returns an error.

Why this matters

This helps:

enforce API rules
improve security
prevent invalid request types
simplify validation logic

Without decorators, manual checks would be needed inside every function.

Django caching decorator

Decorators are also used for performance optimization.

Example:

from django.views.decorators.cache import cache_page
@cache_page(60)
def my_view(request):
   return HttpResponse("Cached")

Here:

@cache_page(60)

stores the response for 60 seconds.
If another user requests the same page during that time:

Django serves the cached version
the function does not run again

Advanced decorator concepts

Once the basic concepts are understood, the next step is to learn how decorators are implemented in production-grade Python applications. Advanced decorator patterns solve practical problems such as preserving function metadata, creating configurable decorators, and combining multiple decorators together.

These concepts are widely used in frameworks, libraries, and enterprise-level Python applications.

Preserving function metadata with `functools.wraps`

One common issue with decorators is that they replace the original function with the wrapper function. As a result, important metadata such as the function name, documentation string, annotations, and debugging information may be lost.

Consider the following decorator:

def decorator(func):

   def wrapper(*args, **kwargs):
       return func(*args, **kwargs)

   return wrapper

Using it:

@decorator
def greet():
   """This function greets the user"""
   print("Hello")

Now checking the function name:

print(greet.__name__)

Output:

wrapper

Instead of returning "greet", Python returns "wrapper" because the original metadata has been overridden by the wrapper function.
This creates problems for:

debugging
logging
API documentation
introspection
testing frameworks

To solve this problem, Python provides functools.wraps.

Using `functools.wraps`

from functools import wraps

def decorator(func):

   @wraps(func)
   def wrapper(*args, **kwargs):
       return func(*args, **kwargs)

   return wrapper

Using it again:

@decorator
def greet():
   """This function greets the user"""
   print("Hello")

Now:

print(greet.__name__)

Output:

greet

The @wraps(func) decorator copies the original function metadata into the wrapper function. This is considered a best practice when writing decorators in production applications.

Decorators with arguments

In many real-world scenarios, decorators need configuration values. This requires creating decorators that accept arguments.
A decorator with arguments introduces an additional level of nesting.
Example:

def repeat(n):

   def decorator(func):

       def wrapper(*args, **kwargs):

           for _ in range(n):
               func(*args, **kwargs)

       return wrapper

   return decorator

Using it:

@repeat(3)
def greet():
   print("Hello")

Calling:

greet()

Output:

Hello
Hello
Hello

Understanding the structure

This example contains three functions:

repeat()        → accepts decorator arguments
decorator()     → accepts the original function
wrapper()       → executes additional logic

The execution flow becomes:

greet = repeat(3)(greet)

This pattern is heavily used in:

retry mechanisms
caching systems
rate limiting
authorization frameworks
logging systems
timeout handling

For example, a retry decorator may accept the number of retries:

@retry(5)

A caching decorator may accept an expiration time:

@cache(expire=60)

Decorator arguments make decorators significantly more flexible and reusable.

Chaining Multiple Decorators

Python allows multiple decorators to be applied to the same function.

Example:

@decorator_one
@decorator_two
def func():
   pass

This is internally interpreted as:

func = decorator_one(decorator_two(func))

The execution order is important.

Python applies decorators from bottom to top:

decorator_two wraps the function first
decorator_one wraps the result next

Example of chained decorators

def decorator_one(func):

   def wrapper():
       print("Decorator One - Before")

       func()

       print("Decorator One - After")

   return wrapper


def decorator_two(func):

   def wrapper():
       print("Decorator Two - Before")

       func()

       print("Decorator Two - After")

   return wrapper

Applying both decorators:

@decorator_one
@decorator_two
def greet():
   print("Hello")

Calling:

greet()

Output:

Decorator One - Before
Decorator Two - Before
Hello
Decorator Two - After
Decorator One - After

Understanding the execution flow

The function call stack becomes:

decorator_one(
   decorator_two(
       greet
   )
)

This creates nested execution layers where each decorator adds behavior before and after the wrapped function. Decorator chaining is extensively used in frameworks. For example, a web route may simultaneously use:

authentication
caching
rate limiting
logging

Example:

@app.route("/dashboard")
@login_required
@cache_page(60)
def dashboard():
   return "Dashboard"

Each decorator contributes a separate layer of functionality while keeping the core business logic clean and isolated.

Conclusion

Python decorators provide a clean and powerful way to add extra functionality to functions without modifying the original code. They help reduce code duplication, improve reusability, and make applications easier to maintain.

From simple logging examples to advanced use cases in frameworks like Flask and Django, decorators play an important role in modern Python development. Understanding how decorators work helps in writing cleaner, more scalable, and more professional Python code.

NVIDIA B300 Blackwell Ultra: A Technical Deep Dive

DigitalOcean — Thu, 07 May 2026 23:53:39 +0000

The NVIDIA B300 (Blackwell Ultra) is NVIDIA's latest data center GPU, built for AI training and inference. In this deep dive, we break down the full architecture, from its dual-die design and 5th-generation tensor cores to NVFP4 precision and NVLink 5 scaling.

What we cover:
00:00 - Introduction
00:56 - Why the B300 exists
02:25 - B300 vs B200 vs H100 — the numbers
03:59 - Dual-reticle design & NV-HBI interconnect
05:04 - 5th-gen tensor cores & NVFP4
07:56 - 288GB HBM3e memory breakdown
09:04 - Multi-GPU & NVLink 5 architecture
10:38 - Performance & efficiency summary

DEV Community: DigitalOcean

How to route LLM requests by cost vs. latency

LLM routing strategies

Routing LLM requests: A quick decision framework

How the DigitalOcean Inference Router routes LLM requests

LLM routing FAQ

References & further reading

Which inference provider has both serverless and dedicated GPU tiers on the same platform with one API?

Why serverless-only or dedicated-only platforms create a migration tax

How the DigitalOcean Inference Engine unifies serverless, dedicated, and batch inference

What to check before assuming "one API" elsewhere means the same thing

What else to verify before standardizing on a provider

References & further reading

What It Actually Costs to Serve a 1M-Token Model in Production

What "serving" a long-context model actually means

Why its worth optimizing for long-context

What to evaluate before you commit to a long-context approach

Resend, cache, retrieve, or both: what each costs per session

The four tradeoffs that decide whether long-context works in production

Memory: the KV cache doesn't scale the way people expect

Latency: why time-to-first-token is the number that breaks

Throughput: the bandwidth ceiling no amount of compute fixes

Accuracy: why a bigger window doesn't guarantee a better answer

How production teams are actually solving the long-context problem today

Do you even need long context? Choosing between caching, retrieval, or using both

Long-context LLM serving FAQ

Serve long-context models without managing the cache yourself

AI token billing continues to cause sticker shock

Tech talk: are there inference providers that are transparent about token usage?

Event calendar

Industry news

Community tutorials

Product updates

When is Serverless Inference Cheaper than Your Self Hosted GPU? I Benchmarked gpt-oss-120b on Both

How long is the cold start on a self-hosted GPU?

Warm latency and throughput

Does serverless inference have a cold start?

When is serverless inference cheaper than your own GPU?

When to use serverless inference, and when not to

Run it yourself

The Hidden Cost of Complex AI Platforms: Why Developer Experience Matters

Key Takeaways

The real cost of building AI systems

Fragmentation: When one platform feels like many

Split product surfaces

Confusing navigation

Broken flow

What fragmentation looks like

The hidden cost

The anti-developer experience

The scaling cliff nobody talks about

A common scaling cliff in inference

Where things start breaking

The forced transition

The real cost

Why this matters

Why it feels like a cliff

Why this happens

The real impact

What good AI platforms actually look like

Conclusion

References

How to Deploy Hermes' Self-Improving AI Agent

Key takeaways

Prerequisites

What are we building

Step 1. Create the Droplet

Step 2. Install Hermes Agent

Step 3. Connect Hermes to Telegram

Step 4. Understand skills and MCP servers

Step 5. Build real-world automation: Grocery tracking

Creating the skill file

Creating the inventory file

Connect to an MCP server

Step 6. Authenticate with your MCP server

Telling Hermes Your Current Stock

Setting up the daily alert

Step 7. Test the full flow

Conclusion

Related links

Preserving function metadata with `functools.wraps`

Using `functools.wraps`