DEV Community: James Skelton

Tutorial: Build a Cost-Aware AI Support Triage API

James Skelton — Tue, 19 May 2026 23:10:07 +0000

Key takeaways

AI applications use a single endpoint to handle multiple complex tasks: classification, urgency scoring, customer-facing drafting, and long-form summarization.
This does not account for varying cost, latency, and quality requirements.
Building a FastAPI and using serverless inference infrastructure makes it possible to address these requirements through effective routing.

Most AI applications start with a single model hard-coded into the app. That works well for a prototype, but it breaks down the moment a single endpoint has to handle multiple complex task categories: classification, urgency scoring, customer-facing drafting, and long-form summarization all benefit from different model choices. Those tasks do not share the same cost, latency, or quality requirements.

Support triage is the cleanest example of this. A user types "how do I reset my password?" and you spend the same per-token rate as you do on a multi-paragraph escalation from an enterprise customer with logs pasted in. You can branch on ticket type in your app code and pick a different model per branch, but now your model selection logic lives inside your handler, your fallback strategy is a try/except, and every pricing change means a redeploy. The consequences include a 70B model classifying one-word tickets, no fallback when that model is slow, and a redeploy every time pricing shifts.

In this tutorial, we'll use serverless inference via DigitalOcean's Inference router to easily and quickly build a FastAPI support triage endpoint that solves all these problems at once. By the end, you'll route classification, urgency scoring, customer replies, and escalation summaries to the right model for each job — automatically, with built-in fallback, and without a single model name in your application code. You'll have a production-ready API that's 71% cheaper than running everything on a frontier model.

What you're building

Let's construct a single endpoint, POST /triage, that takes a ticket payload and returns:

Classification: the issue category (billing, bug, how-to, account, etc.)
Urgency + sentiment: a severity score and a read on customer mood
Drafted reply: a short, customer-facing response
Escalation summary: a structured brief for a human agent, generated only when the ticket is complex enough to need one

The architecture moves from this:

App → hardcoded model (one model handles every task)

to this:

App → Serverless inference via Inference Router → best-fit model per task

The router is what makes the second diagram possible without your app knowing anything about which models exist.

Serverless Inference and DigitalOcean's Inference Router

The Inference Router lets you define tasks and model pools, then routes incoming prompts to the best-fit model based on those task definitions and selection policies. A task is a named job with a description: "classify_ticket, for example. A model pool is the set of candidate models the router can choose from for that task, governed by a selection policy: lowest cost, lowest latency, a manually set ranking, or a fallback order. You configure all of this once at the router level, and your app calls the router instead of any specific model."

Serverless inference lets you send API requests to models without having to create an AI agent or worry about managing infrastructure. This allow you to get started quickly without managing any components behind an inference endpoint.

The API surface is OpenAI-compatible. The base URL is https://inference.do-ai.run/v1/, and a single model access key covers both foundation models and routers.

Project setup

In order to continue, you need Python 3.10+, a DigitalOcean account with Serverless Inference enabled, and a model access key. We have already configured the full project in this repository for your convenience, but follow along in this next section to build out your own version of the API and learn why we made specific choices for the API.

The project layout is intentionally small:

support-triage/
├── main.py
├── sample_tickets.json
├── requirements.txt
└── .env

main.py holds the application code, requirements.txt the required packages, sample_tickets.json is a sample for testing the router, and .env holds the required secrets, keys, and URL base values.

To get started, clone the repo onto your machine and install everything by pasting the following into your terminal:

git clone https://github.com/Jameshskelton/triage_app
cd triage_app
python3 -m venv venv_triage
source venv_triage/bin/activate
pip install -r requirements.txt

The OpenAI SDK works as-is for DigitalOcean's Serverless Inference: you just point base_url and api_key at DigitalOcean instead of OpenAI.

Step 1: The baseline - direct model calls

Before we touch the router, let's build the version most developers would write first: one model, hardcoded, doing all four jobs. The next few step sections outline the work we did to build the application demo. If you would like to just test the final version, check out our repository where we stored this project.

To get started, we created main.py:

import os
import re
from openai import OpenAI
from fastapi import FastAPI
from pydantic import BaseModel
from dotenv import load_dotenv

load_dotenv()

client = OpenAI(
    base_url=os.environ["DO_INFERENCE_BASE_URL"],
    api_key=os.environ["DO_MODEL_ACCESS_KEY"],
)

MODEL = "llama3.3-70b-instruct"  # one model for everything

app = FastAPI()

class Ticket(BaseModel):
    subject: str
    body: str

def call_model(system: str, user: str) -> str:
    resp = client.chat.completions.create(
        model=MODEL,
        messages=[
            {"role": "system", "content": system},
            {"role": "user", "content": user},
        ],
    )
    return resp.choices[0].message.content.strip()

@app.post("/triage")
def triage(ticket: Ticket):
    text = f"Subject: {ticket.subject}\n\n{ticket.body}"

    category = call_model(
        "Classify this support ticket into one of: billing, bug, how-to, account, other. Reply with one word.",
        text,
    )
    urgency = call_model(
        "Score urgency from 1 (low) to 5 (critical) and note sentiment. Reply as 'score: N, sentiment: X'.",
        text,
    )
    reply = call_model(
        "Write a short, professional reply to this customer. Maximum 4 sentences.",
        text,
    )
    summary = call_model(
        "Summarize this ticket for a human agent. Include the problem, what's been tried, and recommended next steps.",
        text,
    )

    return {
        "category": category,
        "urgency": urgency,
        "reply": reply,
        "escalation_summary": summary,
    }

If we have set up our .env file correctly with the right API keys and values, we can run it using the code below:

uvicorn main:app --reload

Let’s test it with two tickets (one trivial, one complex), and audit the results.

Example input 1:

curl -X POST localhost:8000/triage -H "Content-Type: application/json" -d '{
  "subject": "Password reset",
  "body": "How do I reset my password?"
}'

Example output 1:

{
  "category": "account",
  "urgency": "score: 1, sentiment: neutral",
  "reply": "You can reset your password by selecting the Forgot password link on the sign-in page and following the email instructions. If you do not receive the reset email, check your spam folder or contact support for help.",
  "escalation_summary": "The customer is asking how to reset their password. No signs of account compromise, outage, or escalation risk. Recommended next step: provide standard password reset instructions."
}

Example input 2:

curl -X POST localhost:8000/triage -H "Content-Type: application/json" -d '{
  "subject": "Production outage on enterprise account",
  "body": "Our team has been unable to access the dashboard since 09:14 UTC. We have ~200 internal users blocked. Attached are logs showing 502s from the API gateway..."
}'

This gives us something like the corresponding example output 2:

{
  "category": "bug",
  "urgency": "score: 5, sentiment: frustrated",
  "reply": "Thank you for reporting this. We understand that a production dashboard outage affecting around 200 users is urgent, and we are escalating this to our engineering team immediately. Please continue to share any relevant logs or timestamps while we investigate.",
  "escalation_summary": "Enterprise customer reports a production dashboard outage beginning at 09:14 UTC. Approximately 200 internal users are blocked. Logs indicate 502 responses from the API gateway. Recommended next steps: escalate to engineering, inspect gateway and upstream service health, correlate errors around 09:14 UTC, and provide the customer with frequent status updates."
}

Both responses are useful. That is exactly why this baseline is tempting.

But look at what just happened: the same 70B model handled everything. The model classified "How do I reset my password?" into a simple category, scored urgency, drafted a short reply, and wrote an escalation summary that the ticket did not really need. Then it handled the enterprise outage, where the larger model actually makes sense.

That is the problem. The trivial ticket and the production outage have very different cost, latency, and quality requirements, but the app treats them the same. You are paying overkill rates for simple work, there is no fallback if the model is slow or unavailable, and any model-selection change means editing application code and redeploying. Let's fix that.

Step 2: Configure the Inference Router

In the DigitalOcean control panel, navigate to the Inference Router using the left-hand sidebar. Then, create a new Inference Router. Name your Router appropriately, and give it a descriptive description of what it will do. For example, we named ours triage-router, and described it as “Demo Triage API for DO tutorial”.

The router then needs its four tasks, each with a description and a model pool with a selection policy. Each of these is outlined below. If you want to copy them to recreate this experiment, copy and paste the values within to the Router tasks individually. This will make probabilistically similar results to what we have.

Task name	Description (fed to the router)	Model pool strategy
classify_ticket	Categorize short support messages into issue types (billing, bug, how-to, account).	Lowest cost
urgency_detection	Detect severity, sentiment, and escalation risk in a single pass.	Lowest latency
draft_customer_reply	Generate a short, professional customer-facing reply.	Manual ranking
escalate_complex_issue	Summarize complex tickets into structured briefs for a human agent.	Manual ranking

When we are creating the description, selecting the router prioritization policy, and selecting the model, we need to consider the exact task we want completed to optimize our results. Here are a few things worth noting as you configure these:

Task descriptions matter. The router uses them to match incoming requests to the right task. Be specific about what the task does, what kind of input it expects, and the format of the output.
Put at least two models in every pool. A pool of one is a single point of failure. Even your "lowest cost" pool should have a fallback in case the primary is unavailable.
The selection policy is enforced inside the pool, not across pools. "Lowest cost" means "the cheapest model in this pool that's currently healthy," not "the cheapest model on the platform."

Once the router is saved, you'll get a router ID. That's what your app will call.

Step 3: Refactor the app to use the router

Now the satisfying part. Replace the hardcoded MODEL constant with the router ID, and pass the task name through the request. Below is an example of what you could do to make it work, though not exactly what we did in our final release.

ROUTER = "your-router-id"  # from the DigitalOcean control panel

def parse_urgency(urgency_text: str) -> int:
    """Extract the integer score from 'score: N, sentiment: X'. Defaults to 3 if unparseable."""
    match = re.search(r"score:\s*(\d)", urgency_text, re.IGNORECASE)
    return int(match.group(1)) if match else 3

def call_router(task: str, system: str, user: str) -> dict:
    resp = client.chat.completions.create(
        model=ROUTER,
        messages=[
            {"role": "system", "content": system},
            {"role": "user", "content": user},
        ],
        extra_body={"task": task},  # router uses this to pick the pool
    )
    return {
        "content": resp.choices[0].message.content.strip(),
        "served_by": resp.model,  # the model the router actually picked
    }

@app.post("/triage")
def triage(ticket: Ticket):
    text = f"Subject: {ticket.subject}\n\n{ticket.body}"

    category = call_router(
        "classify_ticket",
        "Classify this support ticket into one of: billing, bug, how-to, account, other. Reply with one word.",
        text,
    )
    urgency = call_router(
        "urgency_detection",
        "Score urgency from 1 (low) to 5 (critical) and note sentiment. Reply as 'score: N, sentiment: X'.",
        text,
    )
    reply = call_router(
        "draft_customer_reply",
        "Write a short, professional reply to this customer. Maximum 4 sentences.",
        text,
    )

    # Only escalate when urgency warrants a human brief
    urgency_score = parse_urgency(urgency["content"])
    summary = None
    if urgency_score >= 4:
        summary = call_router(
            "escalate_complex_issue",
            "Summarize this ticket for a human agent. Include the problem, what's been tried, and recommended next steps.",
            text,
        )

    return {
        "category": category["content"],
        "urgency": urgency["content"],
        "urgency_score": urgency_score,
        "reply": reply["content"],
        "escalation_summary": summary["content"] if summary else None,
        "routing": {
            "classify_ticket": category["served_by"],
            "urgency_detection": urgency["served_by"],
            "draft_customer_reply": reply["served_by"],
            "escalate_complex_issue": summary["served_by"] if summary else None,
        },
    }

That's the whole change. It’s already done for you in the GitHub version, so there’s no need to manually do it yourself.

With this, there are no model names anywhere in the app. The router decides which model handles each task, using the policies you configured. If you want to swap the underlying model for draft_customer_reply next month, you do it in the router, not in this file.

The app triages one ticket by breaking it into smaller AI jobs instead of asking one model to do everything at once. When you call POST /triage, main.py builds the ticket text, then sends separate router calls for:

classify_ticket: decides the ticket category, like billing, bug, how-to, account, or other.
urgency_detection: scores severity from 1 to 5 and detects sentiment; the code uses the score to decide whether to escalate.
draft_customer_reply: writes a short customer-facing response.
escalate_complex_issue: Tickets scoring 4 or 5 on urgency trigger the escalation summary; lower scores skip it entirely, which is where most of the cost savings live.

The key thing: the app always calls your DigitalOcean router ID from .env as the model, and the router decides which underlying model should handle each prompt.

Step 4: Run mixed tickets through the router

With the router wired in, let's test it. The interesting behavior shows up when you feed the endpoint a mix of simple and complex examples. Here's a small batch of simple to complex examples in sample_tickets.json:

[
  {"subject": "Password reset", "body": "How do I reset my password?"},
  {"subject": "Invoice question", "body": "Why was I charged twice on invoice INV-3382?"},
  {"subject": "This is ridiculous", "body": "Third time this week your dashboard has gone down during our standup. We're seriously evaluating alternatives."},
  {"subject": "Dashboard weird", "body": "the dashboard is weird since yesterday"},
  {"subject": "Production outage", "body": "Our team has been unable to access the dashboard since 09:14 UTC. ~200 internal users blocked. Logs attached show 502s from the API gateway, traced to..."},
  {"subject": "Feature request + complaint", "body": "Can you add bulk export? Also the existing export is too slow and crashes on >10k rows."},
  {"subject": "API auth", "body": "Getting 401s after rotating my key. Following the docs at /auth/rotate but the new key returns invalid."}
]

In order to test them in sequence, we have provided run_batch.py to facilitate this test. You can run it yourself with the following command:

python3 run_batch.py sample_tickets.json --json

Loop through them and you'll see the routing do its job. The one-line "how do I reset my password?" hits the lowest-cost pool for classification and a small, fast model for urgency. The angry churn-risk message gets flagged high-urgency quickly, but the drafted reply comes from the higher-quality pool because that response is going to a real customer. The production outage gets routed to the higher-quality pool for the escalation summary, because that summary is what a human engineer is going to read at 09:15 UTC.

Because call_router surfaces resp.model as served_by, every response now tells you exactly which model handled each task. Here's what the production outage ticket returns:

{
  "category": "bug",
  "urgency": "score: 5, sentiment: frustrated",
  "urgency_score": 5,
  "reply": "Thank you for reporting this...",
  "escalation_summary": "Enterprise customer reports a production dashboard outage...",
  "routing": {
    "classify_ticket": "openai-gpt-5-nano",
    "urgency_detection": "anthropic-claude-haiku-4.5",
    "draft_customer_reply": "anthropic-claude-sonnet-4.6",
    "escalate_complex_issue": "anthropic-claude-opus-4.7"
  }
}

One request, four different models, zero model names in your application code. The cheap classifier handled the one-word category decision, Haiku scored urgency in a single fast pass, Sonnet drafted the customer-facing reply, and Opus produced the brief your on-call engineer reads. Run the password-reset ticket and the routing.escalate_complex_issue field comes back as null — the urgency score didn't clear the threshold, and that null is real money saved.

What this actually saves you

Let's put numbers on it. Assume an average ticket is 300 input tokens, with output tokens varying by task (40 for classification, 30 for urgency, 150 for a reply, 250 for an escalation summary). In our 7-ticket sample, 2-3 score high enough to escalate; we use 20% as a steady-state estimate.

Using DigitalOcean's published serverless inference rates:

Task	Model	Per-ticket cost
classify_ticket	GPT-5 Nano	$0.000031
urgency_detection	Claude Haiku 4.5	$0.000450
draft_customer_reply	Claude Sonnet 4.6	$0.003150
escalate_complex_issue (fires ~20% of tickets)	Claude Opus 4.7	$0.007750

At 100,000 tickets/month, three strategies compared:

Strategy	Monthly cost
Hardcoded Llama 3.3 70B for everything	$109
Router (cost-aware)	$518
Hardcoded Claude Opus 4.7 for everything	$1,775

The honest result: the router isn't the cheapest option. Hardcoded Llama 70B is. But Llama 70B writing your enterprise outage reply is the cost. You're only saving money by treating a churn-risk ticket the same as a password reset.

The fair comparison is against the realistic alternative: once you decide Llama's customer-facing replies aren't good enough, the choice is Opus-for-everything or the router. The router is 71% cheaper than all-Opus while only routing the expensive Opus 4.7 model to the tickets that actually need it.

Run this math on your own ticket mix before committing. The ratio of trivial-to-complex tickets is the biggest lever: a queue that's 80% password resets saves far more than one that's 80% escalations.

Production checklist

Before you put this in front of real tickets:

Log task type, latency, token usage, and selected model on every call. You can't tune what you can't see, and the router's value is invisible without per-task metrics.
Build a small eval set per task. Maybe 20 tickets per task with known-good outputs. Run it before changing pool composition. The whole point of the router is that you can swap models without code changes, but you still want to know whether the swap was an improvement.
Keep at least one fallback in every pool. A pool of one defeats half the reason to use a router.
Use direct model calls for controlled benchmarks. When you're measuring a specific model's behavior, you don't want the router making your benchmark non-deterministic.
Revisit routing rules quarterly. Model pricing and quality shift. The pool that was "lowest cost" six months ago might not be today.
Treat task descriptions as production config. Version them, review changes, don't edit them in the UI without a record.

Closing thoughts

The app you ended up with isn't bigger than the one you started with: it's actually smaller, because the model selection logic moved out of the code and into the router. The router is doing the work that used to be a match statement: matching tasks to models, falling back when something's unavailable, and giving you a single place to change strategy. Serverless inference via DigitalOcean's Inference Router enables your app more flexibility and efficiency without any of the hassle of a hardcoded setup.

From here, a few natural next steps: stream the draft_customer_reply task back to the client so agents can start reading before generation finishes; wire the escalation summaries into your real ticketing system; or stand up a second router for an unrelated workflow and reuse the same access key.

The full sample code is available in the companion repo, and the router configuration takes about five minutes in the DigitalOcean control panel.

A Complete Guide to Real-Time GPU Usage Monitoring

James Skelton — Wed, 15 Apr 2026 16:30:00 +0000

The fastest way to monitor GPU utilization in real time on Linux is to run nvidia-smi --loop=1, which refreshes GPU stats every second including core utilization, VRAM usage, temperature, and power draw.

Monitoring GPU utilization in real time starts with nvidia-smi, then expands to per-process views, container metrics, and alerts for long-running jobs. This guide shows command-level workflows you can run on Ubuntu, GPU Droplets, Docker hosts, and Kubernetes clusters.

If you are building or operating deep learning systems, pair this guide with How To Set Up a Deep Learning Environment on Ubuntu and DigitalOcean GPU Droplets.

Key Takeaways

Use nvidia-smi --loop=1 for the fastest host-level real-time GPU check on Linux.
Use nvidia-smi pmon -s um to identify which PID is using GPU cores and GPU memory bandwidth.
For terminal dashboards, use nvtop for interactive drill-down and gpustat for lightweight snapshots.
In containers and Kubernetes, expose metrics through NVIDIA runtime support and DCGM Exporter.
Persistent alerting belongs in monitoring platforms such as Datadog Agent or Zabbix templates.
GPU memory utilization and GPU core utilization are separate signals, high memory with low cores is common in input-stalled jobs.
On Windows, Unified GPU Usage Monitoring aggregates engine activity and surfaces it in Task Manager and WMI.

What GPU Utilization Metrics Actually Mean

GPU utilization metrics tell you whether your job is compute-bound, memory-bound, input-bound, or idle between batches. Start by tracking core utilization, memory usage, memory controller load, temperature, and power draw together instead of looking at one metric in isolation.

GPU Core Utilization vs. Memory Utilization

GPU core utilization is the percentage of time kernels are actively executing on SMs during the sampling window. GPU memory utilization in nvidia-smi usually refers to memory controller activity, while memory usage is allocated VRAM in MiB.

Low core utilization with high allocated VRAM often means the model is resident but waiting on data or synchronization. High core utilization with low memory controller activity is more common in compute-heavy kernels.

SM Utilization, Memory Bandwidth, and Power Draw

SM utilization tells you whether CUDA cores are busy, memory bandwidth indicates how hard memory channels are being driven, and power draw shows electrical load relative to the card limit. These three together explain why two workloads with similar utilization percentages can perform differently.

Use power.draw, power.limit, and utilization metrics in the same sample window when tuning batch size and dataloader workers. If power is capped while utilization is high, clock throttling can be the next bottleneck to investigate.

Why These Metrics Matter for Deep Learning Workloads

These metrics matter because training throughput is gated by the slowest stage in the pipeline. If GPU cores are idle while CPU or storage is saturated, adding another GPU will not fix throughput.

<$>[note]
For a practical environment baseline before tuning, follow How To Set Up a Deep Learning Environment on Ubuntu.
<$>

GPU Bottlenecks and Out of Memory Errors

Most GPU incidents in ML pipelines come from input bottlenecks or VRAM pressure. Diagnose both at the same time by sampling GPU, CPU, and process-level memory while a real training job is running.

CPU Preprocessing Bottlenecks

If CPU preprocessing is the bottleneck, GPU utilization drops between mini-batches even when VRAM remains allocated. This pattern appears when image decode, augmentation, or tokenization is slower than kernel execution.

Check host pressure while your training loop runs:

top

vmstat 1

procs -----------memory---------- ---swap-- -----io---- -system-- ------cpu-----
r  b   swpd   free   buff  cache   si   so    bi    bo   in   cs us sy id wa st
2  0      0 824320  74384 901212    0    0     6    10  420  980 18  4 76  2  0

In vmstat, watch r, wa, bi, and us plus sy together. r is runnable processes, and if it stays above your CPU core count, the CPU is saturated. wa is CPU time waiting on I/O, and sustained values above 10 to 15 during training often mean dataloader workers are blocked on disk reads. bi is blocks received from storage, and high bi with high wa points to storage bottlenecks instead of compute. us + sy is total active CPU time, and if it is high while GPU-Util is low, preprocessing is outrunning the GPU. If wa is high, increase dataloader workers or switch to faster storage. If us + sy is high with low GPU-Util, move transforms to GPU with a library such as Kornia.

What Causes OOM Errors and How to Resolve Them

OOM errors happen when requested allocations exceed available VRAM, often due to large batch sizes, long sequence lengths, or concurrent GPU processes. Resolve OOM by lowering memory pressure first, then increasing workload cautiously.

Common fixes:

Reduce batch size or sequence length.
Use gradient accumulation to keep effective batch size.
Enable mixed precision where supported.
Terminate stale GPU processes before restart.
Move expensive transforms to more efficient pipeline stages.

If a stale process is still holding VRAM after a failed run, list active compute processes, verify ownership, terminate the stale PID, then confirm memory was released.

nvidia-smi --query-compute-apps=pid,used_memory,process_name --format=csv,noheader

18211, 17664 MiB, python
18304, 512 MiB, python

ps -p <PID> -o pid,user,etime,cmd

kill -9 <PID>

<$>[warning]
Do not kill unknown PIDs on shared hosts. Verify process ownership and job context first.
<$>

nvidia-smi # Confirm VRAM is now released

Monitoring GPU Utilization with nvidia-smi

nvidia-smi is the fastest built-in tool for real-time GPU telemetry on Linux servers. It is available with NVIDIA drivers and documents fields used by most higher-level integrations.

Reference docs:

Basic nvidia-smi Output and What Each Field Shows

Run nvidia-smi with no flags for a full snapshot of GPU and process state. Focus first on GPU-Util, Memory-Usage, Temp, and Pwr:Usage/Cap.

nvidia-smi

+-----------------------------------------------------------------------------+
| NVIDIA-SMI 550.xx       Driver Version: 550.xx       CUDA Version: 12.x    |
| GPU  Name        Temp   Pwr:Usage/Cap   Memory-Usage   GPU-Util  Compute M. |
| 0    H100        53C    215W / 350W     18240MiB/81920MiB   78%    Default |
+-----------------------------------------------------------------------------+
| Processes:                                                                |
| GPU   PID   Type   Process name                                GPU Memory |
| 0   18211     C    python train.py                                17664MiB|
+-----------------------------------------------------------------------------+

If GPU-Util shows 0% while a job appears to be running, check three common causes. The job may still be in a CPU-bound preprocessing stage and has not submitted work to the GPU yet. The process may have errored and stayed alive but idle. The job may also be running on a different GPU index, so list all devices with nvidia-smi --list-gpus and check each one.

Running nvidia-smi in Continuous Loop Mode

Use loop mode when you need live updates without writing scripts. --loop=1 refreshes once per second.

nvidia-smi --loop=1

Wed Mar 26 12:00:01 2026
... snapshot ...
Wed Mar 26 12:00:02 2026
... snapshot ...

Logging nvidia-smi Output to a File

Write sampled output to a file for post-run inspection. Redirect stdout so each sample is timestamped in your shell history and log stream.

nvidia-smi --loop=5 > gpu.log

# gpu.log now contains one snapshot every 5 seconds

Querying Specific Metrics with nvidia-smi --query-gpu

Use --query-gpu with --format=csv when you need parseable output for scripts. This is the preferred pattern for cron jobs and custom exporters.

nvidia-smi --query-gpu=timestamp,index,name,utilization.gpu,utilization.memory,memory.used,memory.total,temperature.gpu,power.draw --format=csv,noheader,nounits

2026/03/26 12:10:02.123, 0, NVIDIA H100 80GB HBM3, 82, 54, 18420, 81920, 55, 228.31

Per-Process GPU Monitoring

Per-process monitoring answers which application is consuming GPU time right now. Use nvidia-smi pmon to inspect utilization by PID instead of by device only.

Using nvidia-smi pmon for Process-Level Metrics

Run pmon in loop mode to monitor active compute processes. -s um displays utilization and memory throughput related activity by process.

nvidia-smi pmon -s um -d 1

# gpu   pid  type    sm   mem   enc   dec   command
    0 18211     C    76    41     0     0   python
    0 18304     C    12     8     0     0   python

gpu is the GPU index the process is running on. pid is the process ID. type is workload class, where C is compute, G is graphics, and M is mixed. sm is the percentage of time spent executing kernels on streaming multiprocessors. mem is the percentage of time the memory interface was active for that process. enc and dec are encoder and decoder utilization percentages. command is the truncated process name.

Correlating Process IDs to Application Names

Map PIDs to full command lines to identify notebook kernels, training scripts, and inference workers. This is required when multiple Python jobs are running under one user.

ps -p 18211 -o pid,user,etime,cmd

  PID USER     ELAPSED CMD
18211 mlops    01:22:11 python train.py --model llama --batch-size 8

Interactive GPU Monitoring with nvtop and gpustat

Use nvtop when you want interactive process control and gpustat when you want compact snapshots in scripts. Both tools complement nvidia-smi rather than replace it.

Installing and Running nvtop

Install nvtop from Ubuntu repositories, then start it in the terminal. It provides live bars and per-process views similar to htop.

sudo apt update && sudo apt install -y nvtop

nvtop

GPU0  78%  MEM 18240/81920 MiB  TEMP 54C  PWR 221W
PID 18211 python train.py   GPU 72%   MEM 17664MiB

Installing and Running gpustat

Install gpustat with pip, then use watch mode for one-second updates. This is useful in SSH sessions where minimal output matters.

python3 -m pip install --user gpustat

gpustat --watch 1

hostname  Thu Mar 26 12:25:44 2026
[0] NVIDIA H100 | 54C, 79 % | 18420 / 81920 MB | python/18211(17664M)

When to Use nvtop vs. gpustat vs. nvidia-smi

Use nvidia-smi for canonical driver-level data and scripted queries. Use gpustat for low-noise terminal snapshots, and use nvtop for interactive process monitoring during active debugging.

GPU Monitoring with Glances

Use Glances when you need one terminal dashboard for GPU, CPU, memory, disk, and network at once. Install with the GPU extra so NVIDIA metrics are available.

python3 -m pip install 'glances[gpu]'

glances

GPU NVIDIA H100: util 77% | mem 18240/81920MiB | temp 54C | power 220W
CPU: 21.4%  MEM: 62.1%  LOAD: 2.13 1.87 1.66

In the Glances GPU line, util maps to GPU core activity, and mem shows allocated versus total VRAM. temp and power indicate thermal and electrical load during the sample window. Use these values together to identify whether workload pressure is compute, memory, or thermal related. Glances is a better choice than nvidia-smi when you want CPU, memory, disk, and GPU in one non-scrolling view during interactive debugging on a single node.

<$>[note]
If glances shows no GPU section, verify that NVIDIA drivers are installed on the host and the Python environment running Glances can access NVML.
<$>

GPU Monitoring Inside Docker Containers and Kubernetes

Containerized GPU monitoring requires host runtime support first, then workload-level metric collection. Start with NVIDIA Container Toolkit for Docker and DCGM Exporter for Kubernetes clusters.

Exposing GPU Metrics in Docker with the NVIDIA Container Toolkit

Install the NVIDIA Container Toolkit on the host, then run containers with --gpus all. Inside the container, nvidia-smi should show host GPU telemetry.

Use this after setting up Docker by following How To Install and Use Docker on Ubuntu.

curl -fsSL https://nvidia.github.io/libnvidia-container/gpgkey | sudo gpg --dearmor -o /usr/share/keyrings/nvidia-container-toolkit-keyring.gpg

curl -s -L https://nvidia.github.io/libnvidia-container/stable/deb/nvidia-container-toolkit.list | \
  sed 's#deb https://#deb [signed-by=/usr/share/keyrings/nvidia-container-toolkit-keyring.gpg] https://#g' | \
  sudo tee /etc/apt/sources.list.d/nvidia-container-toolkit.list

sudo apt update && sudo apt install -y nvidia-container-toolkit

sudo nvidia-ctk runtime configure --runtime=docker

sudo systemctl restart docker

<$>[note]
The NVIDIA runtime is only active after the Docker daemon restarts. Already-running containers are not affected, but any new container launched after the restart will have GPU access. For full installation details, see the NVIDIA Container Toolkit guide.
<$>

docker run --rm --gpus all nvidia/cuda:12.4.1-base-ubuntu22.04 nvidia-smi

+-----------------------------------------------------------------------------+
| NVIDIA-SMI 550.xx       Driver Version: 550.xx       CUDA Version: 12.x    |
| GPU  Name        Persistence-M| Bus-Id        Disp.A | Volatile Uncorr. ECC |
+-----------------------------------------------------------------------------+

Monitoring GPU Utilization in Kubernetes with DCGM Exporter

Deploy DCGM Exporter as a DaemonSet on GPU nodes to expose Prometheus metrics. This creates scrape targets with per-GPU and per-pod metric labels.

apiVersion: apps/v1
kind: DaemonSet
metadata:
  name: dcgm-exporter
  namespace: gpu-monitoring
spec:
  selector:
    matchLabels:
      app: dcgm-exporter
  template:
    metadata:
      labels:
        app: dcgm-exporter
    spec:
      nodeSelector:
        nvidia.com/gpu.present: "true"
      containers:
        - name: dcgm-exporter
          image: nvcr.io/nvidia/k8s/dcgm-exporter:3.3.8-3.6.0-ubuntu22.04
          ports:
            - containerPort: 9400

# HELP DCGM_FI_DEV_GPU_UTIL GPU utilization (in %).
# TYPE DCGM_FI_DEV_GPU_UTIL gauge
DCGM_FI_DEV_GPU_UTIL{gpu="0",UUID="GPU-..."} 78

Viewing GPU Metrics in a DigitalOcean Managed Kubernetes Cluster

To collect GPU metrics in a DOKS cluster, configure Prometheus to scrape the DCGM Exporter DaemonSet, then visualize the data in Grafana or forward it to a hosted monitoring backend. Separate GPU dashboards by node pool and workload labels to avoid mixed tenancy confusion.

Before deployment, review An Introduction to Kubernetes if your team is new to cluster primitives.

scrape_configs:
  - job_name: dcgm-exporter
    static_configs:
      - targets: ['<node-ip>:9400']

In a DOKS cluster, use DaemonSet pod IPs or a Kubernetes Service DNS name instead of static node IP targets. For Grafana dashboard import details, see NVIDIA DCGM Exporter documentation.

Setting Up Persistent GPU Monitoring with Datadog

Use Datadog when you need long-term retention, tag-based slicing, and alert routing to on-call systems. Install the Agent on each GPU node and enable the NVIDIA integration.

Installing the Datadog Agent with NVIDIA GPU Support

Install Agent 7 on the GPU host, then enable the nvidia_gpu integration. Keep host drivers and NVML available to the Agent process.

DD_API_KEY="<YOUR_DATADOG_API_KEY>" DD_SITE="datadoghq.com" bash -c "$(curl -L https://s3.amazonaws.com/dd-agent/scripts/install_script_agent7.sh)"

<$>[note]
The NVML integration is not bundled with Agent 7 by default. Install it separately, then configure nvml.d/conf.yaml.
<$>

sudo datadog-agent integration install -t datadog-nvml==1.0.9

<$>[note]
Verify the latest available version of the NVML integration before installing.
<$>

Configuring the GPU Integration and Tag Strategy

Define tags at the host and integration level so you can group by cluster, environment, and workload type. This keeps alert routing and dashboard filters usable at scale.

init_config:

instances:
  - min_collection_interval: 15
    tags:
      - env:prod
      - role:training
      - gpu_vendor:nvidia

Save this as /etc/datadog-agent/conf.d/nvml.d/conf.yaml, then restart:

sudo systemctl restart datadog-agent

Building a Real-Time GPU Dashboard and Setting Alerts

Create timeseries panels for nvidia.gpu.utilization, nvidia.gpu.memory.used, and nvidia.gpu.temperature, then alert on sustained saturation. A practical first alert is GPU utilization above 95% for 10 minutes on production training nodes.

Use How To Monitor Your Infrastructure with Datadog for dashboard and monitor fundamentals.

Example monitor query:
avg(last_10m):avg:nvidia.gpu.utilization{env:prod,role:training} by {host,gpu_index} > 95

Setting Up GPU Monitoring with Zabbix

To monitor GPU hosts with Zabbix, install the Zabbix agent on each GPU host, import the NVIDIA GPU template, and configure trigger thresholds for utilization and temperature. Zabbix is the right choice when you need self-hosted monitoring with custom alerting and existing enterprise integrations.

Enabling the NVIDIA GPU Template in Zabbix

Import or attach an NVIDIA GPU template in Zabbix, then bind it to hosts that have NVIDIA drivers installed. Template items should poll utilization, memory, temperature, and power.

Path: Data collection -> Templates -> Import
Template: Nvidia by Zabbix agent 2
For some versions, the active mode variant is: Nvidia by Zabbix agent 2 active
Official template source: https://git.zabbix.com/projects/ZBX/repos/zabbix/browse/templates/app/nvidia_agent2

Configuring Triggers for Utilization Thresholds

Create triggers for sustained high utilization, high temperature, and unexpected drops to zero utilization during scheduled training windows. Use trigger expressions with time windows to avoid noise from short spikes.

Example trigger logic using Zabbix agent 2 template item keys:
avg(/GPU Host/nvidia.smi[{#GPUINDEX},utilization.gpu],10m)>95
and
last(/GPU Host/nvidia.smi[{#GPUINDEX},temperature.gpu])>85

{#GPUINDEX} is a low-level discovery macro populated automatically by the template. You do not need to set it manually.

Enabling Unified GPU Usage Monitoring on Windows

Unified GPU Usage Monitoring aggregates activity from multiple GPU engines into a single usage view that operators can read quickly. Enable it through NVIDIA Control Panel first, then verify registry policy where required by your driver profile.

What Unified GPU Usage Monitoring Is

Unified monitoring combines graphics, compute, copy, and video engine activity into one normalized utilization metric. This improves cross-process visibility when mixed workloads run on the same adapter.

How to Enable It via NVIDIA Control Panel and Registry

In NVIDIA Control Panel, enable the GPU activity monitoring feature and apply settings system-wide. If your environment uses managed policy, set the registry value used by your NVIDIA driver branch to turn on unified usage reporting.

Windows Registry example for GPU performance counter visibility:
HKEY_LOCAL_MACHINE\SOFTWARE\NVIDIA Corporation\Global\NVTweak
Value name: RmProfilingAdminOnly (DWORD)
Set to 0 to allow non-admin access to GPU performance counters, set to 1 for admin-only.
Reference: https://developer.nvidia.com/ERR_NVGPUCTRPERM

reg query "HKLM\SOFTWARE\NVIDIA Corporation\Global" /s

<$>[warning]
Registry value names for unified usage reporting vary by driver branch and policy tooling. Validate the exact key and value against your NVIDIA enterprise driver documentation before changing production systems.
<$>

Reading Unified GPU Data via Task Manager and WMI

After enabling unified monitoring, Task Manager can display GPU engine and aggregate usage per process. WMI queries can then be used for scripted collection in Windows-based monitoring workflows.

powershell -Command "Get-Counter '\GPU Engine(*)\Utilization Percentage' | Select-Object -ExpandProperty CounterSamples | Select-Object InstanceName,CookedValue"

InstanceName                                   CookedValue
pid_1204_luid_0x00000000_0x0000_engtype_3D     27.31
pid_1820_luid_0x00000000_0x0000_engtype_Compute_0  74.02

Comparing GPU Monitoring Tools

Use this table to pick a tool based on data depth, operational overhead, and alerting needs. Start with CLI tools for diagnostics, then add Datadog, Zabbix, or DCGM pipelines for persistent monitoring.

Feature and Trade-off Comparison Table

Tool	Platform	Refresh Rate	Per-Process View	Alerting	Cost
nvidia-smi	Linux, Windows	1s+ (`--loop`)	Yes (process list, `pmon`)	No native alerts	Free
nvtop	Linux	Near real time interactive	Yes	No native alerts	Free
gpustat	Linux	1s+ (`--watch`)	Yes (summary)	No native alerts	Free
Glances	Linux, macOS, Windows	1s+	Partial	No native alerts	Free
atop	Linux	Configurable interval	Indirect for GPU	No native alerts	Free
Datadog Agent	Linux, Windows	15s typical agent interval	Yes (tag and host context)	Yes	Paid
Zabbix	Linux, Windows	Configurable polling	Yes (template dependent)	Yes	Free (self-hosted)
DCGM Exporter	Linux, Kubernetes	Scrape interval based	Yes (label dependent)	Via Prometheus/Grafana Alertmanager	Free

Choosing the Right Tool for Your Use Case

For single-node debugging, start with nvidia-smi and nvtop. For fleet-level visibility across GPU Droplets and Kubernetes nodes, use DCGM Exporter plus your monitoring backend or deploy Datadog or Zabbix for retention and alerting.
If you need a historical record of GPU activity alongside CPU, memory, and disk in a single log, atop captures all of these at configurable intervals and is worth adding to long-running training hosts alongside nvidia-smi.

Conclusion

Real-time GPU utilization monitoring is essential for optimizing deep learning performance, troubleshooting bottlenecks, and achieving efficient resource usage—whether running on single nodes, inside containers, or scaling across clustered environments. The right monitoring tool depends on your specific use case: quick one-off checks, interactive debugging, continuous fleet-wide visibility, or long-term metric retention and alerting.

Start with simple tools like nvidia-smi for instant visibility, and progress to dashboarding, custom alerting, and enterprise-grade solutions as your needs grow. With the strategies and tools outlined in this guide, you can proactively monitor, troubleshoot, and maximize the performance of your GPU workloads—ensuring smoother operation for development, training, and deployment pipelines.