DEV Community: DigitalOcean

How to Optimize LLM Pipeline Builds with DSPy

DigitalOcean — Tue, 21 Apr 2026 19:10:39 +0000

This article was originally written by Adrian Payong (AI Consultant and Technical Writer) and Shaoni Mukherjee (AI Technical Writer, DigitalOcean)

Key takeaways

DSPy turns LLM development into a programmable workflow by using signatures, modules, metrics, and optimizers instead of relying on manual prompt tweaking alone.
It is especially useful for production-style pipelines that combine routing, retrieval, reasoning, tool use, structured output, and evaluation inside one maintainable system.
Core DSPy modules such as Predict, ChainOfThought, ReAct, and Module let you build practical applications like QA systems, RAG pipelines, multi-step agents, and classifiers.
DSPy optimizers such as BootstrapFewShot, MIPROv2, and COPRO help improve program quality automatically by tuning instructions and demonstrations against a metric.
For reliable deployment, DSPy works best when paired with evaluation, grounding checks, typed outputs, constraint enforcement, and stable infrastructure such as DigitalOcean for hosting models, retrieval, and agent pipelines.

LLM application development has grown past simple prompt engineering. As systems become more complex, you need a stronger mental model to structure reasoning, retrieval, tool use, evaluation, and optimization within one maintainable workflow. DSPy was designed to help with that. Rather than manually tuning lengthy prompt templates, you define signatures, compose modules, and then optimize the entire program against a metric. This makes LLM development feel less like prompt trial and error and more like building a measurable, improvable software pipeline.

This article covers practical DSPy use cases you will encounter when building production-quality applications. We dive into how DSPy enables question answering, retrieval-augmented generation, multi-step reasoning agents, text classification, and much more. Along the way, you'll learn about DSPy's approach to metric evaluation, assertion-style constraints, and choosing an optimizer. By the end, you should have a clearer view of how DSPy can help you move from isolated prompts to scalable, structured, production-ready LLM pipelines.

What is DSPy and why use it for LLM pipelines?

DSPy's design philosophy is to program declarative LM programs (signatures, modules, and control flow), then compile them towards a metric, rather than manually engineering long prompt templates.

The authors of DSPy reframe this as compiling declarative LM calls into self-improving pipelines, as in the original paper. The compile step searches for better instructions, few-shot demonstrations, (in some modes) fine-tuned weights. Doing DSPy in practice tends to look more like "lightweight ML" than prompt engineering:

Define your interface: a DSPy prompt signature (inputs/outputs + types).
Implement the pipeline logic as modules (DSPy Predict module, DSPy ChainOfThought module, DSPy ReAct module, etc) + Python control flow with dspy.Module.
Define a metric function to measure quality (often calling an LLM for metric evaluation, sometimes via a DSPy "judge" program).
Run an optimizer (previously known as "teleprompters") such as DSPy BootstrapFewShot optimizer or MIPROv2 optimizer to DSPy to improve your score.

Where DSPy fits versus LangChain and LlamaIndex

DSPy is often compared to orchestration frameworks, such as LangChain, and data-centric RAG frameworks, like LlamaIndex. One helpful way to think about their differences is:

LangChain centers around composing chains together, agents, tools, and integrations (extensive tooling for “wiring things together”).
LlamaIndex centers around data ingestion, building indexes, and querying LLM over your data (it's built around RAG-style retrievers + query engines).
DSPy emphasizes programmatic optimization of the LM behavior within your stack: signatures, modules, metrics, and optimizers that can automatically improve your prompts/demos throughout the system.

Many real-world production stacks combine these approaches: use LlamaIndex (or another retriever) to power ingestion and retrieval, then utilize DSPy to wrap the generation and routing logic to optimize prompts and typed outputs.

DSPy core building blocks you will use in this tutorial

Signatures describe what the model should do: input fields, output fields, and their semantic names. Optionally specify types and instructions. Field names are important because they indicate the role (“question” vs “answer”, “context” vs “summary”, etc).

Modules define how to solve it. Key ones:

dspy.Predict: The basic building block that maps inputs → outputs using an LM. Configured by a signature.
dspy.ChainOfThought: A predictor that reasons step-by-step. Outputs are the same as your signature, but with an additional “reasoning” field prepended.
dspy.ReAct: An iterative “Reasoning and Acting” tool-using agent loop where the model chooses tools and produces final outputs.
dspy.Module: the base class for multi-step programs where you implement forward() and compose submodules.

Adapters determine how “structured” your LM I/O is. ChatAdapter is DSPy’s default field-marker format. JSONAdapter forces models that support structured output formatting to emit JSON so that you can reliably parse typed outputs.

Unified end-to-end pipeline example

This code implements a small but realistic “router” program which brings together Predict, RAG + ChainOfThought, and ReAct end-to-end flow:

# pip install -U dspy  (or: pip install -U dspy-ai)
import dspy
from typing import Literal
# 1) Configure the language model once near the top of your app.
lm = dspy.LM("openai/gpt-4o-mini")  # reads OPENAI_API_KEY from env
dspy.configure(lm=lm, adapter=dspy.JSONAdapter())
# 2) A small intent classifier (Predict) to route requests.
class Route(dspy.Signature):
    """Route the user request to the best handler."""
    query: str = dspy.InputField()
    intent: Literal["rag_qa", "tool_agent", "direct_qa"] = dspy.OutputField()

router = dspy.Predict(Route)
# 3) A RAG-style answerer (we'll implement it fully later).
class RagAnswer(dspy.Signature):
    """Answer using only the provided context passages."""
    context: list[str] = dspy.InputField()
    question: str = dspy.InputField()
    answer: str = dspy.OutputField()
    citations: list[int] = dspy.OutputField(desc="indices of context passages used")

rag_answerer = dspy.ChainOfThought(RagAnswer)
# 4) A ReAct agent with tools (we'll implement tools later).
def add(a: float, b: float) -> float:
    return a + b

agent = dspy.ReAct(signature="question -> answer", tools=[add], max_iters=8)
# 5) Tie it together as a program.
class UnifiedAssistant(dspy.Module):
    def forward(self, query: str, retrieved_passages: list[str] | None = None):
        route = router(query=query).intent
        if route == "rag_qa":
            ctx = retrieved_passages or []
            return rag_answerer(context=ctx, question=query)
        if route == "tool_agent":
            return agent(question=query)
        # default: direct QA, still using a CoT-style module for robustness
        direct = dspy.ChainOfThought("question -> answer")
        return direct(question=query)
assistant = UnifiedAssistant()

The above script builds a lightweight DSPy assistant capable of serving multiple types of user queries within a single workflow. After setting up an LLM and JSON adapter, it creates a Predict router that classifies which of three intents a new query belongs to: RAG-based question answering, tool-based agent reasoning, or direct question answering. Queries that require external knowledge are routed to a ChainOfThought RAG module that answers the question given retrieved passages, and returns citations. Queries that require tool usage are routed to a ReAct agent coupled with an add tool; all other queries fall back to a direct ChainOfThought answer module. This program demonstrates how DSPy can orchestrate routing, retrieval, reasoning, and tool use within a single modular assistant.

Use Case 1: Question answering with ChainOfThought

By default, the DSPy ChainOfThought module is designed towards problems where providing intermediate reasoning improves correctness. Let’s consider the following code:

import os
import dspy
from dspy.evaluate import Evaluate
from dspy.evaluate.metrics import answer_exact_match
# Configure once per process.
# (OPENAI_API_KEY must be set in your environment.)
dspy.configure(lm=dspy.LM("openai/gpt-4o-mini"))
# A minimal CoT QA module.
qa_cot = dspy.ChainOfThought("question -> answer")
# A tiny devset (start small, then grow).
devset = [
    dspy.Example(question="What is the capital of France?", answer="Paris").with_inputs("question"),
    dspy.Example(question="What is 2+2?", answer="4").with_inputs("question"),
]
# Metric: exact match on the final answer field.
def em_metric(example, pred, trace=None):
    return answer_exact_match(example, pred)
evaluator = Evaluate(devset=devset, num_threads=2, display_progress=True)
baseline = evaluator(qa_cot, metric=em_metric)
print("Baseline score:", baseline)

This program set up a small DSPy question-answering evaluation pipeline. It initializes DSPy with the openai/gpt-4o-mini model, then defines a simple ChainOfThought module which accepts a question and generates an answer. The program defines a small development dataset consisting of two example question answering pairs and builds an exact-match metric for evaluating the predicted answer against the expected ones. It then launches DSPy's Evaluate utility to apply that module to each question in the dataset in parallel. It computes and outputs the baseline score, indicating how accurately the unoptimized Chain-of-Thought question answering module answered those sample questions.

Improving question answering with BootstrapFewShot

If you only have a few examples, BootstrapFewShot is a good starting point. This optimizer composes demos from labeled examples + bootstrapped demos created by a teacher, filtering to only keep demos that pass your metric.

from dspy.teleprompt import BootstrapFewShot
# A very small trainset is acceptable (DSPy is designed to start small).
trainset = devset
teleprompter = BootstrapFewShot(
    metric=em_metric,
    max_bootstrapped_demos=2,
    max_labeled_demos=2,
)
qa_optimized = teleprompter.compile(student=qa_cot, trainset=trainset)
optimized_score = evaluator(qa_optimized, metric=em_metric)
print("Optimized score:", optimized_score)

Here, we improved the original qa_cot question-answering module with DSPy's BootstrapFewShot optimizer. We use the small trainset as learning examples for better few-shot demonstrations. Then we compiled an optimized version of the model using up to 2 bootstrapped demos + 2 labeled demos. Finally, we run an evaluation on the new model with the same exact-match metric and print out the optimized score to show whether the performance improved over the baseline.

Use Case 2: Retrieval-augmented generation (RAG) pipeline

Retrieval-augmented generation (RAG) solves a major pain point. Without RAG, LLMs can’t access your private or continuously changing knowledge unless you directly supply it at inference time. A typical end-to-end RAG pipeline consists of ingestion/chunking, embeddings, storage + retrieval, and final generation grounded on retrieved documents.

Step-by-step RAG with typed outputs and structured JSON

In the following program, we define a typed signature (lists and ints), use JSONAdapter, and return citations as indices into retrieved passages.

import dspy

# Configure LM with JSONAdapter so lists (like citations)
# are parsed reliably from model output.
lm = dspy.LM("openai/gpt-4o-mini")  # reads OPENAI_API_KEY from env
dspy.configure(lm=lm, adapter=dspy.JSONAdapter())

# Minimal local corpus for demo; replace with your documents or a vector DB.
corpus = [
    "Linux divides memory into regions; on 32-bit systems highmem is not permanently mapped.",
    "Low memory is directly addressable by the kernel; high memory is mapped on demand.",
    "Unrelated passage about iPhone apps.",
]

# Embedder for dense retrieval.
embedder = dspy.Embedder("openai/text-embedding-3-small", dimensions=512)
search = dspy.retrievers.Embeddings(embedder=embedder, corpus=corpus, k=2)


class RagAnswer(dspy.Signature):
    """Answer using only the provided context passages."""
    context: list[str] = dspy.InputField(desc="retrieved passages")
    question: str = dspy.InputField()
    answer: str = dspy.OutputField(desc="final answer grounded in context")
    citations: list[int] = dspy.OutputField(desc="indices of context passages used")


class RAG(dspy.Module):
    def __init__(self):
        super().__init__()
        self.respond = dspy.ChainOfThought(RagAnswer)

    def forward(self, question: str):
        # Retrieve top‑k passages.
        retrieved = search(question)
        ctx = retrieved.passages

        # Generate answer and citations.
        pred = self.respond(context=ctx, question=question)

        # Lightweight validation of citations indices.
        citations = pred.citations or []
        pred.citations = [i for i in citations if 0 <= i < len(ctx)]

        # Return a structured prediction.
        return dspy.Prediction(
            context=ctx,
            answer=pred.answer,
            citations=pred.citations,
            reasoning=pred.reasoning,
        )

# Instantiate the RAG module.
rag = RAG()

# Run a demo question.
out = rag(question="What are high memory and low memory in Linux?")

print("Answer:")
print(out.answer)
print("\nCitations (indices into context):")
print(out.citations)

Here we retrieve information from a small knowledge base in order to answer a question. The language model is configured with JSONAdapter to properly parse structured output (citation lists). An embedding-based retriever is created to find the most relevant passages from the corpus. Typed Signature defines a structured RAG task with fields for context, question, answer, and citations. The RAG module follows ChainOfThought to produce a grounded answer from the retrieved passages. Lastly, the citation indices are checked for validity before returning structured prediction, and a demo query is run about Linux memory.

Add a RAG metric that checks both correctness and grounding

Here's a small example of a composite metric. It checks if the label matches and whether the predicted answer was found in the retrieved context. It returns a float for evaluation and a boolean for bootstrapping.

from dspy.evaluate import Evaluate
def grounded_answer_metric(example, pred, trace=None):
    # Case‑insensitive exact or near‑exact match on answer.
    answer_match = example.answer.lower() in pred.answer.lower()
    # Answer should appear in at least one retrieved passage.
    context_match = any(pred.answer.lower() in c.lower() for c in pred.context)
    if trace is None:
        # For evaluation: soft score between 0 and 1.
        return (answer_match + context_match) / 2.0
    # For bootstrapping / optimization: require both.
    return answer_match and context_match

devset = [
    dspy.Example(
        question="What is low memory in Linux?",
        answer="directly addressable by the kernel",
    ).with_inputs("question")
]

evaluator = Evaluate(devset=devset, num_threads=2, display_progress=True)
print(evaluator(rag, metric=grounded_answer_metric))

This code computes a custom metric to score how well a DSPy RAG pipeline is answering a question with grounded answers. grounded_answer_metric checks two things: 1) whether the predicted matches the expected answer, and 2) whether that answer can be grounded in the retrieved context passages. Then, Evaluate runs that metric on a small development set to validate whether your RAG pipeline returns grounded, correct answers before using it for optimization or production.

Optimize the RAG program with MIPROv2

Here we use DSPy’s MIPROv2 optimizer to improve the original RAG program against your custom grounding metric, then recompile the module with a small demo set and evaluate whether the optimized version performs better.

from dspy.teleprompt import MIPROv2
# Set up MIPROv2 optimizer with your custom metric.
tp = MIPROv2(
    metric=grounded_answer_metric,
    auto="light",          # or "medium" / "heavy"
    num_threads=4,
)
# Compile the original RAG module using the dev/train set.
rag_optimized = tp.compile(
    rag,
    trainset=devset,
    max_bootstrapped_demos=2,
    max_labeled_demos=2,
)
# Re‑evaluate the optimized RAG module.
print("Evaluation after MIPROv2 optimization:")
print(evaluator(rag_optimized, metric=grounded_answer_metric))

Use Case 3: Multi-Step reasoning agent with ReAct

When you have tasks that require tool use (whether that's doing calculations, calling internal APIs, fetching knowledge, or taking actions), DSPy provides dspy.ReAct, which implements the ReAct ("Reasoning and Acting") paradigm: the model reasons, chooses which tool to call, observes the results, and repeats until it can output final answers. ReAct can be generalized to function over any signature. It can accept either functions or dspy.Tool objects as tools.

A minimal ReAct agent with typed tools

The script below implements a small DSPy ReAct agent that answers questions by utilizing tools as needed. It sets up an LLM, defines two tools - one that returns the current UTC time and another that multiplies numbers - and passes those tools to dspy.ReAct. The agent will reason if it should use a tool, call it if needed, and then return the final answer.

import dspy
from datetime import datetime, timezone
dspy.configure(lm=dspy.LM("openai/gpt-4o-mini"), adapter=dspy.JSONAdapter())

def utc_now() -> str:
    return datetime.now(timezone.utc).isoformat()

def multiply(a: float, b: float) -> float:
    return a * b

# Create a ReAct agent that can use utc_now and multiply.
agent = dspy.ReAct(
    signature="question -> answer",
    tools=[utc_now, multiply],
    max_iters=6,
)
# Example queries.
print(agent(question="What time is it in UTC right now?"))
print(agent(question="What is 19.5 * 4.2?"))

Production concern: agent reliability, costs, and guardrails

Agent loops can silently accumulate high costs (repeated LLM calls, repeated tool calls) or hallucinate invalid actions without guardrails and observability. A reasonable set of guardrails includes cap iterations (max_iters), tightening tool schemas and permissions, and validating on real traffic-like prompts before rollout.

Optimize a ReAct agent with DSPy optimizers

DSPy optimizers can optimize entire programs, including end-to-end complex multi-module systems (such as agents, retrieval, and extraction), as long as you specify a metric to improve. For many teams, a pattern that works well is:

Bootstrap a few demos with BootstrapFewShot(cheap);
Then, run MIPROv2 in auto="light" or auto="medium" depending on budget.

Use Case 4: Text classification with LLM metric evaluation

Classification is an ideal DSPy use case because while success metrics (accuracy, F1) are straightforward, you can still take advantage of DSPy’s programmatic structure, typed outputs, and optimizers.

Build a typed classifier with Predict

Here’s code that builds a simple DSPy text classifier for support tickets. It sets up the model, declares a signature with one input (ti*cket*) and one constrained output (label), then calls dspy.Predict to classify the ticket as one of four types: billing, bug, feature, or security. In this example, the “I was charged twice” complaint is correctly classified as billing.

import dspy
from typing import Literal
dspy.configure(lm=dspy.LM("openai/gpt-4o-mini"), adapter=dspy.JSONAdapter())
class TicketLabel(dspy.Signature):
    """Classify a support ticket into a fixed taxonomy."""
    ticket: str = dspy.InputField()
    label: Literal["billing", "bug", "feature", "security"] = dspy.OutputField()
clf = dspy.Predict(TicketLabel)
example = clf(ticket="I was charged twice for my subscription this month.")
print(example.label)

Evaluate with a metric (and optionally build an LLM-judge metric)

Metrics are ordinary Python functions. They should follow the signature(example, pred, trace=None); for complex outputs, metrics can use AI feedback via additional predictor calls.

The code below uses DSPy’s Evaluate utility to test a classifier, clf, on a small labeled dataset of support tickets. The trainset has three examples. For each example, the text of a ticket is labeled with the correct category (billing, bug, or feature). Passing .with_inputs("ticket") specifies to DSPy that the model should only receive the ticket text as input. The accuracy_metric function checks if the classifier's predicted label matches the true label. It returns 1.0 if the prediction is correct and 0.0 otherwise. Evaluate runs clf on the dataset with 2 threads, displays progress while running, and print(evaluator(clf, metric=accuracy_metric)) prints the final result, which is usually the accuracy of the model on those examples.

from dspy.evaluate import Evaluate
trainset = [
    dspy.Example(ticket="I was charged twice.", label="billing").with_inputs("ticket"),
    dspy.Example(ticket="The app crashes on launch.", label="bug").with_inputs("ticket"),
    dspy.Example(ticket="Please add export to CSV.", label="feature").with_inputs("ticket"),
]
def accuracy_metric(example, pred, trace=None):
    return float(example.label == pred.label)
evaluator = Evaluate(devset=trainset, num_threads=2, display_progress=True)
print(evaluator(clf, metric=accuracy_metric))

Assertion testing and constraint enforcement in modern DSPy

In production, people often ask for “verification” operations: ("assertion testing"; the label must be one of X; JSON must parse; citations must be in range).

dspy.Refine was purpose-built to be a best-of-N refinement loop with reward_fn and threshold. It repeatedly calls the module N times and returns the best prediction, generating feedback between attempts if necessary. Here's a real-world “constraint enforcement” wrapper: retry until output taxonomy is respected. Let’s consider the following code:

import dspy
from typing import Set
allowed: Set[str] = {"billing", "bug", "feature", "security"}
def label_is_valid(args, pred):
    return 1.0 if pred.label in allowed else 0.0
robust_clf = dspy.Refine(module=clf, N=3, reward_fn=label_is_valid, threshold=1.0)
print(robust_clf(ticket="Please add SSO support.").label)

This code wraps the original classifier with dspy.Refine, which allows DSPy to retry up to 3 times and retain only outputs that passed reward_fn. The reward function ensures the predicted label is one of our allowed categories, and the threshold=1.0 means only a fully valid label will be accepted before returning the result.

Choosing the Right DSPy Optimizer

DSPy now refers to these algorithms as optimizers (previously teleprompters). According to the optimizer documentation, an optimizer is an algorithm that tunes a DSPy program’s parameters (prompts and/or LM weights) to maximize your metrics using your program, metric, and training inputs. The training inputs are often a small set of examples.

Practical decision criteria

This table lists the 3 optimizers your brief prioritizes—BootstrapFewShot, MIPROv2, and COPRO—as well as BootstrapFewShotWithRandomSearch, which DSPy recommends after you have more data.

Optimizer	What it does and when to use it	Data guidance and key config knobs
BootstrapFewShot	Tunes few-shot demos assembled from labeled and bootstrapped examples validated by the metric. It works well for fast wins on small datasets and is a strong first compile option.	Start here when you have around 10 examples. Knobs: `max_labeled_demos`, `max_bootstrapped_demos`, `teacher_settings`
BootstrapFewShotWithRandomSearch	Tunes few-shot demos like BootstrapFewShot, but tests multiple candidate demo sets and keeps the best one. It is better for a more robust few-shot selection while staying relatively simple.	Best when you have around 50 or more examples. Knobs: `num_candidate_programs`, plus the BootstrapFewShot knobs
COPRO	Tunes prompt instructions through iterative search, documented as coordinate ascent in the optimizer guide. It is useful when you want instruction tuning without focusing heavily on demos.	Usually needs a train set and a metric. Knobs: `breadth`, `depth`, `init_temperature`
MIPROv2	Jointly tunes instructions and few-shot examples using Bayesian optimization. It is the strongest choice when you want higher-quality prompt optimization and have enough budget and data.	Best for longer runs, such as 40 or more trials, with around 200 or more examples to reduce overfitting risk. Knobs: `auto` (“light/medium”), `num_threads`, plus demo knobs in `compile()`

Running DSPy on DigitalOcean

Deployment should provide you with two things: (1) infrastructure to run your DSPy program (stable runtime) and (2) access to LLMs you can reliably call to run retrieval and add guardrails.

Deployment patterns that map well to DSPy pipelines

Deploy your DSPy service to a Virtual Machine (VM) or GPU instance if you want full control of everything in your stack (vector DB, embeddings, model runtime). Building a RAG application on GPU Droplets is covered in step-by-step detail with DigitalOcean’s RAG tutorials.

Use a fully managed model access for simpler operations. The DigitalOcean Gradient platform describes serverless inference (no infrastructure management) and API access to models hosted by major vendors (OpenAI, Anthropic, etc) as well as managed scalability and security features for open-source models hosted directly in-platform.
Build agentic apps with managed agent features. DigitalOcean’s Gradient AI Platform quickstart describes fully managed agents with knowledge bases for retrieval-augmented generation, multi-agent routing, and guardrails.

Conclusion

DSPy represents a meaningful shift in how modern LLM systems are built. Instead of viewing prompts as static strings, DSPy treats them as components of a larger program composed of signatures, modules, metrics, and control flow. This approach really shines when you graduate from simple completions to authoring tangible application patterns such as ChainOfThought QA, RAG with structured outputs, ReAct-based tool use, and classification pipelines with integrated quality checks.

The larger point here is that DSPy isn’t simply a playground for prompt engineering. DSPy is a practical foundation for building, validating, iterating, and scaling your LLM systems with more rigor. As engineering teams require better guarantees around reliability, observability, and control over agentic behavior, DSPy will be ready to take on a larger role in production AI stacks. The future will belong to those engineers who build LLM workflows that are modular, testable, and optimization-driven from the start.

References

Tutorial: Build an AI-Powered GPU Fleet Optimizer

DigitalOcean — Fri, 17 Apr 2026 19:00:00 +0000

This article was originally written by Shamim Raashid (Senior Solutions Architect) and Anish Singh Walla (Senior Technical Content Strategist and Team Lead)

Key Takeaways

Deploy a serverless LangGraph agent on the DigitalOcean Gradient AI Platform that monitors your GPU fleet using natural language queries.
Scrape real-time NVIDIA DCGM metrics (temperature, power, VRAM, engine utilization) from GPU Droplets over Prometheus-style endpoints on port 9400.
Detect idle and underutilized GPUs automatically by defining configurable threshold dictionaries that compare live metrics against your baseline workload patterns.
Customize the blueprint to your needs: Change target Droplet types, adjust idle detection thresholds, enrich the data payload with additional metrics, and add actionable tools like automated power-off commands.
Reduce GPU cloud costs by replacing reactive dashboard monitoring with a proactive AI agent that identifies waste the moment it starts.

Managing a GPU fleet in the cloud is a constant balancing act between performance and cost. A single idle GPU Droplet left running overnight can add hundreds of dollars to your monthly bill. Traditional monitoring dashboards surface raw metrics, but they still require a human to interpret whether a machine is “working” or “wasting money.”

This tutorial walks you through building an AI-powered GPU fleet optimizer using the DigitalOcean Gradient AI Platform and the Agent Development Kit (ADK). You will deploy a serverless, natural-language AI agent that audits your GPU infrastructure in real time, scrapes NVIDIA DCGM (Data Center GPU Manager) metrics like temperature, power draw, VRAM usage, and engine utilization, and flags idle resources before they inflate your cloud bill.

This blueprint is designed to be forked and customized. By the end of this guide, you will know how to tune the agent's personality and efficiency thresholds, add new monitoring tools, and deploy the agent as a production-ready serverless endpoint.

Reference repository

You can view the complete blueprint code here: dosraashid/do-adk-gpu-monitor.

Prerequisites

DigitalOcean Account: With at least one active GPU Droplet running.
DigitalOcean API Token: A Personal Access Token with read permissions and GenAI scopes.
Gradient Model Access Key: Generated from the Gradient AI Dashboard.
Python 3.12: Recommended for the latest LangGraph and asyncio features.
Familiarity with Python, REST APIs, and Linux command-line basics.

The challenge: “Invisible” cloud waste

When scaling AI workloads, engineering teams often spin up expensive, specialized GPU Droplets (like NVIDIA H100s or H200s) for training or inference tasks.

The Problem: Hidden costs and wasted resources

Once a training script finishes or a model endpoint stops receiving traffic, the Droplet itself remains online and billing by the hour. This creates two compounding issues:

Generic monitoring falls short: Standard cloud dashboards typically show host-level metrics like CPU and RAM. A machine learning node might report 1% CPU utilization, but those monitors do not reveal whether the GPU's VRAM is empty or whether the compute engine is completely idle.
Dashboard fatigue: Even if you install specialized tools like Grafana to track NVIDIA DCGM metrics, an engineer still has to remember to log in, interpret the charts, and manually map the IP address of an idle node back to a specific cloud resource to shut it down.

The Solution: A proactive AI fleet analyst

Instead of waiting for an engineer to check a dashboard, you can build an AI agent that acts as an autonomous infrastructure analyst.

Using the DigitalOcean Gradient ADK, you will deploy a Large Language Model (LLM) equipped with custom Python tools. When you ask the agent a question like, “Are any of my GPUs wasting money right now?”, it executes a multi-step reasoning loop:

Discovery: Calls the DigitalOcean API to get a live inventory of your Droplets.
Interrogation: Pings the NVIDIA DCGM exporter on each node's public IP to read VRAM, temperature, and engine load.
Analysis: Runs those raw metrics against a threshold dictionary you define (e.g., “If VRAM usage is below 5% and engine utilization is below 2%, mark this GPU as IDLE”).
Actionable Output: Replies in plain English, naming the specific node, its current hourly cost, and the exact metrics proving it is idle.

Understanding NVIDIA DCGM metrics for GPU monitoring

NVIDIA Data Center GPU Manager (DCGM) exposes hardware telemetry through a Prometheus-compatible exporter that runs on port 9400.

Metric	What It Measures	Why It Matters
`DCGM_FI_DEV_GPU_TEMP`	GPU die temperature in Celsius	High temperatures indicate active computation.
`DCGM_FI_DEV_POWER_USAGE`	Current power draw in watts	Idle GPUs draw significantly less power than busy ones.
`DCGM_FI_DEV_FB_USED`	Framebuffer (VRAM) memory in use	Empty VRAM means no models are loaded.
`DCGM_FI_DEV_GPU_UTIL`	GPU engine utilization percentage	The most direct indicator of compute work.

You can query these metrics directly:

curl -s http://<DROPLET_PUBLIC_IP>:9400/metrics | grep -E "DCGM_FI_DEV_GPU_TEMP|DCGM_FI_DEV_POWER_USAGE|DCGM_FI_DEV_FB_USED|DCGM_FI_DEV_GPU_UTIL"

The AI agent in this blueprint automates this scraping across your entire fleet, parses the Prometheus text format, and feeds the structured data into the LLM for analysis. If DCGM is not available on a particular node (for example, because the exporter is not installed or port 9400 is blocked by a firewall), the agent falls back to standard CPU and RAM metrics and reports “DCGM Missing” for that node.

For production deployments, consider pairing DCGM data collection with a full Prometheus and Grafana monitoring stack for historical trend analysis alongside the AI agent’s real-time assessments.

Step 1: Clone the blueprint and set up your environment

Start with the foundational repository rather than writing everything from scratch.

Clone the repo and set up your Python environment:

git clone https://github.com/dosraashid/do-adk-gpu-monitor
cd do-adk-gpu-monitor
python3.12 -m venv venv
source venv/bin/activate
pip install -r requirements.txt

Configure your secrets by creating a .env file in the root directory:

DIGITALOCEAN_API_TOKEN="your_do_token"
GRADIENT_MODEL_ACCESS_KEY="your_gradient_key"

Security note: Never commit .env files to version control. The repository’s .gitignore already excludes this file.

Step 2: How it works (the architecture)

Before you customize the blueprint, it helps to understand the data flow inside the code:

User Prompt: You ask the agent a question via the /run endpoint.
LangGraph State: The agent checks its conversation memory (thread_id) via MemorySaver, which enables multi-turn follow-up questions within the same session.
Tool Execution: The LLM decides to call @tool def analyze_gpu_fleet() defined in main.py.
Parallel Scraping: analyzer.py uses Python’s ThreadPoolExecutor to query the DigitalOcean API and each Droplet’s DCGM endpoint (metrics.py) concurrently. This parallel approach prevents network bottlenecks when monitoring dozens of nodes.
The Omniscient Payload: The analyzer packages all raw data (temperature, power, VRAM, RAM, CPU, cost) into a structured JSON dictionary that the LLM can reason about.
Synthesis: The LLM reads the JSON payload and responds in natural language with specific node names, costs, and actionable recommendations.

If you want to learn more about building stateful AI agents with LangGraph, follow the Getting Started with Agentic AI Using LangGraph tutorial.

Step 3: Customizing the blueprint to your needs

This repository is built to be forked and modified. Here are the four main areas you should adjust to match your organization’s requirements.

Customization 1: Tuning the logic (config.py)

Open config.py. This is the control center for your agent’s behavior.

The Persona: Edit AGENT_SYSTEM_PROMPT to change how the AI communicates. For a highly technical DevOps assistant, remove the emojis and instruct it to output raw bullet points. For a management-facing report, tell it to summarize in cost terms.
The Thresholds: The blueprint considers a GPU “Idle” when utilization falls below 2% by default. If your baseline workloads idle at a higher percentage, adjust the THRESHOLDS dictionary:

THRESHOLDS = {
    "gpu": {
        "max_temp_c": 82.0,
        "max_util_percent": 95.0,
        "max_vram_percent": 95.0,

        "idle_util_percent": 2.0,
        "idle_vram_percent": 5.0,

        "optimized_util_percent": 40.0,
        "optimized_vram_percent": 50.0,
    },
    "system": {
        "idle_cpu_percent": 3.0,
        "idle_ram_percent": 15.0,
        "idle_load_15": 0.5,

        "starved_cpu_percent": 85.0,
        "starved_ram_percent": 90.0,

        "optimized_cpu_percent": 40.0,
        "optimized_ram_percent": 50.0,
    },
}

For example, if your inference servers typically idle at 8% GPU utilization between request bursts, set idle_util_percent to 10.0 to avoid false positives.

Customization 2: Changing the target infrastructure (analyzer.py)

By default, the blueprint only scans Droplets with "gpu" in the size_slug to reduce unnecessary API calls. Open analyzer.py and locate the slug filter. If you want the agent to monitor CPU-optimized or standard Droplets, modify this line:

# Change "gpu" to "c-" for CPU-Optimized, or remove the filter entirely to scan all Droplets.
target_droplets = [d for d in all_droplets if "gpu" in d.get("size_slug", "").lower()]

Customization 3: Enriching the omniscient payload (analyzer.py and metrics.py)

The LLM only knows what you explicitly pass to it. The default payload includes temperature, power, and VRAM data. If you install Prometheus Node Exporter on your instances and want the AI to also analyze disk space, you would:

Update metrics.py to scrape disk metrics from Node Exporter on port 9100.
Update the return dictionary at the bottom of process_single_droplet in analyzer.py to include the new field:

return {
    "droplet_id": droplet_id,
    "gpu_temp": temp_val,
    "gpu_power": power_val,
    "vram_used": vram_val,
    "disk_space_free_gb": disk_val,  # New metric
}

Customization 4: Adding actionable tools (main.py)

The default blueprint is read-only. The most powerful upgrade is giving the AI permission to act on your infrastructure. In main.py, you can add a new function with the @tool decorator that uses the DigitalOcean API to power off a specific Droplet:

@tool
def power_off_droplet(droplet_id: str) -> str:
    """Power off a Droplet by ID. Use only when the user explicitly asks to stop an idle node."""
    import requests
    import os

    token = os.getenv("DIGITALOCEAN_API_TOKEN")
    response = requests.post(
        f"https://api.digitalocean.com/v2/droplets/{droplet_id}/actions",
        headers={
            "Authorization": f"Bearer {token}",
            "Content-Type": "application/json",
        },
        json={"type": "power_off"},
    )
    if response.status_code == 201:
        return f"Successfully sent power-off command to Droplet {droplet_id}."
    return f"Failed to power off Droplet {droplet_id}: {response.status_code} {response.text}"

After adding any new tools, bind them to the LLM so the agent can invoke them:

llm_with_tools = llm.bind_tools([analyze_gpu_fleet, power_off_droplet])

Warning: Giving an AI agent write access to your infrastructure requires careful guardrails. Consider adding confirmation prompts, restricting which Droplet tags the agent can act on, and logging all actions for audit purposes.

Step 4: Testing your custom agent

Once you have tailored the code, test it locally before deploying. Start the local development server:

gradient agent run

In a separate terminal, simulate user requests using curl.

Example 1: Deep diagnostic

curl -X POST http://localhost:8080/run \
     -H "Content-Type: application/json" \
     -d '{
           "prompt": "Give me a full diagnostic on my GPU nodes including temperature and power.",
           "thread_id": "audit-session-1"
         }'

Expected Output: The AI uses the Omniscient Payload to report exact temperatures, wattage, and RAM utilization for each GPU Droplet, alongside cost-saving recommendations for any idle nodes.

Example 2: Contextual memory

Because you are passing thread_id: "audit-session-1", the agent retains conversation context. You can ask follow-up questions without triggering a full re-scan of your infrastructure:

curl -X POST http://localhost:8080/run \
     -H "Content-Type: application/json" \
     -d '{
           "prompt": "Which of those nodes was the most expensive?",
           "thread_id": "audit-session-1"
         }'

Example 3: Thread isolation

The memory is strictly scoped by thread_id. A request with a different thread ID sees no prior history and starts a fresh conversation:

curl -X POST http://localhost:8080/run \
     -H "Content-Type: application/json" \
     -d '{
           "prompt": "What was the second question I asked you?",
           "thread_id": "audit-session-2"
         }'

Expected Output: The agent responds that it has no record of previous questions in this session, confirming that thread isolation is working correctly.

Step 5: Cloud deployment:

Once you are satisfied with your customizations, deploy the agent as a serverless endpoint on the DigitalOcean Gradient AI Platform:

gradient agent deploy

You will receive a public endpoint URL that you can integrate into Slack bots, internal dashboards, CI/CD pipelines, or any HTTP client. The Gradient platform handles scaling, so your agent can serve multiple concurrent users without manual infrastructure management.

For more details on building and deploying agents with the ADK, see How to Build Agents Using ADK.

GPU fleet cost optimization: When to use an AI agent vs. static dashboards

One of the most common questions teams face when setting up GPU monitoring is whether to build a custom AI agent or rely on traditional dashboard tooling. The right choice depends on your fleet size, the complexity of your workloads, and how quickly you need to act on idle resources.

Factor	Static Dashboards (Grafana + Prometheus)	AI Agent (This Blueprint)
Setup complexity	Moderate: requires Prometheus server, Grafana, and DCGM exporter configuration	Low: clone the repo, set env vars, deploy with `gradient agent deploy`
Real-time alerting	Rule-based alerts with fixed thresholds	Natural language queries with adaptive reasoning
Multi-metric correlation	Manual: you visually compare multiple charts	Automatic: the LLM correlates temperature, power, VRAM, and cost in a single response
Actionability	Read-only dashboards; separate automation needed	Extensible with `@tool` decorator for direct API actions
Conversational follow-ups	Not supported	Built-in via LangGraph `MemorySaver` and `thread_id` scoping
Best for	Large teams with dedicated SRE/DevOps staff and historical trend analysis	Small-to-mid teams that need fast, conversational GPU auditing without building dashboard infrastructure

For teams running fewer than 20 GPU Droplets, the AI agent approach eliminates the overhead of maintaining a full monitoring stack while still providing actionable insights. For larger fleets, consider running both: use Prometheus and Grafana for long-term trend storage and the AI agent for on-demand, conversational diagnostics.

Advantages and tradeoffs

When adapting this blueprint for production, keep these architectural considerations in mind:

Contextual intelligence: LangGraph’s MemorySaver gives the agent conversation history, allowing natural drill-down investigations. You can ask “Which node is idle?” followed by “How much is it costing me per hour?” without repeating context.
Parallel processing: The analyzer uses Python’s ThreadPoolExecutor to scan dozens of Droplets concurrently, preventing the LLM from timing out while waiting for sequential network calls.
Cost justification: If the AI agent spots a single idle $500/month GPU instance, it pays for itself many times over. The inference cost of running a single diagnostic query on the Gradient platform is negligible compared to the savings.
Graceful degradation: If the DCGM metric scraper cannot reach port 9400 (for example, because of firewall rules or the exporter not being installed), the agent reports “DCGM Missing” for that node and falls back to standard CPU and RAM metrics rather than failing entirely.
Security considerations: The agent requires a DigitalOcean API token with read permissions. If you add write tools (like the power_off_droplet example), scope the token’s permissions carefully and implement audit logging.

Conclusion

You have successfully deployed a multi-tool AI agent using the DigitalOcean Gradient AI Platform that transforms raw infrastructure metrics into conversational, actionable intelligence. By combining DigitalOcean API data with real-time NVIDIA DCGM telemetry and an LLM reasoning engine, you have built a system that addresses three major operational challenges:

1. Stopping the silent budget drain

The most immediate value this agent delivers is catching “forgotten resources.” When engineers spin up GPU Droplets for experiments or temporary training runs, those instances often continue billing long after the work is done. Standard CPU monitors might show background processes at 1%, making the instance look active.

By querying the NVIDIA DCGM exporter directly for engine and VRAM utilization, the AI agent cuts through that noise. It identifies premium GPU nodes that are doing no meaningful compute work, letting you stop the financial drain before it compounds.

2. Eliminating dashboard fatigue

In a traditional workflow, diagnosing a cloud infrastructure issue means opening the DigitalOcean Control Panel to check Droplet status, switching to Grafana to review DCGM metrics, and consulting an architecture diagram to remember what each node is responsible for.

This agent consolidates that entire workflow. Using LangGraph’s conversational memory and the Omniscient Payload, you ask a single question and receive a complete summary of host details, GPU temperature, power usage, and cost impact in one response.

3. Bridging observability and action

Traditional dashboards are read-only. They can alert you that a resource is idle, but they do not provide the tools to act on that information.

Because this blueprint is built on the Gradient ADK, the agent is inherently extensible. By adding a few lines of Python using the @tool decorator, you can upgrade this agent from a passive monitor into an active operator that executes API commands to power off idle nodes, resize underutilized instances, or trigger scaling events automatically.

The do-adk-gpu-monitor repository is your starting point. Clone the code, adjust the efficiency thresholds to match your specific workloads, and start having conversations with your infrastructure today.

Reference and resources

Ready to take your GPU fleet management and AI agent development further? Explore these resources:

DigitalOcean Gradient AI Platform Documentation: Full reference for deploying and managing AI agents, models, and inference endpoints.
How to Build Agents Using ADK: Step-by-step guide to creating custom agents with the Agent Development Kit.
Getting Started with Agentic AI Using LangGraph: Learn the fundamentals of building stateful, multi-step AI agents with LangGraph.
Stable Diffusion on DigitalOcean GPU Droplets: Run GPU-accelerated AI workloads on DigitalOcean GPU Droplets.
Scaling Gradient with GPU Droplets and Networking: Architect production GenAI deployments with GPU Droplets, global load balancers, and VPC networking.

A Complete Guide to Real-Time GPU Usage Monitoring

James Skelton — Wed, 15 Apr 2026 16:30:00 +0000

The fastest way to monitor GPU utilization in real time on Linux is to run nvidia-smi --loop=1, which refreshes GPU stats every second including core utilization, VRAM usage, temperature, and power draw.

Monitoring GPU utilization in real time starts with nvidia-smi, then expands to per-process views, container metrics, and alerts for long-running jobs. This guide shows command-level workflows you can run on Ubuntu, GPU Droplets, Docker hosts, and Kubernetes clusters.

If you are building or operating deep learning systems, pair this guide with How To Set Up a Deep Learning Environment on Ubuntu and DigitalOcean GPU Droplets.

Key Takeaways

Use nvidia-smi --loop=1 for the fastest host-level real-time GPU check on Linux.
Use nvidia-smi pmon -s um to identify which PID is using GPU cores and GPU memory bandwidth.
For terminal dashboards, use nvtop for interactive drill-down and gpustat for lightweight snapshots.
In containers and Kubernetes, expose metrics through NVIDIA runtime support and DCGM Exporter.
Persistent alerting belongs in monitoring platforms such as Datadog Agent or Zabbix templates.
GPU memory utilization and GPU core utilization are separate signals, high memory with low cores is common in input-stalled jobs.
On Windows, Unified GPU Usage Monitoring aggregates engine activity and surfaces it in Task Manager and WMI.

What GPU Utilization Metrics Actually Mean

GPU utilization metrics tell you whether your job is compute-bound, memory-bound, input-bound, or idle between batches. Start by tracking core utilization, memory usage, memory controller load, temperature, and power draw together instead of looking at one metric in isolation.

GPU Core Utilization vs. Memory Utilization

GPU core utilization is the percentage of time kernels are actively executing on SMs during the sampling window. GPU memory utilization in nvidia-smi usually refers to memory controller activity, while memory usage is allocated VRAM in MiB.

Low core utilization with high allocated VRAM often means the model is resident but waiting on data or synchronization. High core utilization with low memory controller activity is more common in compute-heavy kernels.

SM Utilization, Memory Bandwidth, and Power Draw

SM utilization tells you whether CUDA cores are busy, memory bandwidth indicates how hard memory channels are being driven, and power draw shows electrical load relative to the card limit. These three together explain why two workloads with similar utilization percentages can perform differently.

Use power.draw, power.limit, and utilization metrics in the same sample window when tuning batch size and dataloader workers. If power is capped while utilization is high, clock throttling can be the next bottleneck to investigate.

Why These Metrics Matter for Deep Learning Workloads

These metrics matter because training throughput is gated by the slowest stage in the pipeline. If GPU cores are idle while CPU or storage is saturated, adding another GPU will not fix throughput.

<$>[note]
For a practical environment baseline before tuning, follow How To Set Up a Deep Learning Environment on Ubuntu.
<$>

GPU Bottlenecks and Out of Memory Errors

Most GPU incidents in ML pipelines come from input bottlenecks or VRAM pressure. Diagnose both at the same time by sampling GPU, CPU, and process-level memory while a real training job is running.

CPU Preprocessing Bottlenecks

If CPU preprocessing is the bottleneck, GPU utilization drops between mini-batches even when VRAM remains allocated. This pattern appears when image decode, augmentation, or tokenization is slower than kernel execution.

Check host pressure while your training loop runs:

top

vmstat 1

procs -----------memory---------- ---swap-- -----io---- -system-- ------cpu-----
r  b   swpd   free   buff  cache   si   so    bi    bo   in   cs us sy id wa st
2  0      0 824320  74384 901212    0    0     6    10  420  980 18  4 76  2  0

In vmstat, watch r, wa, bi, and us plus sy together. r is runnable processes, and if it stays above your CPU core count, the CPU is saturated. wa is CPU time waiting on I/O, and sustained values above 10 to 15 during training often mean dataloader workers are blocked on disk reads. bi is blocks received from storage, and high bi with high wa points to storage bottlenecks instead of compute. us + sy is total active CPU time, and if it is high while GPU-Util is low, preprocessing is outrunning the GPU. If wa is high, increase dataloader workers or switch to faster storage. If us + sy is high with low GPU-Util, move transforms to GPU with a library such as Kornia.

What Causes OOM Errors and How to Resolve Them

OOM errors happen when requested allocations exceed available VRAM, often due to large batch sizes, long sequence lengths, or concurrent GPU processes. Resolve OOM by lowering memory pressure first, then increasing workload cautiously.

Common fixes:

Reduce batch size or sequence length.
Use gradient accumulation to keep effective batch size.
Enable mixed precision where supported.
Terminate stale GPU processes before restart.
Move expensive transforms to more efficient pipeline stages.

If a stale process is still holding VRAM after a failed run, list active compute processes, verify ownership, terminate the stale PID, then confirm memory was released.

nvidia-smi --query-compute-apps=pid,used_memory,process_name --format=csv,noheader

18211, 17664 MiB, python
18304, 512 MiB, python

ps -p <PID> -o pid,user,etime,cmd

kill -9 <PID>

<$>[warning]
Do not kill unknown PIDs on shared hosts. Verify process ownership and job context first.
<$>

nvidia-smi # Confirm VRAM is now released

Monitoring GPU Utilization with nvidia-smi

nvidia-smi is the fastest built-in tool for real-time GPU telemetry on Linux servers. It is available with NVIDIA drivers and documents fields used by most higher-level integrations.

Reference docs:

Basic nvidia-smi Output and What Each Field Shows

Run nvidia-smi with no flags for a full snapshot of GPU and process state. Focus first on GPU-Util, Memory-Usage, Temp, and Pwr:Usage/Cap.

nvidia-smi

+-----------------------------------------------------------------------------+
| NVIDIA-SMI 550.xx       Driver Version: 550.xx       CUDA Version: 12.x    |
| GPU  Name        Temp   Pwr:Usage/Cap   Memory-Usage   GPU-Util  Compute M. |
| 0    H100        53C    215W / 350W     18240MiB/81920MiB   78%    Default |
+-----------------------------------------------------------------------------+
| Processes:                                                                |
| GPU   PID   Type   Process name                                GPU Memory |
| 0   18211     C    python train.py                                17664MiB|
+-----------------------------------------------------------------------------+

If GPU-Util shows 0% while a job appears to be running, check three common causes. The job may still be in a CPU-bound preprocessing stage and has not submitted work to the GPU yet. The process may have errored and stayed alive but idle. The job may also be running on a different GPU index, so list all devices with nvidia-smi --list-gpus and check each one.

Running nvidia-smi in Continuous Loop Mode

Use loop mode when you need live updates without writing scripts. --loop=1 refreshes once per second.

nvidia-smi --loop=1

Wed Mar 26 12:00:01 2026
... snapshot ...
Wed Mar 26 12:00:02 2026
... snapshot ...

Logging nvidia-smi Output to a File

Write sampled output to a file for post-run inspection. Redirect stdout so each sample is timestamped in your shell history and log stream.

nvidia-smi --loop=5 > gpu.log

# gpu.log now contains one snapshot every 5 seconds

Querying Specific Metrics with nvidia-smi --query-gpu

Use --query-gpu with --format=csv when you need parseable output for scripts. This is the preferred pattern for cron jobs and custom exporters.

nvidia-smi --query-gpu=timestamp,index,name,utilization.gpu,utilization.memory,memory.used,memory.total,temperature.gpu,power.draw --format=csv,noheader,nounits

2026/03/26 12:10:02.123, 0, NVIDIA H100 80GB HBM3, 82, 54, 18420, 81920, 55, 228.31

Per-Process GPU Monitoring

Per-process monitoring answers which application is consuming GPU time right now. Use nvidia-smi pmon to inspect utilization by PID instead of by device only.

Using nvidia-smi pmon for Process-Level Metrics

Run pmon in loop mode to monitor active compute processes. -s um displays utilization and memory throughput related activity by process.

nvidia-smi pmon -s um -d 1

# gpu   pid  type    sm   mem   enc   dec   command
    0 18211     C    76    41     0     0   python
    0 18304     C    12     8     0     0   python

gpu is the GPU index the process is running on. pid is the process ID. type is workload class, where C is compute, G is graphics, and M is mixed. sm is the percentage of time spent executing kernels on streaming multiprocessors. mem is the percentage of time the memory interface was active for that process. enc and dec are encoder and decoder utilization percentages. command is the truncated process name.

Correlating Process IDs to Application Names

Map PIDs to full command lines to identify notebook kernels, training scripts, and inference workers. This is required when multiple Python jobs are running under one user.

ps -p 18211 -o pid,user,etime,cmd

  PID USER     ELAPSED CMD
18211 mlops    01:22:11 python train.py --model llama --batch-size 8

Interactive GPU Monitoring with nvtop and gpustat

Use nvtop when you want interactive process control and gpustat when you want compact snapshots in scripts. Both tools complement nvidia-smi rather than replace it.

Installing and Running nvtop

Install nvtop from Ubuntu repositories, then start it in the terminal. It provides live bars and per-process views similar to htop.

sudo apt update && sudo apt install -y nvtop

nvtop

GPU0  78%  MEM 18240/81920 MiB  TEMP 54C  PWR 221W
PID 18211 python train.py   GPU 72%   MEM 17664MiB

Installing and Running gpustat

Install gpustat with pip, then use watch mode for one-second updates. This is useful in SSH sessions where minimal output matters.

python3 -m pip install --user gpustat

gpustat --watch 1

hostname  Thu Mar 26 12:25:44 2026
[0] NVIDIA H100 | 54C, 79 % | 18420 / 81920 MB | python/18211(17664M)

When to Use nvtop vs. gpustat vs. nvidia-smi

Use nvidia-smi for canonical driver-level data and scripted queries. Use gpustat for low-noise terminal snapshots, and use nvtop for interactive process monitoring during active debugging.

GPU Monitoring with Glances

Use Glances when you need one terminal dashboard for GPU, CPU, memory, disk, and network at once. Install with the GPU extra so NVIDIA metrics are available.

python3 -m pip install 'glances[gpu]'

glances

GPU NVIDIA H100: util 77% | mem 18240/81920MiB | temp 54C | power 220W
CPU: 21.4%  MEM: 62.1%  LOAD: 2.13 1.87 1.66

In the Glances GPU line, util maps to GPU core activity, and mem shows allocated versus total VRAM. temp and power indicate thermal and electrical load during the sample window. Use these values together to identify whether workload pressure is compute, memory, or thermal related. Glances is a better choice than nvidia-smi when you want CPU, memory, disk, and GPU in one non-scrolling view during interactive debugging on a single node.

<$>[note]
If glances shows no GPU section, verify that NVIDIA drivers are installed on the host and the Python environment running Glances can access NVML.
<$>

GPU Monitoring Inside Docker Containers and Kubernetes

Containerized GPU monitoring requires host runtime support first, then workload-level metric collection. Start with NVIDIA Container Toolkit for Docker and DCGM Exporter for Kubernetes clusters.

Exposing GPU Metrics in Docker with the NVIDIA Container Toolkit

Install the NVIDIA Container Toolkit on the host, then run containers with --gpus all. Inside the container, nvidia-smi should show host GPU telemetry.

Use this after setting up Docker by following How To Install and Use Docker on Ubuntu.

curl -fsSL https://nvidia.github.io/libnvidia-container/gpgkey | sudo gpg --dearmor -o /usr/share/keyrings/nvidia-container-toolkit-keyring.gpg

curl -s -L https://nvidia.github.io/libnvidia-container/stable/deb/nvidia-container-toolkit.list | \
  sed 's#deb https://#deb [signed-by=/usr/share/keyrings/nvidia-container-toolkit-keyring.gpg] https://#g' | \
  sudo tee /etc/apt/sources.list.d/nvidia-container-toolkit.list

sudo apt update && sudo apt install -y nvidia-container-toolkit

sudo nvidia-ctk runtime configure --runtime=docker

sudo systemctl restart docker

<$>[note]
The NVIDIA runtime is only active after the Docker daemon restarts. Already-running containers are not affected, but any new container launched after the restart will have GPU access. For full installation details, see the NVIDIA Container Toolkit guide.
<$>

docker run --rm --gpus all nvidia/cuda:12.4.1-base-ubuntu22.04 nvidia-smi

+-----------------------------------------------------------------------------+
| NVIDIA-SMI 550.xx       Driver Version: 550.xx       CUDA Version: 12.x    |
| GPU  Name        Persistence-M| Bus-Id        Disp.A | Volatile Uncorr. ECC |
+-----------------------------------------------------------------------------+

Monitoring GPU Utilization in Kubernetes with DCGM Exporter

Deploy DCGM Exporter as a DaemonSet on GPU nodes to expose Prometheus metrics. This creates scrape targets with per-GPU and per-pod metric labels.

apiVersion: apps/v1
kind: DaemonSet
metadata:
  name: dcgm-exporter
  namespace: gpu-monitoring
spec:
  selector:
    matchLabels:
      app: dcgm-exporter
  template:
    metadata:
      labels:
        app: dcgm-exporter
    spec:
      nodeSelector:
        nvidia.com/gpu.present: "true"
      containers:
        - name: dcgm-exporter
          image: nvcr.io/nvidia/k8s/dcgm-exporter:3.3.8-3.6.0-ubuntu22.04
          ports:
            - containerPort: 9400

# HELP DCGM_FI_DEV_GPU_UTIL GPU utilization (in %).
# TYPE DCGM_FI_DEV_GPU_UTIL gauge
DCGM_FI_DEV_GPU_UTIL{gpu="0",UUID="GPU-..."} 78

Viewing GPU Metrics in a DigitalOcean Managed Kubernetes Cluster

To collect GPU metrics in a DOKS cluster, configure Prometheus to scrape the DCGM Exporter DaemonSet, then visualize the data in Grafana or forward it to a hosted monitoring backend. Separate GPU dashboards by node pool and workload labels to avoid mixed tenancy confusion.

Before deployment, review An Introduction to Kubernetes if your team is new to cluster primitives.

scrape_configs:
  - job_name: dcgm-exporter
    static_configs:
      - targets: ['<node-ip>:9400']

In a DOKS cluster, use DaemonSet pod IPs or a Kubernetes Service DNS name instead of static node IP targets. For Grafana dashboard import details, see NVIDIA DCGM Exporter documentation.

Setting Up Persistent GPU Monitoring with Datadog

Use Datadog when you need long-term retention, tag-based slicing, and alert routing to on-call systems. Install the Agent on each GPU node and enable the NVIDIA integration.

Installing the Datadog Agent with NVIDIA GPU Support

Install Agent 7 on the GPU host, then enable the nvidia_gpu integration. Keep host drivers and NVML available to the Agent process.

DD_API_KEY="<YOUR_DATADOG_API_KEY>" DD_SITE="datadoghq.com" bash -c "$(curl -L https://s3.amazonaws.com/dd-agent/scripts/install_script_agent7.sh)"

<$>[note]
The NVML integration is not bundled with Agent 7 by default. Install it separately, then configure nvml.d/conf.yaml.
<$>

sudo datadog-agent integration install -t datadog-nvml==1.0.9

<$>[note]
Verify the latest available version of the NVML integration before installing.
<$>

Configuring the GPU Integration and Tag Strategy

Define tags at the host and integration level so you can group by cluster, environment, and workload type. This keeps alert routing and dashboard filters usable at scale.

init_config:

instances:
  - min_collection_interval: 15
    tags:
      - env:prod
      - role:training
      - gpu_vendor:nvidia

Save this as /etc/datadog-agent/conf.d/nvml.d/conf.yaml, then restart:

sudo systemctl restart datadog-agent

Building a Real-Time GPU Dashboard and Setting Alerts

Create timeseries panels for nvidia.gpu.utilization, nvidia.gpu.memory.used, and nvidia.gpu.temperature, then alert on sustained saturation. A practical first alert is GPU utilization above 95% for 10 minutes on production training nodes.

Use How To Monitor Your Infrastructure with Datadog for dashboard and monitor fundamentals.

Example monitor query:
avg(last_10m):avg:nvidia.gpu.utilization{env:prod,role:training} by {host,gpu_index} > 95

Setting Up GPU Monitoring with Zabbix

To monitor GPU hosts with Zabbix, install the Zabbix agent on each GPU host, import the NVIDIA GPU template, and configure trigger thresholds for utilization and temperature. Zabbix is the right choice when you need self-hosted monitoring with custom alerting and existing enterprise integrations.

Enabling the NVIDIA GPU Template in Zabbix

Import or attach an NVIDIA GPU template in Zabbix, then bind it to hosts that have NVIDIA drivers installed. Template items should poll utilization, memory, temperature, and power.

Path: Data collection -> Templates -> Import
Template: Nvidia by Zabbix agent 2
For some versions, the active mode variant is: Nvidia by Zabbix agent 2 active
Official template source: https://git.zabbix.com/projects/ZBX/repos/zabbix/browse/templates/app/nvidia_agent2

Configuring Triggers for Utilization Thresholds

Create triggers for sustained high utilization, high temperature, and unexpected drops to zero utilization during scheduled training windows. Use trigger expressions with time windows to avoid noise from short spikes.

Example trigger logic using Zabbix agent 2 template item keys:
avg(/GPU Host/nvidia.smi[{#GPUINDEX},utilization.gpu],10m)>95
and
last(/GPU Host/nvidia.smi[{#GPUINDEX},temperature.gpu])>85

{#GPUINDEX} is a low-level discovery macro populated automatically by the template. You do not need to set it manually.

Enabling Unified GPU Usage Monitoring on Windows

Unified GPU Usage Monitoring aggregates activity from multiple GPU engines into a single usage view that operators can read quickly. Enable it through NVIDIA Control Panel first, then verify registry policy where required by your driver profile.

What Unified GPU Usage Monitoring Is

Unified monitoring combines graphics, compute, copy, and video engine activity into one normalized utilization metric. This improves cross-process visibility when mixed workloads run on the same adapter.

How to Enable It via NVIDIA Control Panel and Registry

In NVIDIA Control Panel, enable the GPU activity monitoring feature and apply settings system-wide. If your environment uses managed policy, set the registry value used by your NVIDIA driver branch to turn on unified usage reporting.

Windows Registry example for GPU performance counter visibility:
HKEY_LOCAL_MACHINE\SOFTWARE\NVIDIA Corporation\Global\NVTweak
Value name: RmProfilingAdminOnly (DWORD)
Set to 0 to allow non-admin access to GPU performance counters, set to 1 for admin-only.
Reference: https://developer.nvidia.com/ERR_NVGPUCTRPERM

reg query "HKLM\SOFTWARE\NVIDIA Corporation\Global" /s

<$>[warning]
Registry value names for unified usage reporting vary by driver branch and policy tooling. Validate the exact key and value against your NVIDIA enterprise driver documentation before changing production systems.
<$>

Reading Unified GPU Data via Task Manager and WMI

After enabling unified monitoring, Task Manager can display GPU engine and aggregate usage per process. WMI queries can then be used for scripted collection in Windows-based monitoring workflows.

powershell -Command "Get-Counter '\GPU Engine(*)\Utilization Percentage' | Select-Object -ExpandProperty CounterSamples | Select-Object InstanceName,CookedValue"

InstanceName                                   CookedValue
pid_1204_luid_0x00000000_0x0000_engtype_3D     27.31
pid_1820_luid_0x00000000_0x0000_engtype_Compute_0  74.02

Comparing GPU Monitoring Tools

Use this table to pick a tool based on data depth, operational overhead, and alerting needs. Start with CLI tools for diagnostics, then add Datadog, Zabbix, or DCGM pipelines for persistent monitoring.

Feature and Trade-off Comparison Table

Tool	Platform	Refresh Rate	Per-Process View	Alerting	Cost
nvidia-smi	Linux, Windows	1s+ (`--loop`)	Yes (process list, `pmon`)	No native alerts	Free
nvtop	Linux	Near real time interactive	Yes	No native alerts	Free
gpustat	Linux	1s+ (`--watch`)	Yes (summary)	No native alerts	Free
Glances	Linux, macOS, Windows	1s+	Partial	No native alerts	Free
atop	Linux	Configurable interval	Indirect for GPU	No native alerts	Free
Datadog Agent	Linux, Windows	15s typical agent interval	Yes (tag and host context)	Yes	Paid
Zabbix	Linux, Windows	Configurable polling	Yes (template dependent)	Yes	Free (self-hosted)
DCGM Exporter	Linux, Kubernetes	Scrape interval based	Yes (label dependent)	Via Prometheus/Grafana Alertmanager	Free

Choosing the Right Tool for Your Use Case

For single-node debugging, start with nvidia-smi and nvtop. For fleet-level visibility across GPU Droplets and Kubernetes nodes, use DCGM Exporter plus your monitoring backend or deploy Datadog or Zabbix for retention and alerting.
If you need a historical record of GPU activity alongside CPU, memory, and disk in a single log, atop captures all of these at configurable intervals and is worth adding to long-running training hosts alongside nvidia-smi.

Conclusion

Real-time GPU utilization monitoring is essential for optimizing deep learning performance, troubleshooting bottlenecks, and achieving efficient resource usage—whether running on single nodes, inside containers, or scaling across clustered environments. The right monitoring tool depends on your specific use case: quick one-off checks, interactive debugging, continuous fleet-wide visibility, or long-term metric retention and alerting.

Start with simple tools like nvidia-smi for instant visibility, and progress to dashboarding, custom alerting, and enterprise-grade solutions as your needs grow. With the strategies and tools outlined in this guide, you can proactively monitor, troubleshoot, and maximize the performance of your GPU workloads—ensuring smoother operation for development, training, and deployment pipelines.

How I Used Nemotron 3 to Help Me Find the Perfect Dishrack

Andrew Dugan — Thu, 09 Apr 2026 18:00:00 +0000

After recently moving into a new apartment, I realized how much time I was spending searching online for household items ranging from storage solutions, to pots and pans, to the furniture thing that sits at the end of the bed. It occurred to me that this seems like the perfect task for an LLM. So I built an app that does just that.

The Nemofinder sorts through dozens of product descriptions to find one that matches your exact needs. This tutorial describes how the application works.

Key Takeaways

Nemotron 3 Nano's efficient Mixture-of-Experts architecture enables cost-effective product filtering at scale, comparing product descriptions against specific requirements while maintaining high accuracy.
The Nemofinder integrates third-party search APIs to gather product listings and leverages Nemotron 3 Nano to intelligently match products based on detailed user requirements, reviews, and pricing.
The application is fully customizable and open source, allowing you to adapt it for any product search use case and integrate it with different search APIs based on your needs.

Why Nemotron 3 Nano?

Nemotron 3 Nano is specifically optimized for cost efficiency in targeted agentic tasks without sacrificing accuracy. This makes it an ideal choice for filtering through dozens of product descriptions and checking whether each one matches specific product requirements. Unlike larger models that may be overkill for focused tasks, Nano delivers strong performance while remaining significantly more efficient. It is also open source, giving you complete control over your personal product queries and output data.

Under the hood, Nemotron 3 Nano uses a hybrid Mixture-of-Experts (MoE) architecture combined with Mamba-2 state-space models, which dramatically reduces computational overhead compared to traditional transformer architectures. Even though the model has 30 billion parameters, only 3.5 billion are active per token during inference. This architectural efficiency translates to faster response times and lower computational costs, making it practical to deploy on smaller GPU instances. Additionally, you can optionally disable Nemotron's reasoning capabilities through a simple configuration flag if you need even faster inference for straightforward product matching tasks, though this may slightly reduce accuracy. Refer to the deployment guide to deploy an instance on a DigitalOcean Droplet.

How the Nemofinder Works

First, the application takes the keyword you would like to search along with a detailed text description of your specific requirements for that item.

It then uses a search API (application programming interface) to look for items using the keyword. The search API can be store-specific, a generic shopping API, or a custom combination that calls multiple APIs. It needs to be able to take a keyword and return a list of products with their descriptions, and ideally reviews, as a response.

The application then goes through each of the product descriptions, prices, reviews, comments, etc., and has Nemotron 3 Nano compare each description to your product requirements. After sorting through and finding matches, it returns the matches to the user. In this case, it found the perfect dish rack to match the requirements in my description.

Improving and Implementing the Nemofinder

The Nemofinder is open source and available on GitHub. You need to add a SerpAPI key or change the API to one that you have access to. You need to set up a DigitalOcean GPU droplet with Nemotron 3. Next, you need to update the Nemotron 3 calls to use your deployment's IP address. Feel free to clone, change, and use the application as you'd like.

FAQ

Can this application buy the product?

No, purchasing functionality could be added, but I wouldn't trust it. The problem being solved in this use case is the time spent looking for the ideal product. Automating purchases without human verification introduces unnecessary risk.

Can it search on all platforms, like Amazon?

Only if you have an API for that particular platform. With the right API, you can search through anything. Amazon does offer a Product Advertising API, though access can be limited. For most e-commerce platforms, you'll need to check their developer documentation.

Can I use a different LLM instead of Nemotron 3 Nano?

Yes, you can adapt the application to use other models. However, Nemotron 3 Nano is recommended for its efficiency and cost-effectiveness on product filtering tasks. Larger models like Claude or GPT may work but could result in higher token costs.

How do I handle price variations across different products?

As long as the API allows, the application passes the price data from the search API alongside the product description to Nemotron 3 Nano. You can modify the prompts to set price thresholds or have the model factor pricing into the matching criteria based on your budget requirements.

Is my product search history private?

It depends on how you deploy it. Running the application locally keeps everything on your machine. If you deploy it on a remote server, be mindful of which APIs you're using and review their privacy policies. Consider using a dedicated API account and limiting what data is logged.

Conclusion

The Nemofinder demonstrates how Nemotron 3 Nano can efficiently handle targeted product discovery tasks without the overhead of larger language models. By combining intelligent search APIs with Nemotron's reasoning capabilities, you can quickly find products that match your exact specifications across multiple product listings and review data. Whether you're searching for household items, specialized equipment, or niche products, the application adapts to your needs through customizable prompts and API integrations.

The beauty of the Nemofinder is its flexibility. You can extend it to search across multiple e-commerce platforms, add additional filtering criteria, or integrate it into a larger workflow. As shown in the related Daily Digest tutorial, these kinds of specialized tools can be combined to create comprehensive AI-driven solutions. If you want to explore further or build your own product search application, the source code is available on GitHub, and the setup process is straightforward with the right API keys and a Nemotron 3 Nano deployment.

March 2026 DigitalOcean Tutorials: GPT-5.4 and Nemotron 3

DigitalOcean — Mon, 06 Apr 2026 16:00:00 +0000

AI development continues to change with the consistent release of new models, standards, and system architectures. It can often be a lot to keep track of and learn. But DigitalOcean has you covered with our community tutorials and resources.

These 10 tutorials from last month cover both practical, hands-on topics (such as building a game with GPT-5.4) and explanatory concepts (like migrating to multi-agent systems). Take a look and try them out—or bookmark them for some weekend coding!

Getting Started with Qwen3.5 Vision-Language Models

This tutorial walks through how to run and experiment with Qwen 3.5, an open-source multimodal model family that handles text, images, and even video. It breaks down the model’s architecture and demonstrates how to deploy it on GPU infrastructure so you can build apps like coding assistants or document analyzers on your own stack. You’ll see how high-performing multimodal AI is becoming accessible without relying on proprietary APIs.

A2A vs MCP: How These AI Agent Protocols Actually Differ

Read about the difference between two emerging standards for agent-based systems: agent-to-agent communication (A2A) and model context protocol (MCP). You’ll learn when to use each—A2A for coordinating multiple agents and MCP for structured tool integration—and why most production systems combine both. It’s a practical breakdown of the protocols shaping how agentic AI systems are actually built.

Nemotron 3 Helped Me Find the Perfect Dish Rack?

Get insight into how NVIDIA’s Nemotron 3 model pairs with NemoFinder to improve retrieval and reasoning workflows. This tutorial demonstrates how combining LLMs with optimized search and ranking pipelines can yield more accurate results, especially in enterprise or knowledge-intensive applications. You’ll also learn more about how retrieval-augmented generation (RAG) systems are evolving with tighter model–tool integration.

Train YOLO26 for Retail Object Detection on DigitalOcean GPUs

This hands-on guide shows how to train a YOLOv26 model for retail use cases such as shelf monitoring and product detection on GPU infrastructure. It walks through dataset prep, training, and deployment so you can build real-world computer vision pipelines. You’ll gain a better understanding of how to move from raw image data to a production-ready detection model.

Building Long-Term Memory in AI Agents with LangGraph and Mem0

If you’re curious about how to add persistent memory to agent workflows using LangGraph and Mem0, check out this tutorial. It shows how agents can retain context across sessions, enabling more personalized and stateful interactions over time. Its key takeaway is how long-term memory transforms agents from stateless responders into systems that can learn and adapt.

Crafting a Game from Scratch with GPT-5.4

This article breaks down GPT-5.4’s capabilities, improvements, and practical use cases. It highlights advancements in reasoning, efficiency, and multimodal performance, and shows how developers can integrate the model into real applications. You’ll see how this frontier model integrates into modern AI stacks and the steps involved in creating a 3D badminton game from the ground up.

What are Text Diffusion Models? An Overview

This guide introduces diffusion models for text generation and explains how they differ from traditional autoregressive LLMs. It walks through how diffusion-based approaches iteratively refine outputs and where they may outperform standard models. You’ll get a conceptual and practical understanding of an emerging alternative to transformers.

LLM Tool Calling with Gradient™ AI Platform and Databases

Discover how to connect LLMs to external tools—like databases—using structured tool calling. It walks through building workflows in which models query, retrieve, and act on real data rather than relying solely on prompts. You’ll get to see that tool integration makes LLMs more reliable and production-ready.

How to Generate Videos with LTX-2.3 on DigitalOcean GPU Droplets

This tutorial explores how to generate videos using LTX 2.3, covering setup, prompts, and rendering workflows. It demonstrates how generative AI is expanding beyond text and images into video creation. After this article, you’ll know how to experiment with video generation pipelines and integrate them into creative or product workflows.

From Single to Multi-Agent Systems: Key Infrastructure Needs

Get an overview of what changes when you move from a single AI agent to a multi-agent system. This tutorial goes through the full infrastructure stack—covering orchestration patterns, communication protocols, memory, and observability—so you can design systems where multiple agents collaborate reliably. Ultimately, multi-agent setups unlock scalability and specialization but require significantly more coordination, state management, and fault tolerance to work in production.

Build an End-to-End RAG Pipeline for LLM Applications

DigitalOcean — Wed, 01 Apr 2026 01:06:34 +0000

This article was originally written by Shaoni Mukherjee (Technical Writer)

Large language models have transformed the way we build intelligent applications. Generative AI Models can summarize documents, generate code, and answer complex questions. However, they still face a major limitation: they cannot access private or continuously changing knowledge unless that information is incorporated into their training data.

Retrieval-Augmented Generation (RAG) addresses this limitation by combining information retrieval systems with generative AI models. Instead of relying entirely on the knowledge embedded in model weights, a RAG system retrieves relevant information from external sources and provides it to the language model during inference. The model then generates a response grounded in this retrieved context.

An end-to-end RAG pipeline refers to the full system that manages this process from beginning to end. It includes ingesting documents, transforming them into embeddings, storing them in a vector database, retrieving relevant information for a user query, and generating an answer using a large language model.

This architecture is increasingly used in modern AI systems such as enterprise knowledge assistants, internal documentation search engines, developer copilots, and AI customer support tools. Organizations adopt RAG because it allows models to remain lightweight while still accessing large knowledge bases that change frequently.

In this tutorial, we will walk through how to design and build a complete RAG pipeline. Along the way, we will explore architectural considerations, optimization strategies, and production challenges developers encounter when deploying retrieval-based AI systems.

Key Takeaways

RAG combines retrieval and generation for more accurate AI systems: Retrieval-Augmented Generation (RAG) bridges the gap between static language models and dynamic, real-world data. Instead of relying only on pre-trained knowledge, it fetches relevant information at runtime and uses it to generate answers. This makes responses more accurate, up-to-date, and context-aware. It is especially useful for applications like chatbots, internal knowledge assistants, and search systems. Overall, RAG helps reduce hallucinations and improves trust in AI-generated outputs.
Vector embeddings are the foundation of semantic search in RAG: Embeddings convert text into numerical vectors that capture meaning rather than exact wording. This allows the system to understand similarity between queries and documents even if they use different phrasing. As a result, retrieval becomes more intelligent and context-driven instead of keyword-based. High-quality embedding models like text-embedding-3-large or bge-large-en can significantly improve retrieval performance. Choosing the right embedding model directly impacts the overall quality of your RAG system.
Each component of the pipeline plays a critical role: A RAG system is made up of multiple steps, including ingestion, chunking, embedding, storage, retrieval, and generation. If any one component is poorly optimized, it can affect the entire pipeline’s performance. For example, bad chunking can lead to irrelevant retrieval, even if your embedding model is strong. Similarly, weak retrieval will result in poor answers, no matter how powerful the language model is. This is why building an end-to-end RAG system requires careful design and tuning at every stage.
Evaluation is essential for building reliable RAG applications: It is not enough to just build a RAG pipeline, but you must also evaluate how well it performs. This includes checking whether the system retrieves the correct documents and whether the generated answers are accurate and grounded. Metrics like precision and recall help measure retrieval quality, while human evaluation helps assess answer correctness. Creating benchmark datasets with known questions and answers makes it easier to track improvements over time. Continuous evaluation ensures your system remains reliable in production.

Understanding the RAG System Architecture

Before implementing the pipeline, it is important to understand how the different components interact. A typical RAG system architecture can be divided into two major workflows: the indexing pipeline and the retrieval pipeline.

The indexing pipeline prepares the knowledge base so that it can be searched efficiently. During this stage, documents are ingested, cleaned, split into chunks, converted into embeddings, and stored in a vector database. This process is usually executed offline or periodically when new data becomes available.

The retrieval pipeline operates during inference. When a user asks a question, the system converts that query into an embedding, searches the vector database for semantically similar chunks, and provides those retrieved passages to the language model. The model then generates a response using both the query and the contextual information.

A simplified representation of the pipeline looks like this:

Document Sources
       (PDFs, Docs, APIs, Knowledge Base)
                        |
                        v
               Document Processing
                        |
                        v
                  Text Chunking
                        |
                        v
               Embedding Generation
                        |
                        v
               Vector Database Index
                        |
                        v
User Query → Query Embedding → Similarity Search
                        |
                        v
             Retrieved Context Chunks
                        |
                        v
                  LLM Generation
                        |
                        v
                  Final Response

This architecture enables the system to retrieve information dynamically rather than relying solely on model training.

Data Ingestion in a RAG Pipeline

The first stage of the pipeline involves gathering the data that the AI system will use as its knowledge source. In many real-world applications, this information is distributed across multiple systems. Organizations may store documentation in internal knowledge bases, PDFs, wikis, product manuals, or database records.

The ingestion stage extracts textual information from these sources and prepares it for processing. Depending on the data format, ingestion may involve parsing HTML pages, converting PDFs to text, or querying APIs to retrieve structured records.

At this stage, developers often implement preprocessing steps such as removing redundant formatting, normalizing whitespace, and filtering irrelevant sections. These steps are important because retrieval performance strongly depends on the quality of the text data stored in the system.

For enterprise knowledge retrieval systems, ingestion pipelines are usually automated and scheduled. For example, an internal documentation chatbot might update its knowledge base daily by ingesting the latest documentation changes from a repository.

Text Chunking: Preparing Documents for Retrieval

After ingestion, documents must be divided into smaller pieces before they can be embedded. This step, known as text chunking, plays a critical role in the overall performance of the RAG pipeline.

Large documents cannot be embedded effectively because embedding models have token limits and because large chunks reduce retrieval precision. Instead, documents are broken into manageable segments that capture a coherent piece of information.

Chunk size is typically chosen between 200 and 500 tokens. Smaller chunks provide more precise retrieval results, while larger chunks preserve more contextual information. Many production pipelines use overlapping chunks to prevent important sentences from being split across boundaries.

The following diagram illustrates how a long document is transformed into multiple overlapping chunks:

Original Document
-------------------------------------------------------
| Paragraph 1 | Paragraph 2 | Paragraph 3 | Paragraph 4 |
-------------------------------------------------------

After Chunking
-------------------------------------------------------
| Chunk 1 | Chunk 2 | Chunk 3 | Chunk 4 | Chunk 5 |
-------------------------------------------------------

Chunk Example
Chunk 1: Paragraph 1 + part of Paragraph 2
Chunk 2: Paragraph 2 + part of Paragraph 3

Choosing an effective chunking strategy significantly improves retrieval accuracy because each chunk represents a focused semantic concept.

Embedding Generation

Once documents are divided into chunks, each chunk must be converted into a numerical representation called an embedding. Embeddings transform text into high-dimensional vectors that capture semantic meaning.

For example, two sentences that express similar ideas will produce vectors that are close to each other in vector space. This property allows vector databases to retrieve semantically related text even when the wording differs.

Embedding models are trained using large datasets and transformer architectures. When a chunk is processed, the model generates a vector with hundreds or thousands of dimensions. These vectors serve as the foundation for similarity search.

Embedding generation occurs during both indexing and retrieval. During indexing, embeddings are generated for each document chunk. During retrieval, the user’s query is also converted into an embedding so that it can be compared against stored vectors.

This mechanism allows the RAG system to perform semantic search, which is far more powerful than traditional keyword matching.

Vector Embedding

Vector embeddings are dense numerical representations of data, which can be text, images, or audio. Vector embeddings are used to capture the semantic meaning of the data in a high-dimensional vector space. In an end-to-end RAG pipeline, embeddings are used to convert both documents and user queries into vectors so that similarity between them can be measured using metrics like cosine similarity. This allows the system to retrieve context based on meaning rather than exact keyword matches, making responses more accurate and relevant.

For example, even if a query doesn’t contain the same words as a document, embeddings can still identify it as relevant if the underlying intent is similar. Popular embedding models used in RAG systems include text-embedding-3-large, all-MiniLM-L6-v2, bge-large-en, and e5-large-v2, each offering different trade-offs in performance, cost, and deployment flexibility.

Storing Vectors in a Database

After embeddings are created, they must be stored in a specialized database capable of performing fast similarity searches. These systems are known as vector databases and form the core of the RAG retrieval infrastructure.

Unlike traditional databases that index numeric or textual fields, vector databases are optimized to search across high-dimensional vectors. They use approximate nearest neighbor algorithms to identify vectors that are closest to a query embedding.

The structure of a stored vector typically includes the embedding itself, the original text chunk, and metadata describing the source of the information. Metadata can include document identifiers, timestamps, or categories that allow filtering during retrieval.

A simplified representation of vector storage looks like this:

Vector Database

ID     Vector Embedding        Text Chunk
---------------------------------------------------------
1   [0.12, -0.44, 0.92...]   "RAG combines retrieval..."
2   [0.55, 0.33, -0.14...]   "Vector databases enable..."
3   [-0.77, 0.08, 0.62...]   "Embeddings represent..."

Popular vector database technologies include managed services and open-source platforms designed specifically for AI workloads. The choice often depends on scale, infrastructure preferences, and latency requirements.

Retrieval in a RAG Pipeline

When a user submits a question, the system begins the retrieval stage. The query is first converted into an embedding using the same embedding model used during indexing. Maintaining the same embedding model is important because similarity comparisons rely on consistent vector representations.

The query embedding is then sent to the vector database. The database performs a similarity search to find document chunks whose embeddings are closest to the query vector. These chunks represent the pieces of information most relevant to the user’s question.

The retrieved chunks are then combined and passed to the language model as contextual input. The model uses this context to generate a response grounded in actual documents rather than relying solely on its training data.

This process ensures that answers are based on real knowledge sources and can be updated whenever the underlying documents change.

Generation with a Large Language Model

The final stage of the pipeline involves generating a response using a language model. At this point, the system already has two pieces of information: the user’s question and the retrieved context.

These elements are combined into a prompt that instructs the model to answer the question using the provided information. Because the context is derived from authoritative documents, the model’s output becomes significantly more reliable and factual.

This stage also allows developers to control how responses are generated. Prompts may instruct the model to summarize information, provide citations, or answer in a specific format. Some systems also include guardrails that prevent hallucinations or restrict responses to retrieved information.

For example, if a user asks a question, the system first pulls the most relevant text from your knowledge base, then the LLM rewrites that content into a helpful answer, making it more conversational, structured, and easy to understand. This step is what makes RAG powerful, because it combines accurate, up-to-date information with fluent natural language generation, reducing hallucinations and improving answer quality.

Code Demo: Building a Simple End-to-End RAG Pipeline

The following example demonstrates how a basic RAG pipeline for LLM applications can be implemented in Python. The example uses document loading, chunking, embeddings, and a vector database to create a minimal working pipeline.

Install dependencies

pip install langchain chromadb sentence-transformers openai

Load documents

from langchain.document_loaders import TextLoader

loader = TextLoader("knowledge_base.txt")
documents = loader.load()

Split documents into chunks

from langchain.text_splitter import RecursiveCharacterTextSplitter

splitter = RecursiveCharacterTextSplitter(
   chunk_size=500,
   chunk_overlap=100
)

chunks = splitter.split_documents(documents)

Generate embeddings

from langchain.embeddings import HuggingFaceEmbeddings

embeddings = HuggingFaceEmbeddings(
   model_name="sentence-transformers/all-MiniLM-L6-v2"
)

Store vectors

from langchain.vectorstores import Chroma

vector_db = Chroma.from_documents(
   documents=chunks,
   embedding=embeddings
)

Retrieval and generation

from langchain.chains import RetrievalQA
from langchain.chat_models import ChatOpenAI

llm = ChatOpenAI()

qa_chain = RetrievalQA.from_chain_type(
   llm=llm,
   retriever=vector_db.as_retriever()
)

response = qa_chain.run(
   "What is retrieval augmented generation?"
)

print(response)

This simple implementation demonstrates how document retrieval and language models can be combined into a working RAG system.

Evaluating RAG System Performance

Evaluating a RAG system is important because you need to be sure that it is not only retrieving the right information but also generating correct and useful answers from it. In simple terms, a good RAG pipeline should find the right content and then explain it correctly.

First, let’s look at retrieval evaluation. This checks whether the system is pulling the right documents from your database. Imagine you have a knowledge base about cloud services, and a user asks, “How can I run AI models on GPUs?”. If your system retrieves documents about GPU Droplets or AI infrastructure, that’s a good sign. But if it returns unrelated content like pricing pages or networking docs, retrieval quality is poor. Metrics like recall (did we find all relevant documents?) and precision (were the retrieved documents actually relevant?) help measure this. For example, if 5 documents are relevant but your system only retrieves 2, recall is low.

Next is generation evaluation, which focuses on the answer produced by the language model. Even if retrieval is correct, the model (like GPT-4 or Llama 3) might still generate incomplete or incorrect responses. For instance, if the retrieved document clearly says “GPU droplets support CUDA workloads”, but the model responds with “GPU support is limited”, that’s a problem. This is why human evaluation is often needed to check if the answer is factually correct, complete, and grounded in the provided context. Automated metrics struggle to detect things like s or subtle inaccuracies.

To make evaluation consistent, teams usually create an evaluation dataset. This is a collection of sample questions along with their correct answers and sometimes the expected source documents. For example:

Question: “What are GPU droplets used for?”
Expected answer: “They are used for AI/ML workloads, training models, and high-performance computing.”

You can then run your RAG system on this dataset and compare its answers against the expected ones. Over time, this helps you track improvements, catch errors, and tune your system (for example, by improving chunking, choosing a better embedding model, or adjusting prompts).

In practice, strong RAG evaluation combines:

Retrieval checks: Did we fetch the right information?
Answer checks: Did we explain it correctly?
Continuous testing: Are we improving over time?

This ensures your RAG pipeline is reliable, accurate, and ready for real-world use.

Scaling and Production Considerations

Prototype RAG pipelines often work well with small datasets, but production deployments introduce additional challenges. Large organizations may store millions of document chunks, requiring scalable infrastructure for indexing and retrieval.

Latency also becomes an important concern. Vector searches, embedding generation, and LLM inference all contribute to response time. Developers must carefully optimize these components to ensure interactive performance.

Production systems frequently incorporate caching layers, query batching, and efficient indexing strategies. Monitoring tools are also used to track retrieval accuracy, system latency, and cost per query.

Cost and Latency Optimization

Operating a RAG pipeline at scale can become expensive if not carefully optimized. Each query may require embedding generation, vector search, and language model inference.

Several strategies help reduce these costs. Caching responses for frequently asked questions prevents repeated model inference. Limiting the number of retrieved chunks also reduces token usage and speeds up generation.

Another important technique is re-ranking. Instead of sending many retrieved documents to the language model, a re-ranking model selects the most relevant passages before generation. This improves response quality while reducing computational overhead.

RAG vs Fine-Tuning

A common question among developers is whether to use retrieval-augmented generation or fine-tuning.

Fine-tuning changes a model’s internal weights by training it on additional datasets. This approach works well for teaching models specific styles or behaviors. However, it is less effective for continuously changing knowledge because retraining the model is expensive and time-consuming.

RAG systems take a different approach by keeping the model unchanged while retrieving knowledge dynamically. This makes them ideal for applications where information changes frequently, such as product documentation or customer support knowledge bases.

For most knowledge-intensive applications, RAG provides a more flexible and maintainable solution.

Conclusion

Building an end-to-end RAG pipeline is about combining the strengths of retrieval systems and large language models to create applications that are both accurate and context-aware. Instead of relying only on pre-trained knowledge, a RAG system can fetch relevant information in real time and use models like GPT-4 or Llama 3 to generate clear, human-like responses grounded in that data. In this article, we understood each of the steps used to create the RAG pipeline from data ingestion and chunking to vector embeddings, retrieval, and response generation. Each component plays a critical role, and even small improvements (like better chunking strategies or choosing the right embedding model) can significantly impact overall performance. As organizations continue to build AI-powered applications, RAG stands out as a practical and scalable approach for use cases like chatbots, knowledge assistants, and document search. By continuously evaluating and refining your pipeline, you can create systems that are not only intelligent but also reliable and production-ready.

Resources

Tutorial: Deploy NVIDIA's NemoClaw in One Click

DigitalOcean — Mon, 23 Mar 2026 18:28:14 +0000

This article was originally written by Amit Jotwani (Staff Developer Advocate at DigitalOcean)

Key Takeaways

NemoClaw is an open-source stack from NVIDIA designed to help developers run OpenClaw securely.
DigitalOcean offers NemoClaw 1-Click Droplets that enable you to set up this stack on a CPU-optimized virtual machine and run NemoClaw.
This tutorial illustrates how to SSH into your Droplet, configure inference settings and policies, connect to NemoClaw, and effectively reconnect after the initial setup.

At GTC 2026, NVIDIA announced NemoClaw, an open-source stack that makes it easy to run OpenClaw autonomous agents securely. OpenClaw is an open-source agent platform that Jensen Huang called “the operating system for personal AI.” We covered how to run OpenClaw on a Droplet in an earlier tutorial. NemoClaw takes a different approach — it wraps OpenClaw with sandboxing, security policies, and inference routing through NVIDIA’s cloud.

NemoClaw is still in alpha, so expect rough edges. Interfaces may change, features might be incomplete, and things could break. But if you’re curious to try it out or just want to see what NVIDIA’s vision for agents looks like, this tutorial will get you up and running on a DigitalOcean Droplet in under 10 minutes.

Prerequisites

Before you begin, you’ll need:

A DigitalOcean account (sign up here if you don’t have one)
An NVIDIA account to generate an API key at build.nvidia.com

Step 1 - Create a Droplet from the Marketplace

Head to the NemoClaw 1-Click Droplet on the DigitalOcean Marketplace. Click Create NemoClaw Droplet.

When configuring the Droplet, select the CPU-Optimized plan with Premium Intel. You’ll want the option with 32 GB of RAM and 16 CPUs. NemoClaw runs Docker containers, a Kubernetes cluster (k3s), and the OpenShell gateway, so it needs the headroom.

Pick a data center region near you, add your SSH key, and hit Create Droplet.

Heads up: This Droplet costs $336/mo, so make sure to destroy it when you’re done experimenting. It adds up fast if you forget about it.

Step 2 - SSH into the Droplet

Once your Droplet is ready, SSH in:

ssh root@your_server_ip

You’ll see the usual Ubuntu login banner, and then the NemoClaw onboarding wizard will kick off automatically. It runs through a series of preflight checks, making sure Docker is running, installing the OpenShell CLI, and spinning up the gateway. You’ll see checkmarks fly by as each step completes.

Step 3 - Walk Through the Onboard Wizard

The onboarding wizard will ask you a few things. Here’s what to do at each prompt:

Sandbox Name

The first prompt asks for a sandbox name. Just press Enter to accept the default (my-assistant). The wizard will then create the sandbox, build the container image, and push it to the gateway. This takes a couple of minutes, and you’ll see it run through about 20 steps as it builds and uploads everything.

NVIDIA API Key

Once the sandbox is ready, the wizard asks for your NVIDIA API key. In this setup, inference is routed through NVIDIA’s cloud using the nvidia/nemotron-3-super-120b-a12b model, so it needs a key to authenticate.

To get your key, head to build.nvidia.com/settings/api-keys, sign in, and click Generate API Key. Give it a name, pick an expiration, and hit Generate Key.

Copy the key (it starts with nvapi-), paste it into the terminal prompt, and press Enter.

The wizard saves the key to ~/.nemoclaw/credentials.json and sets up the inference provider. You’ll see it confirm the model and create an inference route.

Policy Presets

After the inference setup, NemoClaw sets up OpenClaw inside the sandbox and then asks about policy presets. You’ll see a list of available presets including Discord, Docker Hub, Hugging Face, Jira, npm, PyPI, Slack, and more. These control what external services the agent is allowed to reach.

At the bottom, the wizard asks:

Apply suggested presets (pypi, npm)? [Y/n/list]:

Type n and press Enter. These presets grant the sandbox network access to package registries, which you don’t need for a basic setup. You can always add them later if your agent needs to install packages.

Once onboarding finishes, you’ll see a clean summary with your sandbox details and the commands you’ll need going forward:

Sandbox    my-assistant (Landlock + seccomp + netns)
Model      nvidia/nemotron-3-super-120b-a12b (NVIDIA Cloud API)
NIM        not running

Run:       nemoclaw my-assistant connect
Status:    nemoclaw my-assistant status
Logs:      nemoclaw my-assistant logs --follow

Step 4 - Connect to NemoClaw

Now for the fun part. Connect to your sandbox.

nemoclaw my-assistant connect

This drops you into a shell inside the sandboxed environment. From here, launch the OpenClaw TUI (terminal user interface):

openclaw tui

That’s it. You should see the OpenClaw chat interface come up. The agent will greet you and introduce itself, ready to chat.

Type a message and hit Enter. You’re now talking to an AI agent running inside a secure, sandboxed environment on your own Droplet.

Reconnecting After a New SSH Session

If you close your terminal and SSH back into the Droplet later, you’ll find that nemoclaw and related commands aren’t available. That’s because the onboarding script installed everything through nvm in a separate shell, and that doesn’t carry over to new sessions.

Run this once to fix it permanently. It adds nvm to your .bashrc so it loads automatically on every login:

echo 'export NVM_DIR="$HOME/.nvm"' >> ~/.bashrc && echo '[ -s "$NVM_DIR/nvm.sh" ] && \. "$NVM_DIR/nvm.sh"' >> ~/.bashrc && echo '[ -s "$NVM_DIR/bash_completion" ] && \. "$NVM_DIR/bash_completion"' >> ~/.bashrc && source ~/.bashrc

Then reconnect to your sandbox and launch the TUI the same way as before:

nemoclaw my-assistant connect

openclaw tui

Everything picks up right where you left off. Your sandbox and agent are still running.

What's Next

By default, the sandbox has limited network access, so the agent can’t reach external services out of the box. To unlock more capabilities - like connecting to Slack, GitHub, or pulling packages from PyPI - you’ll want to configure policy presets. Check the NemoClaw documentation for the full list of available integrations and how to set them up.

NemoClaw is still very early, so expect things to be rough around the edges. But if you want to get a feel for where always-on agents are headed, this is a good way to start poking around.

Resources

GPT 5.3 Codex is the Next Level for Agentic Coding

DigitalOcean — Thu, 19 Mar 2026 20:00:00 +0000

Agentic Coding models are one of the obvious and most impressive applications of LLM technologies, and their development has gone hand in hand with massive impacts to markets and job growth. There are numerous players vying to create the best new LLM for all sorts of applications, and many would argue no company and their products in this space have more of a significant impact than OpenAI.

GPT‑5.3‑Codex is a truly impressive installment in this quest to create the best model. OpenAI promises that GPT-5.3-Codex is their most capable Codex model yet, advancing both coding performance and professional reasoning beyond GPT-5.2-Codex. Benchmark results show state-of-the-art performance on coding and agentic benchmarks like SWE-Bench Pro and Terminal-Bench, reflecting stronger multi-language and real-world task ability. Furthermore, the model is ~25% faster than GPT-5.2-Codex for Codex users thanks to infrastructure and inference improvements. Overall, GPT‑5.3‑Codex might be the most powerful agentic coding model ever released (Source).

So let’s see what it can do. Now available on the DigitalOcean GradientTM AI Platform and all OpenAI ChatGPT and Codex resources, we can test the model to see how it performs. In this tutorial, we will show how to use Codex to write a completely new project from scratch. We are going to make a Z-Image-Turbo Real-Time image-to-image application using GPT‑5.3‑Codex, without any user coding! Follow along to learn what GPT‑5.3‑Codex has to offer, how to use GPT‑5.3‑Codex for yourself, and a guide to vibe coding new web applications from scratch!

Key Takeaways

State-of-the-Art Agentic Performance: GPT-5.3-Codex delivers impressive results across software engineering and agentic tasks, outperforming GPT-5.2-Codex in reasoning, multi-language capability, and real-world coding evaluations like SWE-Bench Pro and Terminal-Bench 2.0.
Getting Started with GPT-5.3-Codex on GradientTM AI Platform is easy: All you need is access to the DigitalOcean Platform to begin integrating your LLM’s calls seamlessly into your workflows at scale.
From Prototype to Production in Record Time: With roughly 25% improved speed and real-time interactive steering, GPT-5.3-Codex feels less like a static generator and more like a responsive engineering partner capable of iterating, debugging, and refining projects alongside you. By handling scaffolding, architecture decisions, edge cases, and deployment-ready details, GPT-5.3-Codex can dramatically compress development timelines, making it possible to ship fully functional applications from scratch more quickly than ever (Source).

GPT‑5.3‑Codex Overview

GPT-5.3-Codex is a major agentic coding model upgrade that combines stronger reasoning and professional knowledge with enhanced coding performance, runs about 25 % faster than GPT-5.2-Codex, and excels on real-world and multi-language benchmarks like SWE-Bench Pro and Terminal-Bench. It’s designed to go beyond simple code generation to support full software lifecycle tasks (e.g., debugging, deployment, documentation) and lets you interact and steer it in real time while it’s working, making it feel more like a collaborative partner than a generator. It also has expanded capabilities for long-running work and improved responsiveness, with broader availability across IDEs, CLI, and apps for paid plans. (Source)

As we can see from the table above, GPT‑5.3‑Codex is a major step forward over GPT‑5.2‑Codex across software engineering, agentic, and computer use benchmarks. This, paired with the marked improvement in efficiency, make for an incredible indicator of how great this model is. We think this is a significant upgrade to previous GPT Codex model users, as well as new users looking for a powerful agentic coding tool to aid their process.

Getting Started with GPT-5.3-Codex

There are two ways to get started with GPT-5.3-Codex that we recommend to developers. First, is accessing the model with Serverless Inference through the GradientTM AI Platform. With Serverless Inference, we can Pythonically integrate the LLM generations into any pipeline. All you need to do is create a model access key, and begin generating! For more information on getting started, check out the official documentation.

The other way to get started quickly is the official OpenAI Codex application. It’s easy to get started with Codex on your local machine. Simply download the application onto your computer, and launch it. You will then be prompted to log in to your account. From there, simply choose which project you wish to work in, and you’re ready to get started!

Vibe Coding a Z-Image-Turbo Web Application with GPT‑5.3‑Codex

So now that we have heard about how GPT‑5.3‑Codex performs, let’s see it in action. For this experiment, we sought to see how the model performed on a relatively novel assignment that has a basis in past applications. In this case, we asked it to create a real-time image-to-image pipeline for Z-Image-Turbo that uses webcam footage as image input.

To do this, we created a blank new directory/project space to work in. We then asked the model to create a skeleton of the project to begin, and then iteratively added in the missing features on subsequent queries. Overall, we were able to create a full working version of the application with just 5 prompts and 30 minutes of testing. This extreme speed made it possible to ship the project in less than a day, from inspiration to completion. Now let’s take a closer look at the application project itself.

This project, which can be found here, is a real-time webcam-driven image-to-image generation application built in Python around a Gradio interface and a dedicated Z-Image-Turbo inference engine, where the UI in app.py presents side-by-side live input and generated output panes, parameter controls, and explicit Start/Stop gating so inference only runs when requested, while the backend in inference.py loads Tongyi-MAI/Z-Image-Turbo via ZImageImg2ImgPipeline, introspects the pipeline signature to bind the correct image-conditioning argument, enforces true img2img semantics instead of prompt-only generation, and executes inference in torch.inference_mode() with dynamic argument wiring so behavior adapts to the installed diffusers API. Critically, it can compute per-frame target resolution from webcam aspect ratio, snapping dimensions to a model-friendly multiple (default 16), and caps both sides below 1024, then applies post-generation safeguards that made the app stable in practice: dtype strategy (auto preferring bf16 then fp32, avoiding fp16 black-frame failure modes), degenerate-output detection with automatic float32 recovery, robust PIL/NumPy/Tensor output decoding and normalization, effective-strength clamping to preserve source structure, frame-hash seed mixing so scene changes influence results, and configurable structure-preserving input blending, all parameterized in config.py and documented in the README.md, with runtime status reporting latency plus internal diagnostics (pipe, dtype, size, effective strength, blend, seed, warnings) so you can observe exactly how each frame is being processed.

Closing Thoughts

GPT-5.3-Codex feels less like an incremental update and more like a meaningful shift in how developers interact with code. The combination of stronger reasoning, benchmark gains seen in testing, and a noticeable speed improvement makes it clear that agentic coding is maturing into something even more production-ready. What once required hours of boilerplate, debugging, and manual wiring can now be orchestrated through iterative prompts and high-level direction. As we demonstrated with the Z-Image-Turbo real-time application, a fully functional project can move from blank directory to working prototype in much less time traditionally required. While the actual results and performance benefits you experience will vary based on specific project requirements, complexity, and individual developer workflows, we are confident that GPT-5.3-Codex provides a substantial upgrade and a meaningful step forward in agentic coding capability, as evidenced by its stronger reasoning and measurable benchmark gains.

We recommend trying out GPT-5.3-Codex in all contexts, especially with DigitalOcean’s GradientTM AI Platform!

Getting Started with Qwen3.5 Vision-Language Models

DigitalOcean — Tue, 17 Mar 2026 16:00:00 +0000

This article was originally written by James Skelton (Senior AI/ML Technical Content Strategist II)

Vision Language models are one of the most powerful and highest potential applications of deep learning technologies. The reasoning behind such a strong assertion lies in the versatility of VL modeling: from document understanding to object tracking to image captioning, vision language models are likely going to be the building blocks of the incipient, physical AI future. This is because everything that we can interact with that will be powered by AI - from robots to driverless vehicles to medical assistants - will likely have a VL model in its pipeline.

This is why the power of open-source development is so important to all of these disciplines and applications of AI, and why we are so excited about the release of Qwen3.5 from Qwen Team. This suite of completely open source VL models, ranging in size from .8B to 397B (with activated 17B) parameters, is the clear next step forward for VL modeling. They excel at bench marks for everything from agentic coding to computer use to document understanding, and nearly match closed source rivals in terms of capabilities.

In this tutorial, we will examine and show how to make the best use of Qwen3.5 using a Gradienttm GPU Droplet. Follow along for explicit instructions on how to setup and run your GPU Droplet to power Qwen3.5 to power applications like Claude Code and Codex using your own resources.

Key Takeaways

Qwen3.5 VL demonstrates the growing power of open multimodal AI. The fully open-source model suite spans from 0.8B to 397B parameters and achieves strong benchmark performance across tasks like coding, document understanding, and computer interaction, approaching the capabilities of leading proprietary models.
Its architecture enables efficient large-scale multimodal training. By decoupling vision and language parallelism strategies, using sparse activations, and employing an FP8 training pipeline, Qwen3.5 improves hardware utilization, reduces memory usage, and maintains high throughput even when training on mixed text, image, and video data.
Developers can deploy Qwen3.5 on their own infrastructure. With tools like Ollama and GPU Droplets, it is possible to run large Qwen3.5 models locally or in the cloud to power applications such as coding assistants, computer-use agents, and custom AI tools without relying on proprietary APIs.

Qwen3.5: Overview

Qwen3.5 is a fascinating model suite with a unique architecture. It “enables efficient native multimodal training via a heterogeneous infrastructure that decouples parallelism strategies across vision and language components” (Source). This helps to make it avoid uniform approaches’ inefficiencies, such as over-allocating compute to lighter modalities, synchronization bottlenecks between vision and language towers, memory imbalance across devices, and reduced scaling efficiency when both modalities are forced into the same parallelism strategy.

By leveraging sparse activations to enable overlapping computation across model components, the system reaches nearly the same training throughput as pure text-only baselines even when trained on mixed text, image, and video datasets. Alongside this, a native FP8 training pipeline applies low-precision computation to activations, Mixture-of-Experts (MoE) routing, and GEMM operations. Runtime monitoring dynamically preserves BF16 precision in numerically sensitive layers, reducing activation memory usage by roughly 50% and delivering more than a 10% training speed improvement while maintaining stable scaling to tens of trillions of tokens.

To further leverage reinforcement learning at scale, the team developed an asynchronous RL framework capable of training Qwen3.5 models across all sizes, supporting text-only, multimodal, and multi-turn interaction settings. The system uses a fully disaggregated training–inference architecture, allowing training and rollout generation to run independently while improving hardware utilization, enabling dynamic load balancing, and supporting fine-grained fault recovery. Through techniques such as end-to-end FP8 training, rollout router replay, speculative decoding, and multi-turn rollout locking, the framework increases throughput while maintaining strong consistency between training and inference behavior.

This system–algorithm co-design also constrains gradient staleness and reduces data skew during asynchronous updates, preserving both training stability and model performance. In addition, the framework is built to support agentic workflows natively, enabling uninterrupted multi-turn interactions within complex environments. Its decoupled architecture can scale to millions of concurrent agent scaffolds and environments, which helps improve generalization during training. Together, these optimizations produce a 3×–5× improvement in end-to-end training speed while maintaining strong stability, efficiency, and scalability (Source).

Qwen3.5 Demo

Getting started with Qwen3.5 is very simple. Thanks to the foresight of Qwen Team & their collaborators, their are numerous ways to access and run the Qwen3.5 model suite’s models from your own machine. Of course, running the larger models will require significantly more computational resources. We recommend at least an 8x NVIDIA H200 setup for the larger models in particular, though a single H200 is sufficient for this tutorial. We are going to use Ollama to power Qwen3.5-122B-A10B.

To get started, simply start up a GPU Droplet with an NVIDIA H200 with your SSH key attached, and SSH in using the terminal on your local machine. From there, navigate to the base directory of your choice. Create a new directory with mkdir to represent your new workspace, and change into the directory.

Creating a custom game with Qwen3.5 running on Ollama and Claude Code

For this demo, we are going to do something simple: create a Python based video game for one of the most popular Winter Olympics sports: curling. To get started, paste the following code into the remote terminal:

curl -fsSL https://ollama.com/install.sh | sh
ollama launch claude --model qwen3.5:122b

This will launch Claude Code. If everything worked, it should look like above. From here, we can begin giving instructions to our model to begin generating code!

For this demo, provide it with a base set of instructions. Try customizing the following input:

“I want to create a simple game of curling in python code. i want it to be playable on my computer. Please create a sample Python program.

Packages: pygame”

This will give you, if your model ran predictably, a python file named something like “curling_game.py” with a full game’s code inside. Simply download this file onto your local computer, open the terminal and run it with python3.11 curling_game.py. Our game looks like this:

But looks are deceiving: this game is far from playable in the one-shot state. It requires serious work to amend the code to make the game playable, especially for two players. We can either use Claude Code with Qwen3.5 to make those adjustments, switch to an Anthropic Model like Sonnet 4.6 or Opus 4.6, or make the changes manually. From this base state, it took Qwen3.5 over an hour and at least 10 requests to make the game playable. Time was notably constrained by the single H200 GPU deployment we used for this demo, but the code output leaves significant room for improvement nonetheless. We expect that Opus 4.6 could accomplish the same task in a much quicker time frame, given its optimization for Claude Code, relatively superior benchmark scores, and more optimized infrastructure for inference.

If you want to try it out, this file can be found on Github Gist.

Closing Thoughts

Qwen3.5 VL represents an important step forward for open-source multimodal AI, demonstrating that publicly available models can increasingly rival proprietary systems in capability while offering far greater flexibility for developers. With its scalable architecture, efficient training infrastructure, and strong performance across tasks like coding, document understanding, and computer use, the Qwen3.5 suite highlights the growing maturity of the open AI ecosystem. As tools like GPU Droplets and frameworks such as Ollama make deploying large models easier than ever, vision-language systems like Qwen3.5 are poised to become foundational components in the next generation of AI-powered applications and physical AI systems.

7 OpenClaw Security Challenges to Watch for in 2026

DigitalOcean — Thu, 12 Mar 2026 16:00:00 +0000

This article was originally written by Fadeke Adegbuyi (Manager, Content Marketing)

OpenClaw isn’t just another chatbot wrapper. It executes shell commands, controls your browser, manages your calendar, reads and writes files, and remembers everything across sessions. The project runs locally on your machine and connects to WhatsApp, Telegram, iMessage, Discord, Slack, and over a dozen other platforms via pre-built integrations. It functions as a truly connected personal assistant. As a result, the use cases people have dreamed up for OpenClaw are wild.

One user showed an OpenClaw agent making money on Polymarket by monitoring news feeds and executing trades automatically. Another gave their bot access to home surveillance cameras. Someone else unleashed subagents to apply for UpWork freelancing jobs on their behalf.

// Detect dark theme var iframe = document.getElementById('tweet-2019044317621567811-81'); if (document.body.className.includes('dark-theme')) { iframe.src = "https://platform.twitter.com/embed/Tweet.html?id=2019044317621567811&theme=dark" }

But this kind of access to your digital life comes with real consequences when things go wrong. And things have gone wrong. Security researchers found that the agent shipped with serious flaws that made it possible for attackers to hijack machines with a single malicious link. Meanwhile, Moltbook, a Reddit-style platform with over 2.8 million AI agents, had its database completely exposed, so anyone could take control of any AI agent on the platform.

None of this means you should avoid OpenClaw entirely. It means you should understand OpenClaw security challenges and take precautions before spinning up an agent with root access to your laptop. Running OpenClaw in an isolated cloud environment can help neutralize some of these risks—DigitalOcean's 1-Click Deploy for OpenClaw, for example, handles authentication, firewall rules, and container isolation out of the box so your personal machine stays out of the equation.

What are OpenClaw security challenges?

OpenClaw security challenges boil down to a design tension: the tool needs broad system permissions to be useful, but those permissions create a massive attack surface when something goes wrong. The agent runs with whatever privileges your user account has—full disk, terminal, and network access—by design.

It's also agentic and self-improving, meaning it can modify its own behavior, update its memory, and install new skills autonomously. This is impressive from a capability standpoint, but another vector that can cause things to spiral when guardrails are missing. Pair that with defaults that skip authentication, an unvetted skill marketplace, and persistent memory storing weeks of context, and trouble follows. The takeaway: approach with caution, isolate from production systems, and carefully scrutinize the defaults.

To his credit, OpenClaw creator Peter Steinberger has been openly vocal about these risks and actively encourages running OpenClaw in a sandboxed environment, which isolates tool execution inside Docker containers to limit filesystem and process access when the model misbehaves. DigitalOcean's one-click deployment does exactly this out of the box, giving you that isolation without the manual setup.

7 OpenClaw security challenges to watch out for

We've already seen a security audit uncover 512 vulnerabilities (eight critical) and malicious ClawHub skills stealing cryptocurrency wallets. None of these challenges are theoretical. They're all based on incidents that have already played out within weeks of OpenClaw’s launch.

These are the challenges you need to have on your radar if you're experimenting with OpenClaw:

1. One-click remote code execution through WebSocket hijacking

One of the most alarming OpenClaw vulnerabilities discovered so far is CVE-2026-25253, a one-click remote code execution flaw that Mav Levin, a founding researcher at DepthFirst, disclosed in late January 2026. The attack worked because OpenClaw's local server didn’t validate the WebSocket origin header—so any website you visited could silently connect to your running agent. An attacker just needed you to click one link. From there, they chained a cross-site WebSocket hijack into full code execution on your machine. The compromise happened in milliseconds. This is the core danger of running an agent locally on the same machine you're browsing the web with—one careless click and an attacker is already inside.

Levin's proof-of-concept showed that visiting a single malicious webpage was enough to steal authentication tokens and gain operator-level access to the gateway API—giving an attacker access to change your config, read your files, and run commands.

Security checks: In this instance, the fix landed in version 2026.1.29, so update immediately if you’re a version behind. Beyond that, best practices include avoiding running OpenClaw while browsing untrusted sites and considering putting the agent behind a reverse proxy with proper origin validation for an additional layer of protection.

2. Tens of thousands of unprotected OpenClaw instances sitting open on the internet

Here's the thing about OpenClaw's early defaults: the agent trusted any connection from localhost without asking for a password. That sounded fine until the gateway sits behind a misconfigured reverse proxy—at which point every external request got forwarded to 127.0.0.1, and your agent thought the whole internet was a trusted local user. SecurityScorecard's STRIKE team found over 30,000 internet-exposed OpenClaw instances.

Security researcher Jamieson O'Reilly showed just how bad this gets. He accessed Anthropic API keys, Telegram bot tokens, Slack accounts, and complete chat histories from exposed instances, even sending messages on behalf of users and running commands with full admin privileges. No authentication required.

This has since been addressed—gateway auth is now required by default, and the onboarding wizard auto-generates a token even for localhost.

// Detect dark theme var iframe = document.getElementById('tweet-2015401219746128322-801'); if (document.body.className.includes('dark-theme')) { iframe.src = "https://platform.twitter.com/embed/Tweet.html?id=2015401219746128322&theme=dark" }

Security checks: At a minimum, check whether your instance is reachable from the public internet. Use a firewall to restrict access, enable gateway token authentication, and never expose the control plane without a VPN or SSH tunnel in front of it. This is a case where a managed cloud deployment can solve the problem outright—because your personal API keys, chat histories, and credentials aren’t sitting on an exposed local machine in the first place.

3. Malicious skills on ClawHub are poisoning the supply chain

ClawHub, OpenClaw's public skill marketplace, lets anyone publish an extension—the only requirement is a GitHub account older than one week. That low bar has unfortunately turned the marketplace into a target. Koi Security audited all 2,857 skills on ClawHub and found 341 that were outright malicious. Bitdefender's independent scan put the number closer to 900 malicious skills, roughly 20% of all packages. A single account—"hightower6eu"—uploaded 354 malicious packages by itself.

The attack is clever. You install what looks like a useful skill and the documentation looks professional. But buried in a "Prerequisites" section, it asks you to install something first—and that something is Atomic Stealer (AMOS), a macOS credential-stealing malware.

Security checks: OpenClaw has since partnered with VirusTotal to scan new skill uploads, but Steinberger himself admitted this isn't a silver bullet. At a minimum, before installing any skill, read its source code. Check the publisher's account age and history. Put simply, treat every skill as untrusted code running with your agent's full permissions. Unlike some exposure risks, malicious skills are a threat regardless of where OpenClaw runs—a poisoned skill executes the same way on a cloud server as it does on your laptop.

4. Credential storage in plaintext and API key leakage

One of the less glamorous but more dangerous issues is how OpenClaw handles secrets. The platform stores credentials in plaintext—including API keys for your LLM provider and tokens for every messaging platform your agent connects to—and those become targets the moment your instance is accessible to anyone other than you. Prompt injection attacks can also trick the agent into exfiltrating credentials by embedding hidden instructions in content the agent processes.

Cisco's team tested a skill called "What Would Elon Do?" and surfaced nine security findings, two of them critical. The skill instructed the bot to execute a curl command sending data to an external server controlled by the skill's author. Functionally, it was malware hiding behind a joke name.

Security check: At a minimum, rotate your API keys regularly and store secrets using environment variables or a dedicated secrets manager rather than config files. It's also worth setting spending limits on your LLM provider accounts. That way, even if a key is compromised, it can't rack up thousands in charges.

5. Prompt injection attacks amplified by persistent memory

What makes prompt injection in OpenClaw worse than in a typical chatbot is the persistent memory. The agent retains long-term context, preferences, and conversation history across sessions—which is one of its best features. But it also means a malicious instruction embedded in a website, email, or document doesn't have to execute immediately. Palo Alto Networks warned that these become "stateful, delayed-execution attacks". A hidden prompt in a PDF you opened last Tuesday could sit dormant in the agent's memory until a future task triggers it days later.

Security check: There's no perfect fix for prompt injection right now; it's an unresolved problem in agentic AI. But you can reduce the blast radius by limiting what tools and permissions your agent has access to, segmenting its access to sensitive systems, and reviewing its memory and context periodically for anything unexpected.

6. Shadow AI spreading through enterprise networks

This one's for anyone working at a company where developers tinker on their work machines. Token Security found that 22% of their enterprise customers have employees running OpenClaw as shadow AI without IT approval. Bitdefender confirmed the same, showing employees deploying agents on corporate machines connected to internal networks. An OpenClaw agent on a developer's laptop with VPN access to production means every vulnerability above is now a business problem.

Security check: If you're on a security team, you should scan your network for OpenClaw instances now. Set up detection for its WebSocket traffic patterns, and mandate that any approved use runs in an isolated environment—a VM or cloud server—rather than on laptops with internal access. Giving teams an approved, isolated deployment path is the fastest way to get ahead of shadow AI—it's much easier to enforce guardrails when the alternative isn't 'don't use it at all.'

7. The Moltbook database breach exposing millions of agent credentials

The security mess isn't limited to OpenClaw itself. Moltbook, the social network for AI agents built by Matt Schlicht, suffered a database exposure that cybersecurity firm Wiz discovered in early February. The database had zero access controls. Anyone who found it could view 1.5 million API tokens, 35,000 email addresses, and private messages between agents—enough to take control of any agent on the platform. China's Ministry of Industry and Information Technology issued a formal warning about OpenClaw security risks, citing incidents like this breach.

Security check: If you've used Moltbook, rotate every API key and token associated with your agent. Treat third-party platforms in the OpenClaw ecosystem with the same skepticism you'd apply to any new service asking for your credentials and consider additional security checks.

Any references to third-party companies, trademarks, or logos in this document are for informational purposes only and do not imply any affiliation with, sponsorship by, or endorsement of those third parties.

Pricing and product information accurate as of February 2026.

GPU Programming for Beginners: ROCm + AMD Setup to Edge Detection

DigitalOcean — Tue, 10 Mar 2026 16:00:00 +0000

In this hands-on tutorial, we demystify GPU computation and show you how to write your own GPU programs from scratch. Understanding GPU programming is essential for anyone looking to grasp why AI models depend on this specialized hardware.

We'll use ROCm and HIP (AMD's version of CUDA) to take you from zero to running real GPU code, culminating in a computer vision edge detector that processes images in parallel.

You can find the code in the project repository: https://github.com/oconnoob/intro_to_rocm_hip/blob/main/README.md

👇 WHAT YOU'LL LEARN IN THIS VIDEO 👇

🔧 Getting Set Up with ROCm Two ways to get started: spin up a GPU Droplet on DigitalOcean with ROCm pre-installed, or install ROCm yourself on an Ubuntu system with an AMD GPU. We cover both methods step-by-step.

➕ Example 1: Vector Addition (The Basics) Learn the fundamental structure of GPU programs—kernels, threads, blocks, and memory management. We'll add one million elements in parallel and verify our results.

⚡ Example 2: Matrix Multiplication (Why Libraries Matter) Discover why optimized libraries like rocBLAS dramatically outperform naive implementations. This is the operation powering most AI models you use daily.

👁️ Example 3: Edge Detection with Sobel Filter (The Cool Stuff) Apply your GPU programming skills to a real computer vision problem—detecting edges in images using a classic Sobel filter, all running massively parallel on the GPU.

Whether you're an AI enthusiast wanting to understand the hardware layer or a developer looking to harness GPU compute power, this tutorial gives you the foundation to start writing efficient parallel programs.

February 2026 DigitalOcean Tutorials: Claude 4.6 and AI Agents

Jess Lulka — Thu, 05 Mar 2026 17:00:00 +0000

Whether you’ve found yourself exploring Anthropic’s latest Claude Opus 4.6 release or following along with the OpenClaw frenzy, DigitalOcean has tutorials and guides to help you get the most out of the latest AI advancements.

These 10 tutorials from last month cover AI agent development, RAG troubleshooting, CUDA performance tuning, and OpenClaw on DigitalOcean. Bookmark them for later or keep them open among your 50 browser tabs to come back to.

What’s New With Claude Opus 4.6

Claude Opus 4.6’s agentic coding model feels less like a coding assistant and more like a collaborative engineer. Developers now have a massive 1M-token context window, which lets the model reason across entire codebases, docs, and long workflows without constantly re-prompting. This means faster refactors, more reliable debugging, and the ability to make iterative UI or architecture changes with just a few guided prompts. Long context plus agentic planning dramatically reduces the time between the idea and working implementation, especially when the model is directly integrated into your cloud stack.

Self-Learning AI Agents: A High-Level Overview

Self-learning agents follow a fundamental loop: observe, act, get feedback, and improve. For developers, these systems aren’t just prompt-driven. They’re built around policies, reward signals, and evolving memory. We make the concept approachable by showing how you can prototype simple versions with standard Python ML tooling. This tutorial can help you determine whether your agent needs to adapt to changing environments or user behavior. You’ll also get a look at how reinforcement-style learning and persistent memory become essential design choices.

CUDA Guide: Workflow for Performance Tuning

Frustrated by the guesswork involved in GPU optimization? We’ve got a step-by-step guide for you. Learn how to profile first, identify the real bottleneck—memory, compute, or occupancy—and then apply targeted optimizations rather than random tweaks. For developers working with AI or HPC workloads, the biggest win is understanding that most performance gains come from a structured workflow, not exotic kernel tricks. You’ll learn that knowing how to measure, optimize, and re-measure is the only reliable path to predictable CUDA speedups.

A Simple Guide to Building AI Agents Correctly

This tutorial is a production blueprint for agentic systems. It covers why naive agent loops fail—runaway costs, hallucinated tool calls, and silent errors—and provides a modular architecture that includes an orchestrator, structured tools, memory, guardrails, and full observability. The most valuable takeaway for real deployments is the “start with the least autonomy” principle: Use deterministic workflows first, and add agent behavior only where it’s truly needed. You want to treat agents like serious software systems with testing, logging, and permissions, not clever prompt chains to get them running correctly.

Why Your RAG Is Not Working Effectively

If your RAG app feels inaccurate or inconsistent, this tutorial helps you diagnose the real cause; it’s usually retrieval quality, chunking strategy, or missing evaluation rather than the model itself. You’ll walk through concrete fixes like better indexing, query rewriting, and relevance filtering so your system actually returns grounded answers. The key takeaway is that RAG performance is mostly a data-pipeline and retrieval-engineering problem, not an LLM problem.

How to Connect Google to OpenClaw

If you’re looking for how to connect AI assistants to real-time data, this guide shows how to wire external data sources into your agent workflow so it can act on real user content instead of static prompts. The practical win is learning how authentication, connectors, and permissions shape what your agent can safely do in production. You'll learn how to deploy OpenClaw on a DigitalOcean Droplet and connect it to Google services like Gmail, Calendar, and Drive using OAuth authentication.

So You Installed OpenClaw on a DigitalOcean Droplet. Now What?

We’ve penned plenty of resources on how to get started with OpenClaw on DigitalOcean (how to run it and how we built a security-hardened Droplet). This follow-up focuses on moving from a working prototype to a more capable, extensible system. You learn how to layer in new tools, expand automation flows, and structure your project so it scales beyond a demo. The key takeaway is architectural: design your agent environment so new capabilities are plug-and-play rather than requiring rewrites.

Effective Context Engineering to Build Better AI Agents

The prompts you feed your AI agent matter just as much as the model behind it. Instead of cramming everything into a single prompt, this article shows you how to structure memory, retrieval, tool outputs, and task state so the model always sees the right information at the right time. You’ll see how using enough context is your real control surface for agent reliability, latency, and cost. Good context engineering often beats switching to a larger model.

Sliding Window Attention: Efficient Long-Context Modeling

Sliding window attention makes long-context transformers far more practical by limiting how many tokens each position can “see.” Instead of every token attending to every other token (which gets expensive fast), the model focuses on a fixed local window—cutting compute costs from quadratic to linear growth. You’ll get a breakdown of how this works, how modern variants improve positional awareness, and why it’s especially useful for long documents, extended chat histories, or agent memory systems. Smarter attention design—not just bigger models—is what makes long-context AI scalable.