DEV Community

bytebantz
bytebantz

Posted on

Mastering Datadog AI Agent Observability: A Beginners Guide

#ai #llm #webdev

Agent Observability Is a Must-Have for LLM Applications

Originally published on Medium.

When something goes wrong, identifying the actual cause isn’t always straightforward due to non-deterministic nature of LLM applications.

Adding logs helps, but logs alone often aren’t enough for LLM applications.

You might have no visibility of the complete workflow if you are using logs alone.

This is where Datadog Agent Observability comes in.

Datadog gives you end-to-end visibility into your AI application’s execution.

Instead of guessing what happened, you can trace a user’s request through out the complete execution flow.

Implementing Datadog agent observability involves 3 core components:

⚪ Traces — shows the complete journey of a single request.

⚪🟡 Spans — represent individual operations within a trace.

⚪🟡⚪ Instrumentation is the process of capturing this information automatically or manually and sending it to Datadog for analysis.

Press enter or click to view image in full size

1. TRACE

A trace represents the complete execution path of a single request handled by the LLM application.

A trace captures everything that happens while the application processes that request, including LLM calls, agent decisions, tool calls, and other operations.

A trace can contain multiple spans, where each span represents an individual operation within the overall request.

LLM Agent workflow

An agent trace groups together all workflows and contextual operations such as the tool calls and reasoning steps executed by an agent to complete a user request.

For example, a user asks:

"What's the weather in Nairobi?"
Enter fullscreen mode Exit fullscreen mode

The AI agent might:

  1. Receive the user’s request.

  2. Call an LLM to determine what to do.

  3. Call a weather tool.

  4. Send the weather data back to the LLM.

  5. Generate the final response.

All of these operations can be represented within the same trace.

Trace Structure Example:

Press enter or click to view image in full size

TRACE
────────────────────────────────
trace_id: trace_001
────────────────────────────────

Agent / Workflow Span
span_id: span_001
trace_id: trace_001
│
├── LLM Span
│   span_id: span_002
│   trace_id: trace_001
│
│   Input:
│   "What's the weather in Nairobi?"
│
│   Output:
│   "I need to check the weather."
│
├── Tool Span
│   span_id: span_003
│   trace_id: trace_001
│
│   Input:
│   {
│     "location": "Nairobi"
│   }
│
│   Output:
│   {
│     "temperature": 23,
│     "condition": "Partly cloudy"
│   }
│
└── LLM Span
    span_id: span_004
    trace_id: trace_001

    Input:
    Weather data from tool

    Output:
    "It's 23°C and partly cloudy in Nairobi."

────────────────────────────────
Enter fullscreen mode Exit fullscreen mode

TRACE SAMPLING

Trace Sampling is a method datadog uses to determine the number of traces to retain for observability.

Agent Observability billing is based on the volume of spans sent.

Setting a sample rate is a direct way to control the cost for agent obervability.

Example:

Suppose your application receives:
10,000 agent requests


And each request generates approximately:
10 spans

You could therefore send to DataDog approximately:
10,000 × 10 = 100,000 spans
Enter fullscreen mode Exit fullscreen mode

Since Agent Observability billing is based on the volume of spans sent, this can become expensive at scale.

This is where trace sampling comes in.

You can configure the sample rate (a float between 0.0 and 1.0)

SETTING A SAMPLE RATE

Suppose we configure:

sample rate = 0.5
Enter fullscreen mode Exit fullscreen mode

The above sample rate means approximately 50% of traces (50,000) will be retained instead of 100,000.

While trace sampling allows you to control the ingestion volume and storage costs for your individual traces and spans, it does not impact these operational metrics

Such metrics includes:

🟡 Token metrics

🟡🟡 Cost metrics

🟡🟡🟡 Performance metrics

Press enter or click to view image in full size

2. SPAN

A span represents a single operation in a trace.

A span serves as the fundamental building block of a trace

Press enter or click to view image in full size

1. SPAN ATTRIBUTES

A span attribute describes the operation represented by the span.

Span attributes represent the content of the work being done within the span

For example:
Suppose the span represents this operation:

SELECT * FROM users WHERE id = 123;
Enter fullscreen mode Exit fullscreen mode

The span could contain the following span attributes:

@db.system:postgres
@db.operation:SELECT
@db.table:users
Enter fullscreen mode Exit fullscreen mode

WHERE DO SPAN ATTRIBUTES COME FROM?

Automatic instrumentation

Datadog’s APM libraries can automatically collect information.

This means there is no need to write code to record values for such fields.

For example, your application makes an HTTP request:

GET /products
Enter fullscreen mode Exit fullscreen mode

The instrumentation may automatically record things like:

@http.method:GET
@http.route:/products
Enter fullscreen mode Exit fullscreen mode
2. Manual/custom instrumentation

This means we have to explicitly specify the fields to record.

This is useful because it allows you to put business-specific information into your traces.

For example:
Suppose your application calculates the shopping cart value:

cart_total = 1299.99
Enter fullscreen mode Exit fullscreen mode

You might instrument your application so the span contains:

@cart.total:1299.99
Enter fullscreen mode Exit fullscreen mode

Example:

from ddtrace import tracer

# Get the current active span
span = tracer.current_span()
if span:
    # Manually set a custom attribute
    span.set_tag('cart.total', 1299.99)
Enter fullscreen mode Exit fullscreen mode

RESERVED SPAN ATTRIBUTES

Datadog has certain reserved span attributes

Reserved span attributes are automatically instrumented and are available on every span.

You do not need to perform any manual setup to ensure these attributes are collected for your traces:

env: environment name (e.g. production) where the service is running.

service: The name of the service that generated the span.

operation_name: The specific name of the operation being performed.

resource_name: A human-readable name for the resource associated with the operation (e.g., an endpoint path or a database query).

status: The operational status of the span (e.g., ok or error).

span_id: The unique identifier for that specific span.

trace_id: The unique identifier that groups all spans belonging to the same request or workflow.

timestamp: The exact time the span occurred.

type: The category of operation (e.g., web, db, or cache).

link: References connecting the span to other spans or traces.

ingestion_reason: The reason why the trace or span was retained by the Datadog ingestion pipeline.

When working with reserved attributes, don’t put @ in front.

For example:

service:checkout
env:production
status:error
Enter fullscreen mode Exit fullscreen mode

rather than:

@service:checkout
@env:production
Enter fullscreen mode Exit fullscreen mode

2. SPAN TAGS

Span tags are contextual information about where the span came from for example infrastructure/environment context.

Tags don’t require @.

For example:

hostname:web-server-17
availability-zone:us-east-1a
cluster-name:production-cluster
container_name:api-frontend
Enter fullscreen mode Exit fullscreen mode

These tags provide context about the infrastructure or orchestration environment where the service is running

3. ANNOTATIONS

Annotations are additional pieces of context that you can attach to a span to make the trace more useful for analysis.

Annotations describe business-specific context about an operation.

This helps explain operations from a business perspective.

Example:

A checkout span could have annotations like customer_tier=premium or order_type=subscription.

Spans automatically capture common information such as: duration, status, errors, input, output

But sometimes you want to add your own context.

Example:
Suppose you’re building a customer-support AI.

You may want to know additional information about each request, such as:

In Datadog, annotations are filtered using attribute search syntax, where attributes are prefixed with an @ symbol.

@customer_tier:premium 
@language:en 
@ticket_category:refund 
@department:electronics
Enter fullscreen mode Exit fullscreen mode

Using ddtrace, you can attach this business context to the current span with LLMObs.annotate():

from ddtrace.llmobs import LLMObs
from ddtrace.llmobs.decorators import tool
@tool
def handle_support_request(message, customer_id):
    response = process_support_request(message, customer_id)
    LLMObs.annotate(
        tags={
            "customer_tier": "premium",
            "language": "en",
            "ticket_category": "refund",
            "department": "electronics"
        }
    )
    return response
Enter fullscreen mode Exit fullscreen mode

The resulting span now contains both its normal observability information and your additional business context:

Tool Span: handle_support_request
────────────────────────────────────────
Duration:
    4.2 sec
Status:
    OK
Input:
    "I want a refund for my headphones."
Output:
    "I'll help you process your refund..."
Annotations:
    customer_tier:   premium
    language:        en
    ticket_category: refund
    department:      electronics
────────────────────────────────────────
Enter fullscreen mode Exit fullscreen mode

Using annotations, allow you to filter, group, and analyze traces based on your application’s specific business context.

For example, you could filter traces where:

@customer_tier:premium
Enter fullscreen mode Exit fullscreen mode

to investigate premium-customer requests.

SPANS CATEGORIZATION

Datadog allows spans to be categorized according to the type of operation they represent.

Instead of representing every operation as a generic span (Span), Datadog allows spans to be categorized according to the type of operation they represent:

Categorizing spans makes your traces:

🟢- Easier to understand

🟢🟢- Easier to troubleshoot

🟢🟢🟢- Easier to analyze

Press enter or click to view image in full size

TYPES OF SPANS

Press enter or click to view image in full size

1. Agent Span

An Agent is an AI system that can decide what actions it needs to take and in what order to accomplish a goal.

An Agent span represents the complete execution of an AI agent.

Example:
Consider a travel-planning agent that helps a user plan a trip.

The agent can be implemented as:

from ddtrace.llmobs.decorators import agent
from openai import OpenAI

llm = OpenAI()

# Define tools
def search_flights(destination):
    # Tool execution logic
    pass

def search_hotels(destination):
    # Tool execution logic
    pass

def check_weather(destination):
    # Tool execution logic
    pass

@agent
def plan_trip(customer_query):
    # The LLM analyzes the request and decides which tools to use
    llm_response = llm.chat.completions.create(
        model="gpt-4",
        messages=[{"role": "user", "content": customer_query}],
        tools=[
            search_flights,
            search_hotels,
            check_weather
        ]
    )

    return llm_response.choices[0].message.content
Enter fullscreen mode Exit fullscreen mode

2. Root Span

A root span is defined by its position in the trace hierarchy.

A span is considered a root span, if it is the span at the very top, with no parent span.

An agent span may be the root span, particularly when the agent execution is the top-level operation you’re tracing.

But if the agent is itself invoked by another operation — such as a workflow — it can be a child span.

Example:

def travel_workflow(customer_query):
    validate_request(customer_query)

    plan_trip(customer_query)

    save_itinerary()
    return "Workflow complete"
Enter fullscreen mode Exit fullscreen mode

Conceptually, the trace looks like this:

ROOT SPAN: travel_workflow
│
├── Span: validate_request
│
├── Agent Span: plan_trip
│   │
│   ├── LLM Span: understand_requirements
│   │
│   ├── Tool Span: search_flights
│   │
│   ├── Tool Span: search_hotels
│   │
│   ├── LLM Span: compare_options
│   │
│   ├── Tool Span: check_weather
│   │
│   └── LLM Span: generate_itinerary
│
└── Span: save_itinerary
Enter fullscreen mode Exit fullscreen mode

3. Workflow Span

A workflow span represents the execution of a complete, high-level process within an application.

A workflow in Agents Observability is a predefined sequence of operations that may include LLM calls, tool calls, data processing, and other steps required to complete a task.

A Workflow Span represents the complete execution of the workflow.

Unlike an Agent, which can autonomously decide which actions to take, a workflow generally follows a predefined sequence of operations defined by the application.

Example:
Consider a customer support ticket workflow:

The workflow can be implemented using the @workflow decorator:

from ddtrace.llmobs.decorators import workflow, llm, tool, retrieval, task

@tool
def retrieve_customer_data(customer_id):
    # Logic to fetch customer information from an external database
    pass

@llm
def classify_ticket(ticket):
    # LLM call to categorize the support ticket
    pass

@retrieval
def search_knowledge_base(category):
    # Logic to search a vector database or external KB
    pass

@llm
def generate_response(ticket, customer_data, knowledge):
    # LLM call to generate the ticket response
    pass

@task
def send_response(response):
    # Logic to format/send the response (no external call)
    pass

@workflow
def process_support_ticket(ticket):
    customer_data = retrieve_customer_data(ticket.customer_id)
    ticket_category = classify_ticket(ticket)
    knowledge = search_knowledge_base(ticket_category)
    response = generate_response(ticket, customer_data, knowledge)
    send_response(response)

    return response
Enter fullscreen mode Exit fullscreen mode

4. Task Span

A task span represents an individual processing operation performed by an application that don’t call an external service.

It is typically used for application-level logic that happens within the application itself, rather than an operation that calls an external service.

Example:
Consider an application that calculates a discount for a customer.

The task can be implemented using the @task decorator:

from ddtrace.llmobs.decorators import task

@task
def calculate_discount(price, customer_type):
    if customer_type == "premium":
        return price * 0.80
    return price
Enter fullscreen mode Exit fullscreen mode

5. LLM Span

An LLM span represents an LLM call.

It captures the model invocation, including the model used, input, output, token usage, latency, and status.

For example:

from ddtrace.llmobs.decorators import llm

@llm(model_name="gpt-4")  # model_name is optional
def generate_product_description(product_data):
    response = llm.complete(
        prompt=f"Generate description for: {product_data['name']}"
    )
    return response.text
Enter fullscreen mode Exit fullscreen mode

The above example could be represented as:

6. Tool Span

A tool represents calls to external services like databases, APIs made by an AI application.

A tool span represents the details of the external operation, such as what tool was called, the input provided, the result returned, and how long the operation took.

Example:
Consider an AI customer-support assistant, that calls an external API to fetch order details:

from ddtrace.llmobs.decorators import tool

@tool
def fetch_inventory_status(product_id):
    inventory = database.query(
        "SELECT quantity, location FROM inventory WHERE product_id = ?",
        product_id
    )
    return inventory
Enter fullscreen mode Exit fullscreen mode

7. Embedding span

An Embedding Span represents the operation of converting text or other data into an embedding vector using an embedding model.

Example:

from ddtrace.llmobs.decorators import embedding

@embedding(model_name="text-embedding-ada-002")  # model_name is required, model_provider is optional
def generate_product_embedding(product_text):
    vector = embedding_model.encode(product_text)
    return vector.tolist()
Enter fullscreen mode Exit fullscreen mode

8. Retrieval Span

A retrieval span represents the operation of searching for and retrieving relevant information from a data source, such as a vector database.

The retrieval span captures the search operation and the information returned by it.

Example:

For example, in an AI shopping assistant, a user might ask:

“Find me a black running shoe under $100.”

The retrieval operation can be instrumented using:

from ddtrace.llmobs.decorators import retrieval

@retrieval
def search_product_catalog(query):
    query_vector = generate_embedding(query)
    results = vector_db.similarity_search(query_vector, top_k=5)
    return results
Enter fullscreen mode Exit fullscreen mode

Conclusion

Agent observability helps you see what happens inside your AI application when it processes a request.

Datadog uses traces, spans, and instrumentation to help you follow a request from beginning to end, inspect individual steps, and find out why something failed or took too long.

Logs are useful, but observability gives you a clearer picture of how the different parts of your application work together.

Top comments (0)