DEV Community

Cover image for Every Token Leaves a Trace: Benchmarking Ollama with SigNoz
Aravind Kannan Rathinasabapathi
Aravind Kannan Rathinasabapathi

Posted on

Every Token Leaves a Trace: Benchmarking Ollama with SigNoz

Local Large Language Models (LLMs) have made it possible to run powerful AI models entirely on your own machine. They offer greater privacy, eliminate external API costs, and make rapid experimentation much easier. With tools like Ollama, launching a model locally is often as simple as running a single command.

But once the model starts generating responses, visibility quickly disappears.

How fast is the model generating tokens? Where is inference time actually spent? And what is the performance impact of moving from a lightweight 1.5B model to an 8B model?

These are observability questions.

Instead of building a benchmark from scratch, we'll build on top of the excellent open-source aidatatools/ollama-benchmark project. By wrapping it with OpenTelemetry, we'll capture benchmark metrics and export them to SigNoz over OTLP/HTTP.

By the end of this guide, you'll have a lightweight observability pipeline that transforms local LLM benchmark runs into traces, metrics, and dashboards you can analyze and compare over time.

📋 Prerequisites

Before getting started, ensure you have the following installed:

  • Python 3.10+: Required for the benchmarking wrapper.
  • Docker & Docker Compose (v20.10+): Required to run the self-hosted SigNoz stack.
  • Ollama: Ensure the Ollama daemon is running (ollama serve) so the benchmark can execute local inference.
  • Hardware: At least 8 GB RAM (16 GB or more recommended) to run both SigNoz and local LLMs comfortably.

🏗️ What We Are Building

Instead of waiting for a benchmark to finish and parsing a static text summary, we'll observe the benchmark as it runs.

Our monitoring pipeline wraps the benchmark execution with OpenTelemetry, capturing performance telemetry in real time and exporting it to SigNoz for visualization and analysis.

┌──────────────────┐
│ ollama-benchmark │
└────────┬─────────┘
         │ OpenTelemetry
         ▼
┌──────────────────┐
│ Python Wrapper   │
└────────┬─────────┘
         │ OTLP/HTTP :4318
         ▼
┌──────────────────┐
│ SigNoz           │
└──────────────────┘
Enter fullscreen mode Exit fullscreen mode

🛠️ Step 1: Spin Up SigNoz with Docker Compose

SigNoz is an open-source observability platform that collects and visualizes metrics, traces, and logs in a single interface.

We'll use Foundry, SigNoz's official configuration CLI, to bootstrap a standalone deployment using Docker Compose.

Before you begin, ensure Docker Engine (v20.10+) or Docker Desktop is running, then execute the following commands:

# 1. Download and install the official Foundry management binary wrapper
curl -fsSL https://signoz.io/foundry.sh | bash

# 2. Generate a clean declarative container environment target configuration file
cat <<EOF> casting.yaml
apiVersion: v1alpha1
kind: Installation
metadata:
  name: signoz
spec:
  deployment:
    flavor: compose
    mode: docker
EOF

# 3. Trigger the deployment engine pipeline to build configurations and boot the stack
foundryctl cast -f casting.yaml
Enter fullscreen mode Exit fullscreen mode

💡 What just happened?

The foundryctl cast command combines three steps into one:

  • gauge validates your system requirements.
  • forge generates the required Docker Compose configuration.
  • cast starts the SigNoz services.

⚠️ Common issue:

ClickHouse requires sufficient memory during startup. If the ClickHouse container crashes or repeatedly restarts with exit code 137, increase Docker's memory allocation to at least 4 GB under Docker Desktop → Settings → Resources.

Once all containers are healthy, open http://localhost:8080 in your browser and create your administrator account.

SigNoz is now ready to receive OpenTelemetry data on the default OTLP endpoints:

  • 4317 (gRPC)
  • 4318 (HTTP)

🐍 Step 2: Build the Benchmark Wrapper

Rather than modifying llm-benchmark, we'll wrap it with a lightweight Python script that adds OpenTelemetry instrumentation. The wrapper forwards benchmark execution to the original library while capturing performance metrics and traces, exporting them to SigNoz in real time.

Local Environment Setup

Create a virtual environment, download the wrapper from GitHub Gist, and install the required dependencies:

# Create a virtual environment
python3 -m venv venv

# Activate it (macOS/Linux)
source venv/bin/activate

# Windows
# .\venv\Scripts\activate

# Download the project files
curl -o requirements.txt https://gist.githubusercontent.com/Aravind-Kannan/9634d05c7fbe0ebd9be89ca514720b21/raw/bf12809472685c1f60e0f697e54850b6970735e6/requirements.txt
curl -o ollama_bench.py https://gist.githubusercontent.com/Aravind-Kannan/9634d05c7fbe0ebd9be89ca514720b21/raw/f53d04dcbdfedd9c2cdd6f8867b17c258441c8d8/ollama_bench.py

# Install dependencies
pip install --upgrade pip
pip install -r requirements.txt
Enter fullscreen mode Exit fullscreen mode

Running the Benchmark

The wrapper supports benchmarking one or more models using standard command-line arguments.

Default benchmark (recommended)

Run the wrapper without any arguments to benchmark the default set of models:

python ollama_bench.py
Enter fullscreen mode Exit fullscreen mode

Single model

Benchmark a specific model:

python ollama_bench.py --model gpt-oss:20b
Enter fullscreen mode Exit fullscreen mode

Multiple models

Benchmark multiple models in a single run:

python ollama_bench.py --model phi4:14b deepseek-r1:14b gpt-oss:20b
Enter fullscreen mode Exit fullscreen mode

Terminal animation showing the Python benchmark script sequentially pulling 'deepseek-r1:1.5b' and running evaluation prompts. The terminal displays real-time structured logging messages prefixed with 'RunScopedLogger' alongside live token throughput statistics.

💎 Exploring the Distributed Trace

Once the benchmark completes, open the Traces page in SigNoz (http://localhost:8080) and search for the root span llm_benchmark_execution.

Each benchmark run is visualized as a hierarchical trace, making it easy to follow the execution flow from model preparation to individual prompt execution.

Distributed trace of a benchmark run in SigNoz

What you'll see

  • Model preparation spans (pull_model.*) showing how long each model takes to become ready.
  • Benchmark spans grouping different benchmark categories, such as instruction following, question answering, and vision tasks.
  • Model spans for each model being evaluated.
  • Prompt spans (prompt.0, prompt.1, ...) that expose the latency of every prompt within a benchmark.

Why this is useful

Instead of relying on a single completion time, you can pinpoint exactly where time is spent during a benchmark. Whether a slowdown comes from pulling a model, a specific benchmark category, or an individual prompt, the waterfall view makes it immediately visible.

📊 Building a Performance Dashboard

Now that benchmark telemetry is flowing into SigNoz, let's build a dashboard to compare model performance.

Navigate to Dashboards → + New Dashboard and create a dashboard named Local LLM Performance.

Click + Add Panel and configure the following visualizations.

Panel 1: Prompt Throughput

  • Panel Type: Timeseries
  • Title: Prompt Tokens/sec
  • Metric: llm_benchmark.prompt_tokens_per_sec
  • Aggregation: avg
  • Group By: model

This produces a separate throughput trend for each model, making it easy to compare token generation over time.

Panel 2: Model Leaderboard

  • Panel Type: Table
  • Title: Throughput Leaderboard
  • Metric: llm_benchmark.tokens_per_sec
  • Aggregation: max
  • Group By: model

Sort the table by Value in descending order to quickly identify the fastest model.

Panel 3: Prompt Latency

  • Panel Type: Bar Chart
  • Title: Average Prompt Latency
  • Metric: llm_benchmark.prompt_duration_seconds
  • Aggregation: avg
  • Group By: model, prompt_index

This highlights how latency varies across prompts and models.

📈 The Final Dashboard

After configuring the panels, your dashboard should provide a consolidated view of model performance, making it easy to compare throughput and latency across benchmark runs.

SigNoz Timeseries Panel Configuration

Suggested caption: The completed Local LLM Performance dashboard comparing throughput and latency across multiple Ollama models.

Optional: Query with ClickHouse SQL

If you prefer writing SQL instead of using the Query Builder, switch to the ClickHouse SQL tab and run:

SELECT
    toStartOfInterval(timestamp, INTERVAL 5 SECOND) AS time,
    labels['model'] AS model,
    avg(value) AS value
FROM signoz_metrics.samples_v4
WHERE metric_name = 'llm_benchmark.prompt_tokens_per_sec'
  AND timestamp >= NOW() - INTERVAL 30 MINUTE
GROUP BY time, model
ORDER BY time;
Enter fullscreen mode Exit fullscreen mode

This query produces the same time-series data used by the Prompt Throughput panel, while giving you full control over aggregation and filtering using ClickHouse SQL.

📈 Results & Findings

To validate the observability pipeline, I benchmarked three Ollama models on the following hardware:

Hardware Specification
Laptop Apple M3 Pro
Memory 36 GB RAM

Benchmark Results

Model Peak Throughput (Tokens/sec)
gpt-oss:20b 30.71
deepseek-r1:14b 13.60
phi4:14b 13.22

Key takeaway: On this particular machine and benchmark configuration, gpt-oss:20b delivered the highest token throughput, more than doubling the throughput of the two 14B models.

Note: Higher throughput does not necessarily mean a model is better. It simply means the model generated tokens faster under this specific workload on my Apple M3 Pro laptop. Model quality, reasoning ability, accuracy, latency, memory usage, and suitability for a task are independent factors that should also be considered.

This is exactly why observability matters. Rather than assuming larger or newer models will perform better, you can use traces, metrics, and dashboards to measure how different models behave on your own hardware.

🎯 Conclusion

By combining aidatatools/ollama-benchmark, OpenTelemetry, and SigNoz, we've transformed a simple benchmark script into an observable benchmarking pipeline. Instead of relying on terminal output alone, every benchmark run is captured as traces and metrics that can be explored, compared, and analyzed over time.

The same approach can be applied to many existing CLI tools—adding observability without modifying their source code. Whether you're comparing lightweight 1.5B models or larger 8B models, OpenTelemetry provides the visibility needed to understand where time is spent and how performance changes across runs.

Every token leaves a trace.

🚀 What's Next?

Clone the project, run the benchmark against your favorite Ollama models, and explore the results in SigNoz. Extend the dashboard with your own metrics, compare different hardware configurations, or instrument additional benchmarking tools using the same pattern.

If you build on this project or discover interesting performance insights, I'd love to hear about your findings!

📚 References

Top comments (0)