DEV Community

Sumit Roy
Sumit Roy

Posted on

📊 Adding Observability to Gemma 2B on Kubernetes with Prometheus & Grafana

In Article 1

When we first set up Prometheus + Grafana for Gemma 2B on Kubernetes, I expected to see nice dashboards with:

  • Tokens per request
  • Latency per inference
  • Number of inferences processed

…but all we got were boring container metrics: CPU%, memory usage, restarts.

Sure, they told us the pod was alive, but nothing about the model itself.
No clue if inference was slow, if requests were timing out, or how many tokens were processed.

🔍 Debugging the Metrics Problem

We checked:

  • Prometheus scraping the Ollama pod? ✅
  • Grafana dashboards connected? ✅
  • Metrics endpoint on Ollama? ❌

That’s when we realized:

  • Ollama by default doesn’t expose model-level metrics.
  • It only serves the API for inference, nothing else.
  • Prometheus was scraping… nothing useful.

💡 The Fix: Ollama Exporter as Sidecar

While digging through GitHub issues, we found a project: Ollama Exporter

It runs as a sidecar container inside the same pod as Ollama, talks to the Ollama API, and exposes real metrics at /metrics for Prometheus.

Basically:

[ Ollama Pod ]
    ├── Ollama Server (API → 11434)
    └── Ollama Exporter (Metrics → 11435)
Enter fullscreen mode Exit fullscreen mode

🛠 How We Integrated It

Here’s the snippet we added to the Ollama deployment:

- name: ollama-exporter
  image: ghcr.io/jmorganca/ollama-exporter:latest
  ports:
    - containerPort: 11435
  env:
    - name: OLLAMA_HOST
      value: "http://localhost:11434"
Enter fullscreen mode Exit fullscreen mode

And in Prometheus config:

scrape_configs:
  - job_name: 'ollama'
    static_configs:
      - targets: ['ollama-service:11435']
Enter fullscreen mode Exit fullscreen mode

📊 The Metrics We Finally Got

After adding the exporter, Grafana lit up with:

Metric Name         What It Shows
ollama_requests_total   Number of inference requests
ollama_latency_seconds  Latency per inference request
ollama_tokens_processed Tokens processed per inference
ollama_model_load_time  Time taken to load Gemma 2B model
Enter fullscreen mode Exit fullscreen mode

Suddenly, we had real model observability, not just pod health.

🚀 Lessons Learned

  • Default Kubernetes metrics ≠ Model metrics → You need a sidecar like Ollama Exporter.
  • One scrape job away → Prometheus won’t scrape what you don’t tell it to.
  • Metrics help tuning → We later used these metrics to set CPU/memory requests properl

🔮 What’s Next?

  • Now that we have model-level observability, the next step is:
  • Adding alerting rules for latency spikes or token errors.
  • Exporting historical metrics into long-term storage (e.g., Loki, Thanos).

Trying multiple models Gemma 3, LLaMA 3, Phi-3 and comparing inference latency across them.

💬 Let’s Connect

If you try this setup or improve it, I’d love to hear from you!

Drop a star ⭐ on the repo if it helped you — it keeps me motivated to write more experiments like this!

Top comments (0)