My AI Chatbot's Latency Was 99.8% One Thing — and I Only Found Out Because I Traced It

Abhinav Khedwal — Sat, 18 Jul 2026 18:44:53 +0000

I built EcoIQ, a carbon footprint tracker with a Groq-powered AI assistant, for a different hackathon a while back. It worked, it shipped, and the chat replies always felt a little slow — somewhere in the 1-1.5 second range. I assumed it was "just how AI apps are." I never actually knew where that time went.

This weekend I wired the backend up to a self-hosted SigNoz instance to find out. The answer surprised me: 99.80% of a 1.45-second chat response was spent waiting on Groq's API. Not my validation logic, not my sanitization code, not JSON parsing — one outbound HTTPS call, sitting there, accounting for almost the entire trace.

Here's the full story: the setup pain, the code, what the traces actually showed, and what I'd tell someone trying this for the first time.

The problem, before I could see it

EcoIQ's backend is a plain Node.js http server — no Express — with one route that matters for this story: POST /api/chat. It validates incoming messages, sanitizes them, checks a basic per-IP rate limit, and then proxies the request to Groq's chat completions endpoint:

jsconst groqRes = await fetch('https://api.groq.com/openai/v1/chat/completions', {
method: 'POST',
headers: {
Authorization: Bearer ${process.env.GROQ_API_KEY},
'Content-Type': 'application/json',
},
body: JSON.stringify({
model: process.env.GROQ_MODEL || 'llama-3.3-70b-versatile',
messages: validation.sanitized,
temperature: 0.7,
max_tokens: 800,
stream: false,
}),
});

Before this weekend, if you'd asked me "why does this take 1.4 seconds," I'd have guessed. Maybe my validation loop was heavier than I thought. Maybe the rate-limiter's Map lookups added up. Maybe it was my own network. I had no actual evidence either way — just a route that felt slow and no way to break that feeling into numbers.

Setting up SigNoz: further from docker compose up than I expected

I went in expecting the classic install: clone the repo, docker compose up -d, done. That's not how it works anymore. SigNoz recently retired the bundled docker-compose files in favor of a new CLI called Foundry — I only found out because I hit a deprecation notice sitting in the repo's own deploy/README.md, after cd-ing into a folder that no longer had a compose file in it.

The real friction came from Windows, though. Docker Desktop is the obvious default there, but SigNoz's own docs quietly warn that ClickHouse Keeper crashes under Docker Desktop's virtualization layer on Windows — segfaults (exit code 139) and restart loops. The reason, as best I can tell from their docs, comes down to how Docker Desktop's virtualization layer handles file system syncing between the Windows host and its utility VM — ClickHouse Keeper's disk access patterns don't play well with that translation layer. Their fix is to skip Docker Desktop entirely for this and run native Docker Engine inside WSL2 instead.

So the real path looked like this:

Confirm WSL2 is installed: wsl --version
Install a real Ubuntu distro — not the hidden docker-desktop distro WSL drops you into by default, which has no package manager and no sudo:

wsl --install -d Ubuntu

Inside that Ubuntu shell, install Docker Engine natively:

bash curl -fsSL https://get.docker.com | sh
sudo usermod -aG docker $USER

Install Foundry, SigNoz's new installer CLI:

bash curl -fsSL https://signoz.io/foundry.sh | bash

Write a minimal casting.yaml:

yaml apiVersion: v1alpha1
kind: Installation
metadata:
name: signoz
spec:
deployment:
flavor: compose
mode: docker

Deploy:

bash foundryctl cast -f casting.yaml

That last command pulled five images — ClickHouse, ClickHouse Keeper, Postgres, the SigNoz backend, and the OTel collector — and started everything healthy on the first real attempt, but only after the WSL detour. If you're on Windows and want to avoid the crash loop entirely, start in WSL2 from minute one instead of finding out the hard way.

A couple of minutes later, docker ps showed all five containers Up (healthy), and localhost:8080 gave me a live SigNoz workspace.

Instrumenting the server without touching the route

Adding OpenTelemetry took four packages and one small file — zero changes to server.cjs itself:

bashnpm install --save-dev @opentelemetry/api @opentelemetry/sdk-node \
@opentelemetry/auto-instrumentations-node @opentelemetry/exporter-trace-otlp-http

js// tracing.cjs
const { NodeSDK } = require('@opentelemetry/sdk-node');
const { getNodeAutoInstrumentations } = require('@opentelemetry/auto-instrumentations-node');
const { OTLPTraceExporter } = require('@opentelemetry/exporter-trace-otlp-http');
const { resourceFromAttributes } = require('@opentelemetry/resources');

const sdk = new NodeSDK({
resource: resourceFromAttributes({ 'service.name': 'ecoiq-server' }),
traceExporter: new OTLPTraceExporter({ url: 'http://localhost:4318/v1/traces' }),
instrumentations: [getNodeAutoInstrumentations()],
});

sdk.start();

Then I preloaded it when starting the server instead of editing the entry point:

bashnode -r ./tracing.cjs server.cjs

That's the part that actually impressed me. getNodeAutoInstrumentations() recognizes Node's built-in http/https clients automatically — including the native fetch call inside handleChatAPI. I expected I'd need to hand-write a span around the Groq call to see anything about it. I didn't. It showed up on its own, tagged with the exact host, port, and full request URL.

What the traces actually showed

I opened the app, clicked through the calculator, and sent the AI assistant a message. Then I opened SigNoz's Traces Explorer.

Show Image

The list view alone told a story: plain page loads (GET requests for static HTML/CSS) landed in the 20-40ms range. Every POST to /api/chat landed between 1.2 and 1.45 seconds — a 30-40x gap that had nothing to do with my code and everything to do with the fact that one route calls out to an LLM and the other doesn't.

Clicking into one of those POST traces (1.45s total) opened the waterfall view:

Show Image

The root span is POST /api/chat at 1.45s. Directly underneath it, stretching across almost the entire width of the flame graph, is a nested client span to api.groq.com — also 1.45s, marked as 99.80% of total execution time. My validation, sanitization, and rate-limit bookkeeping combined didn't register as a visible sliver on the graph. The span details panel showed exactly what got captured automatically: http.request.method: POST, http.response.status_code: 200, server.address: api.groq.com, server.port: 443, and the full url.full: https://api.groq.com/openai/v1/chat/completions — twelve attributes total, none of which I wrote a line of code for.

I also checked the Service view (the auto-generated overview page SigNoz builds per service once it starts receiving data). Without any dashboard configuration on my part, it had already plotted the standard RED metrics — request rate, error rate, and duration percentiles — for ecoiq-server, splitting /api/chat out from the static-file routes by latency. Seeing the P99 duration line sitting close to 1.5s on that graph, next to a flat near-zero line for static assets, confirmed the same story from a different angle: this service has exactly one slow path, and it's not a mystery anymore.

That's a genuinely useful thing to know if I ever want EcoIQ to feel faster. There's no point micro-optimizing validateMessages or the rate-limiter — the numbers say those cost microseconds, not milliseconds. The only lever that matters is the model call itself: a smaller/faster model, stream: true instead of waiting for the full completion before responding, or just an honest loading state that manages expectations instead of chasing a speedup that isn't there to find.

What I'd tell past-me

On Windows, start in WSL2, not Docker Desktop, for this stack specifically. The docs will tell you, but only if you read the fine print before you hit the crash yourself.
Auto-instrumentation is not a toy. I assumed I'd need custom spans to see anything about an outbound fetch call. I didn't — the HTTP client instrumentation caught it automatically, full URL and all.
A feeling ("this route is slow") is not a diagnosis. I'd had that feeling about /api/chat for weeks. It took one afternoon of tracing to turn it into an actual number I could act on, instead of a guess I kept deferring.
Check the service overview, not just individual traces. The RED metrics view gave me the same conclusion as the waterfall, but as a pattern across many requests instead of one anecdote — worth doing both.

Try it yourself

If you're running any Node service that calls out to an LLM API and have a vague sense it's "slow," this setup takes under an hour end to end — most of that being the WSL detour if you're on Windows. The payoff is a concrete, attributed answer to "where does my latency actually go," instead of a guess you never quite get around to checking.

SigNoz self-host Docker guide
OpenTelemetry Node.js instrumentation
EcoIQ live demo

DEV Community: Abhinav Khedwal

My AI Chatbot's Latency Was 99.8% One Thing — and I Only Found Out Because I Traced It