I set out to do something simple for the Agents of SigNoz hackathon: build a small RAG service, point OpenTelemetry at a self-hosted SigNoz, and see what the traces told me. The plan took an afternoon. Almost nothing went the way I assumed it would, and the most useful things I learned came from the trace data disagreeing with me.
The setup: SigNoz via Foundry, on a machine that had no Docker
SigNoz deprecated its old install.sh and the bundled compose files under deploy/ as of v0.130.0. The supported path now is Foundry, a CLI that provisions the stack from a small YAML file:
# casting.yaml
apiVersion: v1alpha1
kind: Installation
metadata:
name: signoz
spec:
deployment:
flavor: compose
mode: docker
curl -fsSL https://signoz.io/foundry.sh | bash
foundryctl cast -f casting.yaml
Worth flagging, because most of the tutorials I found while searching still tell you to clone the repo and run docker compose up from deploy/docker. Those files aren't maintained anymore.
My laptop is not a generous host: 7.8 GB of RAM total, and no Docker installed at all. The SigNoz docs also carry a warning I wasn't expecting. On Windows you're told to run Docker Engine inside WSL2 rather than Docker Desktop, because ClickHouse Keeper segfaults under Desktop's virtualization layer. So the real install was Ubuntu into WSL2 (under a minute), Docker Engine via get.docker.com (about 80 seconds, zero drama), then Foundry.
One thing bit me before I got that far. WSL2 defaults to allocating half your RAM, which on my machine came out to 3.7 GiB, just under SigNoz's 4 GB minimum. A .wslconfig with memory=5GB fixed it. If your host has 8 GB, budget for this.
The cast itself was the easy part. It pulled ClickHouse (1.13 GB), the OTel collector, Postgres and Keeper, and had the stack up in about four minutes with no errors. I want to be fair to SigNoz here: the install was the smoothest step of the day.
The restart loop that wasn't
Then every docker ps showed all containers as "Up 2 seconds." Every single time. That is what a crash loop looks like, so I went off chasing memory pressure that didn't exist.
The actual cause had nothing to do with SigNoz. WSL2 terminates the distro a few seconds after your last command exits. Each time I checked, I was cold-booting systemd, dockerd and the entire SigNoz stack, which then honestly reported "Up 2 seconds," because it had been up for two seconds. The journal gave it away: a fresh Starting docker.service... with a new PID on every single check.
The fix is slightly embarrassing. Keep any process alive in the distro:
# leave this running in a spare terminal, it matters
wsl -d Ubuntu -- sleep infinity
Without it, ClickHouse reboots forever and nothing ever reaches healthy.
SigNoz will not ingest a single span until you create your account
With the stack finally healthy, my app's exporter started failing:
Failed to export traces to localhost:4317, error code: StatusCode.UNAVAILABLE,
error details: ... UNAVAILABLE: ipv6:[::1]:4317: End of TCP stream
The port was reachable. TCP would connect, then the stream just ended. I assumed a WSL2 networking quirk and switched the exporter to 127.0.0.1. Same error on IPv4. ClickHouse's trace table: zero rows.
The answer was sitting in the SigNoz server logs the whole time, repeating every 30 seconds. This is why you read the logs:
"failed to find or create agent" ... "cannot create agent without orgId"
SigNoz's collector pulls its pipeline config from the server over OpAMP, and the server won't register the collector until an organization exists. The organization gets created when you fill in the admin signup form on first load. Until you do that, the OTLP receiver never starts, and from the outside it is indistinguishable from a broken network.
Seconds after I created the account, the collector logged Starting GRPC server [::]:4317 ... Everything is ready and spans started flowing. Thirty minutes of debugging, resolved by a signup page.
The service: 15 chunks, three spans
The app is deliberately tiny. FastAPI, ChromaDB with 15 hardcoded chunks about coffee brewing, local MiniLM ONNX embeddings, and Groq's llama-3.3-70b-versatile for answers. Auto-instrumentation for FastAPI and httpx, plus three manual spans around the pipeline stages:
with tracer.start_as_current_span("embed_query") as span:
span.set_attribute("embedding.model", "all-MiniLM-L6-v2")
query_embedding = embed_fn([req.question])[0]
with tracer.start_as_current_span("vector_search") as span:
span.set_attribute("top_k", req.top_k)
results = collection.query(query_embeddings=[query_embedding], n_results=req.top_k)
span.set_attribute("chunks_retrieved", len(results["documents"][0]))
with tracer.start_as_current_span("llm_call") as span:
span.set_attribute("gen_ai.request.model", GROQ_MODEL)
# ... Groq call, prompt_tokens recorded from the response
I fired 45 requests at it: 35 normal, 5 with top_k=12 to stress retrieval, and 5 rigged to fail with a bad API key or an injected exception.
What the traces actually said
The latency split was not where I expected. Averaged over successful requests, POST /ask took roughly 1.54 s server-side. llm_call was 878 ms (57%), embed_query was 648 ms (42%), and vector_search was 3 ms (0.2%).
I had assumed the LLM would swallow the whole request and everything local would be rounding error. Embedding one short sentence with a "free" local model cost nearly as much as a full round trip to a 70B model on Groq's hardware. On a memory-starved laptop, local inference is the second biggest line item in the request, and I would not have known that without the span breakdown.
The single trace above is a good example of the shape: 1.62 s total, llm_call 938.42 ms, embed_query 673.48 ms. Two bars, nearly the same height, and only one of them is the model I was paying attention to.
The client and the server disagreed by 2.6 seconds, and both were right. My load script measured p50 at 4,214 ms. SigNoz showed the server span at p50 1,590 ms.
That gap lives outside the traced service, which is the useful part: the trace draws a border around what my code is responsible for. A quick A/B nailed it. The same request took about 4.3 s via http://localhost:8001 and about 1.9 s via http://127.0.0.1:8001. Windows resolves localhost to IPv6 ::1 first, uvicorn was listening only on IPv4, and every request paid a ~2.4 s fallback tax before a single byte reached my code. No server-side metric would ever have surfaced that.
The errors looked like the failures they were. The bad-API-key traces show an errored llm_call span of about 587 ms, a real round trip that came back 401 Unauthorized from api.groq.com, with the message preserved verbatim in the span status. The injected exceptions error out in 2 ms, dying instantly instead of after a network hop. You can tell the two failure modes apart from their shape alone, before reading anything.
One detail I only noticed while scrolling that error list: the root POST /ask span for an injected-exception trace is 678.54 ms even though its llm_call child is 2.74 ms. The request fails almost instantly at the LLM step and still pays the full local embedding cost first. Fail-fast at the end of a pipeline isn't fast.
And an honest null result. Cranking top_k from 3 to 12 did nothing measurable. Vector search went from 3.3 ms to 3.6 ms, and tripling the prompt (about 115 to 350 tokens) didn't move Groq's latency in any way I could distinguish from noise. A 15-document corpus can't make retrieval expensive. I'm reporting that rather than pretending I found something.
The 11.11% error rate on that overview is just my 5 rigged failures out of 45 calls, which was a small relief to see land where it should.
What I'd tell myself before starting
The trace waterfall's value on day one wasn't finding a slow function. It was accounting for 99% of server time, which meant the missing 2.6 s could only be client-side. Subtraction turned out to be the useful operation.
Instrument before you optimize your assumptions. I would have gone straight at the LLM call. The data pointed at a local embedding model and a hostname.
Self-hosting observability on a small machine works, but it's unforgiving about setup order. Know your WSL2 memory default, keep the distro alive, and create the admin account before you start debugging ingestion.
One accidental lesson too: an early ChromaDB bug of mine dumped an entire 384-dimension embedding vector into a span's status message. Worth watching what your error paths put into telemetry.
One afternoon, no cloud spend, and two of the three bugs I found were in my environment rather than my code.
Setup used: SigNoz self-hosted via Foundry on Docker in WSL2, FastAPI + ChromaDB + Groq, OpenTelemetry Python SDK. Install docs: signoz.io/docs/install/docker. OpenTelemetry Python: opentelemetry.io.
Top comments (0)