We Traced a Fake AI Agent Before Building a Real One (Here's What Broke)

#signoz #wemakedevs #agents #hackathon

Before the real hackathon starts on the 20th, there's a small warm-up task. Self-host SigNoz, send it some data, and write about a feature you liked. We could have just run the demo app SigNoz gives you and finished in ten minutes. We didn't do that. Our hackathon track is about AI agent observability, so we wanted to actually try tracing something that looks like an AI agent before the real clock starts.

It took longer than we thought. Docker on Windows was the main reason.

Installing SigNoz was easy. WSL was not.

First thing to know: if you go looking for a docker-compose.yaml file in the SigNoz repo like older guides say, you won't find one. SigNoz now uses a new tool called Foundry. It handles the whole install for you.

curl -fsSL https://signoz.io/foundry.sh | bash
export PATH="$HOME/.local/bin:$PATH"

mkdir signoz-deploy && cd signoz-deploy
cat > casting.yaml << 'EOF'
apiVersion: v1alpha1
kind: Installation
metadata:
  name: signoz
spec:
  deployment:
    flavor: compose
    mode: docker
EOF

foundryctl cast -f casting.yaml

That last command does everything. It checks if Docker is working, sets up the config in the background, pulls the images, and starts all the containers — ClickHouse, ClickHouse Keeper, Postgres, the OTel collector, and SigNoz itself.

Here's where we lost about 40 minutes. My teammate, Sagar is on Windows using WSL2, and Docker just would not connect. Every docker command failed with "command not found," even though Docker Desktop was clearly open and running.

Turns out you have to manually turn on WSL Integration. It's not automatic. You go to Docker Desktop → Settings → Resources → WSL Integration, and there are two switches to turn on: one for "Enable integration with my default WSL distro," and another for the specific distro itself (Ubuntu, in our case). We only found the first switch at first, which is why nothing worked.

After fixing that, we ran docker --version again from Ubuntu, and it finally showed a real version number.

Then we hit a new error:

permission denied while trying to connect to the docker API at unix:///var/run/docker.sock

The fix was one line:

sudo usermod -aG docker $USER

We ran it, tried docker ps again, and it still failed. Turns out this change only works after you close the terminal and open a new one. We ran that same command three times before we figured that out.

After all that, the actual SigNoz install only took about three minutes. We opened localhost:8080, made an account, and landed on an empty dashboard that said "you're not sending any data yet."

We sent it a fake AI agent instead of the boring demo

SigNoz comes with a sample app called HotROD — a fake ride-hailing service. We skipped it. Not because it's bad, but because it wouldn't teach us anything about the thing this hackathon actually cares about: watching an AI agent do its work.

So we wrote a small Python script instead. It fakes an AI agent handling a request, making a fake LLM call, and doing a bit of post-processing. Every step is wrapped in an OpenTelemetry span so SigNoz can trace it.

from opentelemetry import trace
from opentelemetry.sdk.trace import TracerProvider
from opentelemetry.sdk.trace.export import BatchSpanProcessor
from opentelemetry.exporter.otlp.proto.grpc.trace_exporter import OTLPSpanExporter
from opentelemetry.sdk.resources import Resource

resource = Resource(attributes={"service.name": "demo-agent-service"})
provider = TracerProvider(resource=resource)
provider.add_span_processor(
    BatchSpanProcessor(OTLPSpanExporter(endpoint="localhost:4317", insecure=True))
)
trace.set_tracer_provider(provider)
tracer = trace.get_tracer(__name__)

def fake_llm_call(prompt):
    with tracer.start_as_current_span("llm_call") as span:
        span.set_attribute("llm.prompt", prompt)
        time.sleep(random.uniform(0.3, 1.2))
        span.set_attribute("llm.tokens", random.randint(50, 300))
        return "response"

def handle_request(user_query):
    with tracer.start_as_current_span("handle_request") as span:
        span.set_attribute("user.query", user_query)
        result = fake_llm_call(user_query)
        with tracer.start_as_current_span("post_process"):
            time.sleep(0.1)
        return result

Nothing complicated here. handle_request is the main span. llm_call fakes a model call with a random delay so the numbers look real. post_process doesn't do much, but we added it so we'd have three steps to look at in the trace.

We ran this script 20 times with random test questions like "What's the weather?" and "Book a flight." It sent data over gRPC to port 4317, which is the default port SigNoz listens on. When we checked the Traces tab in the browser, the data was already there. That felt good after the whole WSL mess.

The trace waterfall is the feature that got us

We looked at Dashboards and Alerts too, but clicking into a single trace and seeing the waterfall view is what actually made things click for us.

We picked one handle_request trace. It took 702ms total. Inside it, the llm_call step alone took 601ms. That's about 85% of the whole request just sitting inside the fake model call. The post_process step barely shows up next to it.

If you click on a span, you get a side panel showing every detail we added to it, like user.query: "Book a flight" and the token count. So you're not just seeing "this took 601ms." You're seeing exactly what the agent was doing during that time.

If you're trying to debug a real AI agent — one that calls an LLM, maybe checks a database, maybe uses a tool — this view is exactly what you'd want to open when someone says "the agent felt slow today." Instead of guessing, you can just look and see where the time actually went.

We also tried dashboards and alerts

Once tracing worked, we figured we'd try the rest too.

We built one dashboard panel showing the average llm_call duration over time. We used the Traces data source and filtered on name = 'llm_call'. It took a couple of tries because we first left the average function empty, and SigNoz told us "function avg requires arguments" until we typed avg(durationNano) correctly.

Then we set up one alert. If the p99 duration of llm_call goes above 1000ms, it should send a notification. One thing that confused us: durations are stored in nanoseconds, so typing "1000ms" actually means typing 1000000000 into a plain number box. There was no unit dropdown for this. We named the alert "Slow LLM Call," connected it to a test webhook channel, and saved it.

Was the setup pain worth it

Yes. The trace waterfall is the one feature we'd point someone to first. Seeing that 601 out of 702ms came from a single fake LLM call made the whole idea of "observability" make sense in a way that just reading the docs never did. It was also nice that the same filter, name = 'llm_call', worked the exact same way in the trace explorer, the dashboard, and the alert. We didn't have to rewrite it each time.

For the real hackathon project next week, the plan is simple: use the same setup, but replace time.sleep() with an actual LLM call, and see what the real numbers look like.