DEV Community: Dani Muhammad

I built an AI that refuses to send my application

Dani Muhammad — Mon, 31 Aug 2026 21:44:51 +0000

In July I recorded myself saying my English is not good enough, and that I feel like I need a script when I try to present. I was talking to myself, not to an audience. Three weeks later this hackathon opened, and I kept coming back to that recording.

I'll be honest, my first idea was to build something that writes better English than me. Then I looked at what actually goes wrong when I apply for things, and it was never the English. I have closed deals in English. I have ten full application kits sitting on my laptop from this year alone, CVs, cover letters, answer sheets. The grammar was never the problem. Grammarly solved grammar a decade ago.

The real problem is that I cannot tell what one specific institution is going to reject me for. And no writing assistant knows that either, because none of them have read the call document.

So I stopped trying to write better than myself. I built something that knows the rules.

What it actually does

Berkas is Indonesian for the file, the dossier, the papers you hand in. You give it the call for applications, a PDF or just a photo of the page. It reads it and tells you what it requires: the deadline, the word caps, the sections you cannot skip, how formal the writing has to be.

Then it stops and makes you correct it.

That pause is the whole thing. The model is allowed to be wrong about what it read. What it is not allowed to be is wrong and final at the same time. So the requirements come back as fields you can edit, nothing gets written until you have been through them, and if you change a word cap from 500 to 150 that gets saved next to what the model originally said, so later anyone can see which of us decided what.

After that it interviews you, but only about the things your own files do not already answer. Ten kits sitting on my disk, and it does not make me retype any of them. Then it drafts against the spec, in a writing voice I documented months ago for a completely different reason.

And then it checks the draft, and quite often it refuses to send it.

The part I actually care about

There is one line I wrote before any of the code:

It never invents a claim about your experience. Every sentence traces to a file you wrote or an answer you gave.

That is easy to put in a README and hard to make true. The way it is enforced is small and kind of stupid, and somehow that is the part that works. The drafting agent sees my corpus and my interview answers and nothing else. When it needs a fact it does not have, it is told to write a marker instead of a sentence:

[NEEDS: the specific thing it would need to finish this sentence]

And the checker treats that marker as a hard violation. Not a warning. A packet with one in it cannot be submitted.

The first time I ran the whole thing end to end I did not answer any of the interview questions. I just wanted to see if the pipes connected. It drafted four sections and then blocked itself three times, because it would not invent a host university, a household income, or a community project it had no evidence for. It wrote down exactly what it would need instead. I did not stage that. It was the first real run.

Why the checker has no AI in it

The checker is plain Python. Standard library only. No model call anywhere in that path.

This was the main decision and everything else follows from it. The model perceives. Code decides.

If you ask a language model whether a draft is under 500 words, you are asking something that cannot count to do the counting, and then trusting the answer. Worse than that, a good essay can talk its way past a hard rule, because that is what fluent writing does to whoever is reading it. So the model reports what the document says, and 263 lines of Python decide whether you are allowed to submit. Same draft, same verdict, every time.

It is also the only part of the system I could write real tests for, and I did not expect that to matter as much as it did.

Two bugs that only existed in production

Neither of these could fail on my laptop.

The first one: gcloud run deploy --source . falls back to reading .gitignore when there is no .gcloudignore. My evidence corpus is gitignored on purpose, because it is my personal career history and I am not publishing it. So the service deployed cleanly, reported healthy, and quietly drafted from nothing at all. No error, no crash, just a system that had forgotten who it was writing for. I only caught it because the health endpoint reports how many corpus files it can see. A health check that says ok while the product is empty is not a health check.

The second one: I called Secret Manager with access_secret_version(name={"name": ...}), which passes a dictionary where a string belongs. That branch only runs on Cloud Run, because locally the token is read off disk. So it passed every test I had, deployed fine, cleared both gates, and then threw a protobuf error on the single request this whole project exists to make.

Both of them are written down in the README now, because they cost me time and they will cost someone else time.

What I would tell myself at the start

A gate is only worth what you tested it at. "The button is disabled" is a claim about a browser. "POST /api/send returns 409" is a claim about a system. Those are not the same sentence, and only one of them survives someone with curl.

And the hard part was never the writing. It was deciding what the thing is not allowed to do, and then making that decision expensive to undo.

I built this for the All Things Agentic hackathon, on Google ADK, Gemini 3.7 Flash and Gemma 4 through Vertex AI, Cloud Run, Firestore and the Gmail API. But the reason I built it is smaller than all of that. I wanted one thing in my applications that I would not have to be afraid of.

What's one thing you have sent that you wish something had checked first?

I created this piece of content for the purposes of entering the All Things Agentic hackathon. Repo · Live · Demo video

AllThingsAgenticHackathon

I Put SigNoz on My AI Agent to See Why It Was Slow. Then I Found Out What It Was Saving About My Users.

Dani Muhammad — Sun, 19 Jul 2026 13:45:51 +0000

I wasn't hunting for a privacy problem. I just wanted to see how slow my AI agent really was.

For the "Agents of SigNoz" hackathon I built a small tool-using agent, wired it up to SigNoz with OpenTelemetry, and started poking at what got recorded every time it ran. Somewhere in the middle of that I found myself staring at a database row with the exact text a user had typed to my agent, sitting there in plaintext, forever. This post is what I built, what actually caught me off guard, and what I did after I saw that row.

Where I'm coming from

I'm not a devops person. During my internship I built factory monitoring dashboards, MQTT sensors going into Node-RED going into Grafana, watching machine temperatures and status codes. So I already knew what a metrics dashboard looks like. What I didn't know was distributed tracing, or what "observability" even means for something that isn't a sensor, like an AI agent making LLM calls.

I made myself one rule: I wouldn't write about anything I hadn't actually run. So everything below, I ran it myself.

The agent is small. It's OpenAI-powered with four tools: a weather lookup, a calculator, a fake "web search" that sleeps for 2.5 seconds on purpose, and a fake "API call" that throws an error on purpose. I instrumented it by hand with OpenTelemetry (traces, logs, metrics) and pointed it at a SigNoz instance I self-hosted locally with Docker.

Getting it running

One thing worth knowing first: SigNoz recently dropped its old docker-compose install in favor of a new CLI called Foundry (foundryctl). If you find an older tutorial it won't match what you actually get today, which cost me some confusion. I was on SigNoz v0.133.0 running ClickHouse 25.12.5.44.

The only real snag before I even reached SigNoz was Python. Homebrew's python@3.14 build ships without a working ssl module, so pip couldn't even reach PyPI ("Can't connect to HTTPS URL because the SSL module is not available"). I gave up on it and built my virtualenv against system Python (3.9.6) instead, which just worked. Small thing, but it's the kind of thing you only hit by actually sitting down and doing it.

The thing that changed how I think about "slow"

Once traces were flowing I opened the waterfall view for one of my agent's runs for the first time. That view takes a single request and breaks it into every step it took, laid out left to right by how long each step lasted. It was new to me. My old Grafana dashboards would tell me a number crossed a threshold. This told me which exact step in which exact request was the reason.

And it showed me something I wasn't expecting. In a normal run, my own tool code (the calculator, the weather lookup) ran in under a tenth of a millisecond. Every single OpenAI call took somewhere between 1200 and 3700 milliseconds. For an AI agent the model calls basically are the latency, and my own code is close to free by comparison. I would never have gone looking for that number on my own. The waterfall just put it in front of me.

Breaking it on purpose

I broke the agent two different ways to see how each one shows up in SigNoz.

First, a bad API key. That crashed the whole request, and both the failing LLM call and the parent trace came back marked as errored. Second, I made one tool throw an exception internally but caught it in my own code so the agent could recover and keep going. That time only the one tool's span went red. The overall run wasn't marked as failed, because it genuinely wasn't, the agent handled it and told the user something went wrong. So the shape of the trace told me which kind of failure I was looking at before I read a single line of text.

That second failure is also what I used to test alerts for real. I set up a Slack webhook, added it to SigNoz as a notification channel, and built a trace-based alert on "more than zero error spans in a rolling 5-minute window." Then I ran my broken tool six times in a row on purpose and watched the alert flip to Firing, with a real message landing in Slack a few seconds later. I didn't just configure it and trust that it works. I forced it to go off.

The dashboard side was messier, and honestly that's the part I most wanted to write about. My first panel, a P90 latency chart grouped by tool name, failed with a generic "something went wrong" error in the UI. Instead of giving up I went into SigNoz's own backend container logs and found the real cause: the query builder had generated a ClickHouse SQL query calling a function named histogramQuantile, which is a PromQL function, not a ClickHouse one. It doesn't exist in that engine. That's a real, reproducible bug in this version, not me holding it wrong. Switching the aggregation from a percentile to a plain average got around it.

Then I hit a second, dumber problem. The chart said "No Data" because ClickHouse stores timestamps in UTC and I was checking against my local time, eight hours off, and on top of that I hadn't re-triggered one of my tools since adding metrics. Once I fixed both, the chart was genuinely satisfying: one sharp 2.6-second spike for my slow tool, flat near-zero for everything else. Same insight the waterfall gave me, now as a time-series chart. It just took two real bugs to get there.

What I actually found

Here's the part I didn't see coming.

I queried what SigNoz had actually stored about one of my agent's runs, expecting spans, timings, maybe some metadata. Instead I found the complete, word-for-word text of a user's request and the model's full reply, sitting in plaintext in the trace data:

gen_ai.prompt.0.content: "Fetch live data by calling the /orders API endpoint."
gen_ai.completion.0.content: "It seems that the /orders API endpoint is currently
unavailable, and I couldn't fetch the required live data. If there's anything
else you need or if you'd like to try again later, please let me know!"

This isn't a SigNoz problem exactly. SigNoz just stores whatever spans it receives. The instrumentation library I was using (opentelemetry-instrumentation-openai, part of the OpenLLMetry ecosystem) is the one deciding to capture that content. But the result is the same either way. In a real product that's every user's prompt, and whatever they put inside it, sitting in your observability backend by default, with nothing warning you it's happening.

I didn't want to just claim that and move on, so I checked it two ways.

First I opened the actual installed package source and found the switch: a function called should_send_prompts(), gated by an environment variable TRACELOOP_TRACE_CONTENT that defaults to "true" if you never set it. Then I checked it against the official OpenTelemetry GenAI semantic conventions spec, not the library's own docs but the actual OTel spec. It explicitly calls out gen_ai.input.messages and gen_ai.output.messages as "likely to contain sensitive information including user/PII data," and recommends that capturing them should be opt-in. So this isn't only my opinion. The people who wrote the spec already flagged this exact risk. The library just still defaults the other way.

Then I tested the fix, not only the setting. I set TRACELOOP_TRACE_CONTENT=false, ran the same kind of request again, and looked at the new span. The content fields were completely gone. The token counts (input, output, total) were all still there, accurate, untouched. So you get to keep the number you actually need for cost and debugging, and drop the part that's a liability.

Fair is fair, though. ClickHouse and Postgres, where all this data actually lives, had zero ports exposed to my host machine in this setup, locked down correctly by default. The OTLP port that receives telemetry was the opposite, open with no authentication in front of it. Neither of those is a reason to panic. They're both just things worth knowing before you point any of this at something real.

What I'd tell myself starting over

The biggest shift for me wasn't a SigNoz feature, it was the mental model. Coming from dashboards built for physical sensors, I already understood metrics. I didn't understand tracing, and tracing turned out to be the thing that actually answers "why was this slow" for anything that isn't a sensor.

The second one is more of a warning than a lesson. If you bolt observability onto an AI feature just to debug it, go check what your instrumentation captures by default. "Observability" and "a permanent record of what your users typed" can quietly be the same thing, and nothing is going to warn you.

And two small practical ones: check your timezones before you assume "no data" means something is broken, and check the actual backend logs before you assume a UI error is your fault.

Where this leaves me

I only wrote about things I actually ran, so let me be clear about what that covers: traces, logs, metrics, a working alert, a dashboard panel, and one real, tested privacy behavior with a fix I verified myself. I didn't load-test any of this, and I haven't touched SigNoz Cloud. So take it as what happened on my one self-hosted setup, not a universal claim.

If you're building with LLMs and adding observability, it's worth five minutes to go check what your traces are actually holding onto. Mine had a lot more in them than I expected.