A production error is exactly when an AI agent should be least creative.
That sentence became the design rule behind Missio, a terminal mission control system I built for the Agents of SigNoz hackathon. Missio starts with a real incident reported in Slack, reconstructs what happened from SigNoz, finds the exact source revision that produced the failure, asks an agent for a tightly bounded change, and opens a draft pull request only after a human approves the action.
The entire journey stays visible in one Rust TUI:
SIGNAL → EVIDENCE → CODE → PATCH → HUMAN → PR → CI → HANDOFF
This is not a dashboard with an AI summary beside it. Every stage represents something the system has actually observed or completed. If evidence is missing, the mission stops. If GitHub cannot
resolve the production revision, it stops. If the model returns an unsafe patch, it stops.
Why Incident Agents Need Evidence, Not Confidence
Observability tools are already good at showing that a request failed. The harder question is what an agent should be allowed to do next.
An error message alone does not establish which code was deployed. A repository's default branch may have moved since the failure. A model can produce a convincing explanation using the wrong version of a file. It can also quietly fill missing context with assumptions.
I wanted the opposite behavior. Missio should be useful only when it can build a chain from a production signal to a specific piece of source code. The agent's job is to reason inside that chain, not invent one.
That made SigNoz the evidence system of record rather than a decorative integration.
Turning a Real Checkout Failure Into a Mission
For the demo, I use an instrumented checkout service. A bad discount request reaches a real HTTP endpoint and produces a failing OpenTelemetry span. The span is exported to SigNoz with the normal service and error fields, plus the details Missio needs to connect runtime behavior to source:
- The affected resource;
- The environment;
- The code file and function;
- The repository URL;
- The full Git commit running in production.
Then an operator opens the incident from an allowlisted Slack channel:
/missio-sev SEV1 service production/checkout checkout requests are failing
Slack is only the admission point. The message does not tell the agent what the root cause is, and Missio does not treat the wording as evidence. It uses the resource and incident window to query the corresponding telemetry in SigNoz.
The Indexing Delay That Changed the Design
The most useful lesson from building Missio did not come from reading documentation. It came from a demo that failed even though every individual component appeared healthy.
The checkout service successfully exported its span. SigNoz accepted the telemetry. But Missio queried immediately and received no matching rows. A short time later, the same query returned the failure exactly as expected.
Ingestion acceptance and query availability are not the same moment.
That distinction sounds small, but it changes the behavior of an autonomous workflow. If an empty first response means “there was no incident,” the agent can discard a genuine production failure.
If the system waits forever, the operator cannot tell whether it is making progress.
Missio now performs bounded evidence acquisition and makes the wait visible in the TUI. It retries for a limited period while SigNoz indexes the span. If a valid evidence bundle still cannot be formed, the EVIDENCE stage fails closed. It never replaces missing telemetry with a fabricated
incident just to keep the animation moving.
This was also why I kept the signal horizon at the bottom of the interface. The operator can see whether Missio is listening, acquiring evidence, or holding because the observation is incomplete.
Letting Telemetry Choose the Source
Once the trace is available, Missio extracts the repository and the full deployed commit from the span. That commit matters more than the current state of the default branch: it identifies the code that actually produced the observed failure.
Missio reads the repository tree at that immutable revision and selects source linked to the observed code location. In a monorepo, it can keep the investigation inside the affected service. Only that bounded context reaches the agent.
This gives the model a much more honest question:
Given this observed failure and the exact source that produced it, is there enough evidence for a small fix?
The model can either hold the mission or propose replacement content for source it was given. It cannot add an unrelated file, edit the CI workflow, reach outside the service boundary, or claim that an external action succeeded. Those decisions are enforced by Rust after the model responds.
Missio uses Rig for the agent runtime and an explicitly configured model through OpenRouter. The model never receives Slack, SigNoz, or GitHub credentials.
Making the TUI an Authority Boundary
I originally thought of the interface as the visual identity of the project: a mission-control orbit instead of another grid of cards and tables. While building the real workflow, it became something more useful.
The TUI shows who currently has authority.
SigNoz can provide evidence. The agent can propose a patch. Neither can create a pull request. When a proposed change passes policy, the mission pauses at HUMAN and shows the target repository. The operator must explicitly approve one draft-PR attempt.
That approval is deliberately narrow. It does not authorize a merge or deployment. If GitHub rejects the request, Missio preserves the mission and returns control to the human instead of silently repeating a mutation.
Ending at Handoff, Not Pretending Recovery
After approval, Missio creates a branch and draft pull request from the production revision identified by SigNoz. The evidence digest travels into the pull request, connecting the proposed change back to the incident that caused it.
Missio then watches GitHub CI for the exact draft commit. When the checks pass, the final state is HANDOFF.
That wording is intentional. Missio does not pretend that a green check means production has recovered. It tells the operator that an evidence-bound draft is ready for normal review, merge, and deployment. The pull request remains the place where engineers can inspect the change and make
the final call.
I considered extending the demo through automatic deployment, but it would have weakened the product's trust model. A hackathon animation should not manufacture authority that the real system
does not have.
What SigNoz Made Possible
Without SigNoz, this would be a code agent triggered by a Slack message. With SigNoz, the agent is grounded in an observed resource, time window, trace, code location, repository, and deployed revision.
That context affects every later step:
- It decides whether a mission exists;
- It decides which source can be read;
- It constrains what the agent can change;
- It binds the draft pull request back to production evidence.
The deepest lesson for me was that agent observability is not only about tracing what a model did. Observability can also control what the model is allowed to do next.
Missio still leaves the final operational decision with a person. Its job is to turn a noisy incident into a reviewable, evidence-bound change without hiding the gaps along the way.
You can explore Missio on GitHub.
If you want to trace the pieces I built on, the most useful references were the SigNoz guides for
Top comments (0)