The Payment Succeeded. The Governance Failed: Observing AI Agent Boundary Violations with OpenTelemetry and SigNoz

Faadil Boussari — Mon, 27 Jul 2026 03:36:29 +0000

AI agents are increasingly being trusted with actions that affect money, permissions, customer data, and operational systems.

But there is a dangerous assumption hidden inside many agent workflows:

If the action completed successfully, the agent behaved correctly.

That is not always true.

A payment can succeed technically while violating an approval policy. An API call can return 200 OK while crossing a governance boundary. A workflow can complete exactly as requested and still be unsafe.

I built Second Signature for the Agents of SigNoz hackathon to explore that distinction.

Second Signature is an observability system for high-risk actions performed by AI agents. It reconstructs the complete path from intent to policy evaluation, approval, payment execution, and ledger impact.

The core idea is simple:

Execution success is not governance success.

The scenario

The demo uses one controlled financial scenario:

Operation: refund
Amount: CAD 12,500
Beneficiary: Northstar Vendor
Beneficiary status: new
Reason: contract dispute

The policy is:

Transfers above CAD 5,000 to a new beneficiary require human approval.

The project compares three outcomes from the same request.

Outcome 1: Unsafe success

A legacy agent executes the payment without obtaining the required approval.

The payment succeeds, the ledger changes, and only afterward does the policy audit record that the action should have required approval.

Outcome 2: Safe stop

A guarded agent evaluates the policy, requests approval, receives a rejection, and blocks the payment.

The ledger remains unchanged.

Outcome 3: Safe success

A guarded agent evaluates the policy, requests approval, receives approval from FINANCE_LEAD, and only then executes the payment.

The same financial action becomes valid because the required second signature exists.

What Second Signature does

Second Signature separates three responsibilities clearly:

The application guard enforces the policy.
OpenTelemetry emits the evidence.
SigNoz observes, queries, audits, and alerts.

SigNoz does not approve or block the payment directly. The application guard performs enforcement.

SigNoz provides the observability layer that makes each decision path visible and auditable.

The project uses:

TypeScript and Node.js
The official OpenTelemetry Node.js SDK
OTLP over HTTP/protobuf
SigNoz Cloud
Custom traces and metrics
Trace-based alerting
Foundry configuration for reproducibility
A static Vercel deployment for the judge-facing demo

The ledger in the demo is controlled in-memory application state. It exists to prove whether the simulated financial side effect occurred. This is not a production banking integration and no real money is moved.

Designing telemetry around business meaning

A normal observability setup often focuses on technical signals:

latency
errors
throughput
CPU
memory
HTTP status codes

Those signals are important, but they do not answer the central question in this project:

Was the agent allowed to perform the action?

To answer that, I added domain-specific attributes to the traces.

Examples include:

agent.variant
policy.name
policy.mode
policy.matched
approval.required
approval.status
approval.approver
payment.executed
payment.amount
payment.currency
governance.outcome

This makes it possible to investigate both technical execution and governance correctness.

For example, an action can have:

payment.executed = true
approval.required = true
approval.status = missing
governance.outcome = unsafe_success

Technically, the payment completed.

From a governance perspective, it failed.

The three trace structures

Each scenario produces one root trace with nested spans that preserve the chain of causality.

Unsafe success
second_signature.run
└── agent.request
├── payment.transfer
└── policy.audit

The critical fact is the order:

payment.transfer
before
policy.audit

The legacy agent executes first. The policy audit records the violation afterward.

The relevant attributes include:

agent.variant = legacy
policy.matched = true
policy.mode = observe_only
approval.required = true
approval.status = missing
payment.executed = true
payment.amount = 12500
payment.currency = CAD
governance.outcome = unsafe_success

This is the most important failure mode in the project.

The agent appears successful because the refund completed, but the approval boundary was violated.

[Insert screenshot: unsafe trace waterfall showing payment.transfer before policy.audit]

Safe stop
second_signature.run
└── agent.intent.transfer
└── policy.evaluate
└── approval.request
└── payment.blocked

The guarded agent recognizes the policy before execution.

The approval is rejected, so the payment is blocked.

Relevant attributes include:

agent.variant = guarded
approval.required = true
approval.status = rejected
payment.executed = false
payment.amount = 0
governance.outcome = safe_stop

The ledger remains:

CAD 50,000

The key result is not an error. It is a safe operational stop.

[Insert screenshot: safe-stop trace ending in payment.blocked]

Safe success
second_signature.run
└── agent.intent.transfer
└── policy.evaluate
└── approval.request
└── approval.granted
└── payment.transfer

The guarded agent waits for approval.

The approval comes from:

FINANCE_LEAD

Only after approval.granted does payment.transfer occur.

Relevant attributes include:

agent.variant = guarded
approval.status = approved
approval.approver = FINANCE_LEAD
payment.executed = true
payment.amount = 12500
payment.currency = CAD
governance.outcome = safe_success

The ledger changes from:

CAD 50,000

to:

CAD 37,500

The payment still succeeds, but this time the governance path is valid.

[Insert screenshot: safe-success trace showing approval.granted before payment.transfer]

A custom governance metric

I also created a custom OpenTelemetry metric:

agent_governance_outcomes_total

It separates the three outcomes using labels such as:

outcome
agent_variant
policy_name

The resulting series distinguish:

unsafe_success
safe_stop
safe_success

and:

legacy
guarded

This metric provides a compact operational view of how often agent actions end in each governance state.

It is useful because a normal success counter would treat both unsafe_success and safe_success as successful executions.

The governance metric makes the difference explicit.

[Insert screenshot: agent_governance_outcomes_total in SigNoz]

Proving the guard invariant

A dashboard can look convincing while still hiding dangerous edge cases.

I wanted stronger evidence than a visual claim.

So I created a trace-matching invariant in SigNoz.

Query A finds guarded traces where approval was rejected:

service.name = "second-signature"
AND agent.variant = "guarded"
AND approval.status = "rejected"

Query B finds traces containing an executed payment:

service.name = "second-signature"
AND name = "payment.transfer"
AND payment.executed = true

The trace operator is:

A && B

The result was:

No results

This proves that no guarded trace with rejected approval also contained an executed payment.

The invariant is important because it validates the safety property across the complete trace, not just inside one individual span.

[Insert screenshot: SigNoz trace-matching query showing no results]

Detecting unsafe autonomous actions with a critical alert

I created a trace-based alert named:

Unsafe autonomous financial action detected

The alert filter is:

service.name = "second-signature"
AND name = "policy.audit"
AND governance.outcome = "unsafe_success"

The aggregation is:

count()

The condition is:

Above 0

The severity is:

Critical

When the unsafe legacy scenario was detected, the alert entered the:

Firing

state.

SigNoz also delivered a real email notification.

After the alert condition was no longer active, a later notification showed:

Resolved

This demonstrated the full lifecycle:

unsafe action detected
→ alert firing
→ email delivered
→ alert resolved

Reproducibility with SigNoz Foundry

The submission repository includes:

casting.yaml
casting.yaml.lock

The lockfile was genuinely generated using the official SigNoz Foundry CLI.

The verified command was:

foundryctl forge -f casting.yaml

This also generated the reproducibility output stored under:

pours/

For accuracy, I want to be explicit about what was and was not completed.

Verified:

foundryctl forge
casting.yaml.lock generation
Foundry deployment files generation

Not claimed:

foundryctl cast
complete local SigNoz deployment
local Foundry runtime validation

The telemetry used in the hackathon evidence was captured in SigNoz Cloud.

This distinction matters because reproducibility should be documented honestly, not implied.

What I learned

The most important lesson was that observability for agents cannot stop at technical success.

An agent can:

return the correct response
complete the API call
update the ledger
avoid throwing an exception

and still violate policy.

That means agent observability needs business and governance context.

Attributes such as:

approval.status
guard.enforced
payment.executed
policy.mode
governance.outcome

can be as important as latency or error rate.

I also learned several practical OpenTelemetry lessons:

preserve one root span per scenario
build correct parent-child relationships
instrument short-lived Node.js processes carefully
export metrics before process shutdown
separate application enforcement from observability
use trace-level queries for cross-span safety properties
design alerts around domain failures, not only technical exceptions

The hardest part was not building the UI.

It was proving that each outcome was real, queryable, and distinguishable inside SigNoz.

Final result

Second Signature demonstrates that one request can produce three very different governance outcomes:

Missing approval
→ Unsafe success

Rejected approval
→ Safe stop

Approved by FINANCE_LEAD
→ Safe success

The payment alone does not tell the full story.

The trace does.

Second Signature turns invisible agent boundary violations into evidence that operators can investigate, audit, and act on.

Live demo

https://second-signature.vercel.app

Source code and verified evidence

https://github.com/Faadil1/second-signature

Video demo

https://youtu.be/5OKJ5xvg0Bc

Built for Agents of SigNoz
Track 1: AI & Agent Observability

DEV Community: Faadil Boussari

The Payment Succeeded. The Governance Failed: Observing AI Agent Boundary Violations with OpenTelemetry and SigNoz