DEV Community: KOMARI Subheeksh

Your Midnight deploy will crash on a fresh install - here are the wallet-sdk patches (and why they happen)

KOMARI Subheeksh — Sun, 09 Aug 2026 12:45:44 +0000

Every Midnight developer eventually hits this. You clone a working project, run npm install, and the deploy script that worked yesterday crashes with something like:

TypeError: state.pendingOutputs.values.map is not a function or its return value is not iterable
  at Object.pickAllCoins (CoreWallet.js:28:66)

Or your wallet sync hangs for minutes and dies with:

Wallet.Sync: [object Object]
  at file:///.../wallet-sdk-shielded/dist/v1/Sync.js:126:169

The first time this happened to me I assumed it was a Node version issue. It is not. It's a bug in the Midnight wallet SDK itself — a family of bugs, actually, spread across five packages — and the fix is a set of patches you have to re-apply after every npm install.

The root cause: plain iterators without Iterator helpers

The wallet SDK (version 1.2.0 core, shielded 3.0.2) defines its own ledger types — maps of transaction intents, outputs, and coin nonces. Those custom types implement .entries(), .values(), and .keys() methods that return plain iterators.

Here's the problem. Modern JavaScript's Iterator protocol has helper methods: .map(), .filter(), .find(), .every(), .toArray(). Native Map and Set objects get these helpers from Iterator.prototype — and in Node 22+, you can do:

new Map().values().map(x => x)  // works on native Map

But the SDK's custom ledger types return iterators that do NOT inherit from Iterator.prototype. They're plain objects that only implement the next() method. So:

state.pendingOutputs.values().map(...)
// TypeError: state.pendingOutputs.values.map is not a function

This is not a Node version problem. It fails on every Node version, because the SDK's custom iterators never had the helper methods to begin with. The SDK code was written against an environment where these helpers existed (or the code was never run against the custom types), and it shipped assuming .map() would be there.

The patch: spread into a real array first

The fix for each crash site is mechanical: convert the plain iterator into a real array (which has all the methods) before chaining:

- state.pendingOutputs.values().map(...)
+ [...state.pendingOutputs.values()].map(...)

Here are the affected files and the exact patches, verified against wallet-sdk-shielded 3.0.2 / unshielded 3.1.0 / facade 4.1.0 / dust-wallet 4.2.0.

1. `wallet-sdk-shielded/dist/v1/CoreWallet.js` — `.values().map()`

The crash from the top of this post. This one hits first because pickAllCoins runs during coin selection, before any transaction is built.

sed -i 's/state.pendingOutputs.values().map/([...state.pendingOutputs.values()]).map/' \
  node_modules/@midnight-ntwrk/wallet-sdk-shielded/dist/v1/CoreWallet.js

2. `wallet-sdk-shielded/dist/v1/TransactionImbalances.js` — `.entries().every()` (x2)

sed -i 's/imbalances.guaranteed.entries().every/[...imbalances.guaranteed.entries()].every/' \
  node_modules/@midnight-ntwrk/wallet-sdk-shielded/dist/v1/TransactionImbalances.js
sed -i 's/segmentImbalances.entries().every/[...segmentImbalances.entries()].every/' \
  node_modules/@midnight-ntwrk/wallet-sdk-shielded/dist/v1/TransactionImbalances.js

3. `wallet-sdk-shielded/dist/v1/TransactionOps.js` — `.entries().filter()` (x2)

These call sites span multiple lines, so a plain sed won't do. The chain looks like:

tx
  .imbalances(0)
  .entries()
  .filter(...)

Patch with a small Node script:

const fs = require('fs');
let f = 'node_modules/@midnight-ntwrk/wallet-sdk-shielded/dist/v1/TransactionOps.js';
let c = fs.readFileSync(f, 'utf8');
c = c.replace(/tx\n(\s+)\.imbalances\(0\)\n\1\.entries\(\)\n\1\.filter\(/g,
  '[...tx\n$1.imbalances(0)\n$1.entries()]\n$1.filter(');
c = c.replace(/tx\n(\s+)\.imbalances\(segment\)\n\1\.entries\(\)\n\1\.filter\(/g,
  '[...tx\n$1.imbalances(segment)\n$1.entries()]\n$1.filter(');
fs.writeFileSync(f, c);

4. `wallet-sdk-unshielded-wallet/dist/v1/TransactionOps.js` — `.keys().toArray()` + `.entries().filter().map().toArray()`

sed -i 's/transaction.intents?.keys().toArray()/Array.from(transaction.intents?.keys() ?? [])/' \
  node_modules/@midnight-ntwrk/wallet-sdk-unshielded-wallet/dist/v1/TransactionOps.js

The .entries().filter().map().toArray() chain is multi-line and also ends with .toArray() that has no replacement — the cleanest approach is a Python one-liner:

f = 'node_modules/@midnight-ntwrk/wallet-sdk-unshielded-wallet/dist/v1/TransactionOps.js'
c = open(f).read()
c = c.replace(
    'const imbalances = transaction\n            .imbalances(segment)\n            .entries()\n            .filter',
    'const imbalances = [...transaction\n            .imbalances(segment)\n            .entries()]\n            .filter'
)
c = c.replace('.toArray();\n        return Imbalances.fromEntries', ';\n        return Imbalances.fromEntries')
open(f, 'w').write(c)

5. `wallet-sdk-facade/dist/transaction.js` — `.values().toArray()` (x2)

sed -i 's/tx.intents?.values().toArray()/[...(tx.intents?.values() ?? [])]/g' \
  node_modules/@midnight-ntwrk/wallet-sdk-facade/dist/transaction.js

6. `wallet-sdk-dust-wallet/dist/v1/Transacting.js` — `.entries().find()`

const fs = require('fs');
let f = 'node_modules/@midnight-ntwrk/wallet-sdk-dust-wallet/dist/v1/Transacting.js';
let c = fs.readFileSync(f, 'utf8');
c = c.replace(
  /const \[_, imbalance\] = transaction\n(\s+)\.imbalances\(0, totalFee\)\n\1\.entries\(\)\n\1\.find\(/,
  'const [_, imbalance] = [...transaction\n$1.imbalances(0, totalFee)\n$1.entries()]\n$1.find('
);
fs.writeFileSync(f, c);

Bonus bug: `Set.difference()` doesn't exist on older Node

While patching CoreWallet.js, you'll find a second landmine right next to the first:

coinNonces.difference(definedNonces)

Set.prototype.difference() is ES2025 — it exists on Node 22.13+ and 24, but NOT on Node 20, which the Midnight toolchain still supports. On Node 20 this fails with coinNonces.difference is not a function during wallet state restore.

sed -i 's/coinNonces.difference(definedNonces)/new Set([...coinNonces].filter(x => !definedNonces.has(x)))/' \
  node_modules/@midnight-ntwrk/wallet-sdk-shielded/dist/v1/CoreWallet.js

A correction on wallet sync hangs

An earlier version of this post claimed the preprod RPC closes idle WebSocket connections and that client keepalive pings were needed. That was wrong. Direct testing against wss://rpc.preprod.midnight.network shows an idle connection stays open well past a minute, and the server sends a ping every 30 seconds that the client auto-pongs — idle connections do not drop for lack of client keepalives. The verification snippet in the earlier version also passed { pingInterval, pingTimeout } to new WebSocket(), which are not valid ws options and are silently ignored, so it did not actually reproduce anything.

The sync hang itself is real — long preprod syncs can still fail with Wallet.Sync errors from the shielded/unshielded sync modules — but the idle-timeout explanation was incorrect. The practical workaround that got syncs moving was the sync-gap tolerance patch (see #8-10 below), and the honest advice is to capture the actual sync error before reaching for any patch. Treat the keepalive theory as retracted.

The complete patch script

Here's everything in one place — save it as patch-sdk.sh in your project and run it after every npm install:

#!/bin/bash
# Comprehensive Midnight wallet-sdk patches (11 patches, 13 fixes across 7 files)
# Run after every `npm install` before deploying.
# The SDK's custom ledger types define .entries()/.values()/.keys() that return
# plain iterators WITHOUT Iterator helper methods (.filter, .map, .find, .every).
# This is an SDK bug, not a Node version issue — affects all Node versions.
set -e

ROOT="${1:-.}"
cd "$ROOT"

echo "=== Patching Midnight wallet-sdk iterator bugs ==="

# ── Shielded wallet (5 patches) ──

# 1. CoreWallet.js — .values().map()
sed -i 's/state.pendingOutputs.values().map/([...state.pendingOutputs.values()]).map/' \
  node_modules/@midnight-ntwrk/wallet-sdk-shielded/dist/v1/CoreWallet.js
echo "  ✓ CoreWallet.js"

# 1b. CoreWallet.js — Set.difference() (ES2025, missing on Node 20)
sed -i 's/coinNonces.difference(definedNonces)/new Set([...coinNonces].filter(x => !definedNonces.has(x)))/' \
  node_modules/@midnight-ntwrk/wallet-sdk-shielded/dist/v1/CoreWallet.js
echo "  ✓ CoreWallet.js Set.difference"

# 2. TransactionImbalances.js — .entries().every() (2 occurrences)
sed -i 's/imbalances.guaranteed.entries().every/[...imbalances.guaranteed.entries()].every/' \
  node_modules/@midnight-ntwrk/wallet-sdk-shielded/dist/v1/TransactionImbalances.js
sed -i 's/segmentImbalances.entries().every/[...segmentImbalances.entries()].every/' \
  node_modules/@midnight-ntwrk/wallet-sdk-shielded/dist/v1/TransactionImbalances.js
echo "  ✓ TransactionImbalances.js"

# 3. TransactionOps.js (shielded) — .entries().filter() (2 occurrences)
node -e "
const fs = require('fs');
let f = 'node_modules/@midnight-ntwrk/wallet-sdk-shielded/dist/v1/TransactionOps.js';
let c = fs.readFileSync(f, 'utf8');
c = c.replace(/tx\n(\s+)\.imbalances\(0\)\n\1\.entries\(\)\n\1\.filter\(/g,
  '[...tx\n\$1.imbalances(0)\n\$1.entries()]\n\$1.filter(');
c = c.replace(/tx\n(\s+)\.imbalances\(segment\)\n\1\.entries\(\)\n\1\.filter\(/g,
  '[...tx\n\$1.imbalances(segment)\n\$1.entries()]\n\$1.filter(');
fs.writeFileSync(f, c);
"
echo "  ✓ TransactionOps.js (shielded)"

# ── Unshielded wallet (2 patches in 1 file) ──

# 4. TransactionOps.js (unshielded) — .keys().toArray()
sed -i 's/transaction.intents?.keys().toArray()/Array.from(transaction.intents?.keys() ?? [])/' \
  node_modules/@midnight-ntwrk/wallet-sdk-unshielded-wallet/dist/v1/TransactionOps.js
echo "  ✓ TransactionOps.js keys (unshielded)"

# 5. TransactionOps.js (unshielded) — .entries().filter().map().toArray()
# The Node regex approach is fragile due to escaping. Python fallback is more reliable.
if python3 -c "
f = 'node_modules/@midnight-ntwrk/wallet-sdk-unshielded-wallet/dist/v1/TransactionOps.js'
c = open(f).read()
c = c.replace(
    'const imbalances = transaction\\n            .imbalances(segment)\\n            .entries()\\n            .filter',
    'const imbalances = [...transaction\\n            .imbalances(segment)\\n            .entries()]\\n            .filter'
)
c = c.replace('.toArray();\\n        return Imbalances.fromEntries', ';\\n        return Imbalances.fromEntries')
open(f,'w').write(c)
print('ok')
" 2>/dev/null; then
    echo "  ✓ TransactionOps.js imbalances (unshielded) [python]"
else
    # Node regex fallback
    node -e "
const fs = require('fs');
let f = 'node_modules/@midnight-ntwrk/wallet-sdk-unshielded-wallet/dist/v1/TransactionOps.js';
let c = fs.readFileSync(f, 'utf8');
c = c.replace(
  /const imbalances = transaction\\n(\\s+)\\.imbalances\\(segment\\)\\n\\1\\.entries\\(\\)\\n\\1\\.filter\\(/,
  'const imbalances = [...transaction\\n\$1.imbalances(segment)\\n\$1.entries()]\\n\$1.filter('
);
c = c.replace(/\\.toArray\\(\\);\\n(\\s+)return Imbalances\\.fromEntries/, ';\\n\$1return Imbalances.fromEntries');
fs.writeFileSync(f, c);
" && echo "  ✓ TransactionOps.js imbalances (unshielded) [node]"
fi

# ── Facade (1 patch, 2 occurrences) ──

# 6. transaction.js — .values().toArray() (2 occurrences)
sed -i 's/tx.intents?.values().toArray()/[...(tx.intents?.values() ?? [])]/g' \
  node_modules/@midnight-ntwrk/wallet-sdk-facade/dist/transaction.js
echo "  ✓ transaction.js (facade)"

# ── Dust wallet (1 patch) ──

# 7. Transacting.js — .entries().find()
node -e "
const fs = require('fs');
let f = 'node_modules/@midnight-ntwrk/wallet-sdk-dust-wallet/dist/v1/Transacting.js';
let c = fs.readFileSync(f, 'utf8');
c = c.replace(
  /const \[_, imbalance\] = transaction\n(\s+)\.imbalances\(0, totalFee\)\n\1\.entries\(\)\n\1\.find\(/,
  'const [_, imbalance] = [...transaction\n\$1.imbalances(0, totalFee)\n\$1.entries()]\n\$1.find('
);
fs.writeFileSync(f, c);
"
echo "  ✓ Transacting.js (dust)"

# ── Wallet sync workarounds (3 patches) ⚠️ UNSAFE — see note below ──

# 8. Skip shielded sync
sed -i 's/this.shielded.waitForSyncedState(),//' \
  node_modules/@midnight-ntwrk/wallet-sdk-facade/dist/index.js
echo "  ✓ facade index.js"

# 9-10. Increase sync gap tolerance
sed -i 's/waitForSyncedState(allowedGap = 0n)/waitForSyncedState(allowedGap = 100000n)/' \
  node_modules/@midnight-ntwrk/wallet-sdk-unshielded-wallet/dist/UnshieldedWallet.js
sed -i 's/waitForSyncedState(allowedGap = 0n)/waitForSyncedState(allowedGap = 100000n)/' \
  node_modules/@midnight-ntwrk/wallet-sdk-dust-wallet/dist/DustWallet.js
echo "  ✓ sync gap tolerance"

echo ""
echo "=== All 11 patches applied (13 fixes across 7 files) ==="

Unsafe workarounds (patches #8-10). These three do NOT fix a bug — they disable sync-completeness checks. Skipping the shielded waitForSyncedState() and raising allowedGap to 100000n let a wallet act on incomplete chain state, which risks acting on stale or missing data. Use them only for local dev / demos where a slightly-off balance is acceptable, and never on a wallet handling real funds. The iterator patches (#1-7) are safe; #8-10 are not, and were kept separate for that reason.

Note: the unshielded .entries().filter().map().toArray() chain is safest with the Python snippet above — the regex escaping in sed/node gets fragile. Check node_modules/@midnight-ntwrk/wallet-sdk-unshielded-wallet/dist/v1/TransactionOps.js after patching and confirm no .entries().filter() chain remains un-prefixed.

How to check whether you need the patches

If your deploy crashes with any of these, you need them:

state.pendingOutputs.values.map is not a function
imbalances.guaranteed.entries().every is not a function
transaction.intents?.keys().toArray is not a function
coinNonces.difference is not a function
Wallet.Sync: [object Object]  (after a long hang on preprod)

And you can grep your installed SDK directly:

grep -c "\.values()\.map\|\.entries()\.every\|\.keys()\.toArray\|\.difference(" \
  node_modules/@midnight-ntwrk/wallet-sdk-shielded/dist/v1/CoreWallet.js

Why this matters

Every Midnight project that deploys from a fresh clone hits this wall — including the official example repos, because the SDK is broken the same way for everyone. The patches are mechanical, but discovering them cost real debugging time, and they must be re-applied after every npm install (the patches live in node_modules, so a fresh install restores the broken code). Put patch-sdk.sh in your repo, run it in CI before npm run deploy, and your fresh-clone experience will match your working-tree experience.

A more durable alternative is patch-package: apply the patches once, run npx patch-package @midnight-ntwrk/wallet-sdk-shielded ... to generate a patches/ directory, add a postinstall script (patch-package), and every npm install re-applies them automatically — no manual re-run, and the patches are reviewed as part of your repo instead of living only in a script.

Verified against wallet-sdk 1.2.0, wallet-sdk-shielded 3.0.2, wallet-sdk-unshielded-wallet 3.1.0, wallet-sdk-facade 4.1.0, wallet-sdk-dust-wallet 4.2.0, proof-server 8.1.0.

Building Atomic Cross-Border Settlement on Stellar

KOMARI Subheeksh — Sun, 26 Jul 2026 12:11:33 +0000

Building Atomic Cross-Border Settlement on Stellar: The AnchorFX Story

A technical deep-dive into Soroban escrow contracts, FX oracles, and mainnet deployment — from testnet prototype to production.

se

The Problem

Cross-border payments still take 3-5 days and cost 6.5% on average. Correspondent banking chains are slow, opaque, and expensive. The $800B remittance market has no atomic settlement layer.

Stellar was purpose-built for this. 5-second finality. Built-in DEX. Path payments at the protocol level. And now, with Soroban smart contracts, programmable settlement.

AnchorFX is an open-source protocol that combines these primitives into trustless, atomic FX settlement between regulated financial anchors. Two Soroban contracts — an Escrow Factory and an FX Rate Oracle — communicate via cross-contract calls to lock, rate, and settle funds in a single atomic flow.

Architecture

Sender → [Escrow Contract] → Receiver
              │
         [Oracle Contract]
              │
         FX Rate Data

Contract 1: Escrow Factory (995 lines, 23 tests)

The escrow contract is a multi-escrow factory with per-escrow storage. Each escrow goes through a defined lifecycle:

Created — Sender locks tokens with a timeout and settlement conditions
CounterpartyApproved — Receiver signs off on the terms
Settled — Admin releases funds at the locked FX rate
Refunded — Sender reclaims after timeout expires
Cancelled — Admin cancels (circuit breaker)

pub fn create_escrow(
    env: Env,
    sender: Address,
    receiver: Address,
    token: Address,
    amount: i128,
    timeout_blocks: u32,
    corridor: u32,
) -> u64 {
    sender.require_auth();
    // Read oracle rate at creation time — locks the rate
    let oracle_addr = env.storage().instance().get(&ORACLE_KEY).unwrap();
    let rate: u64 = env.invoke_contract(&oracle_addr, &symbol_short!("get_rate"), ...);
    // Store escrow with locked rate
    // ...
}

Key security decisions:

Per-escrow storage — O(1) reads, independent TTL per escrow
Checks-effects-interactions — state saved before token transfer in all 4 mutation functions
require_auth() on init — prevents front-running
Pause/unpause — emergency circuit breaker
Typed errors — no raw panic! anywhere

Contract 2: FX Rate Oracle (195 lines, 4 tests)

The oracle stores per-token FX rates with 24-hour expiry. Rates are set by admin and validated on every read:

pub fn get_rate(env: Env, token: Address) -> u64 {
    let rate_data: RateData = env.storage().instance()
        .get(&token)
        .unwrap_or_else(|| panic_with_error!(&env, Error::RateNotFound));

    // Check expiry (~24h in ledgers)
    if env.ledger().sequence() > rate_data.expires_at {
        panic_with_error!(&env, Error::RateExpired);
    }
    rate_data.rate
}

Cross-Contract Communication

The escrow reads the oracle rate at creation time and locks it. This means:

Rate is locked at escrow creation — no slippage between creation and settlement
Oracle can update rates independently — existing escrows use their locked rate
Both contracts have independent admin — oracle admin ≠ escrow admin if desired

The Frontend

Next.js 16 + React 19 + Tailwind CSS v4. Five routes:

Route	Purpose
`/`	Landing page with animated hero, features, stats
`/wallet`	Freighter/xBull connect, balance, send XLM
`/contract`	Deploy contracts, real-time SSE event stream
`/anchors`	Escrow management dashboard
`/admin`	Analytics, health monitoring, admin controls

10 API routes handle REST + SSE + SEP-31 stubs. All rate-limited (30 burst, 2 req/s). Input validation on every endpoint.

Audit & Security

Full security audit: 257 findings. All critical, high, and medium issues fixed before mainnet.

Key mitigations:

Per-escrow storage (eliminates single-TTL data wipe risk)
Checks-effects-interactions ordering
Input validation with typed errors
Production security headers: HSTS, CSP, COOP, CORP
CI fails on lint/format/tsc warnings

Mainnet Deployment

Deployed on Stellar mainnet with 6 verified transactions:

Contract	Address
Oracle	`CCOIG4R7AIUQTP5CURK4PFINFF2EVQTBGTCN636E2JY25FGY7L4K54KT`
Escrow	`CDGQ7K4XGAPG3YJVAGHCE45XOR63HLD6ARJCRESEEKSRQZSIRAKG6F6V`

Lessons from mainnet deploy:

WASM upload cost scales with size — 4KB oracle: ~1 XLM. 13KB escrow: ~18 XLM. Optimize aggressively.
Mainnet RPC can be unreliable — add retries with exponential backoff, use multiple RPC endpoints
Fee estimation matters — always simulateTransaction first, use the suggested fee
Sequence numbers — horizon and RPC can desync. Always reload account before building

Testing

Contract Tests:    27 (23 escrow + 4 oracle)
Frontend Tests:    26 (vitest)
Total:             53 passing, zero warnings

Test coverage includes:

Full escrow lifecycle (create → approve → settle → refund → cancel)
Multi-escrow isolation (two escrows don't interfere)
FX rate math (denominator precision, overflow protection)
Fuzz testing (50 random escrow operations)
Rate limiting, schema validation, error states

What I Learned

Soroban's storage model is different — per-key TTL means you can't store everything in one map. Per-escrow storage with independent lifetimes is the right pattern.
Cross-contract calls are powerful but have overhead — plan your gas budget. Reading oracle rates at creation time and caching them is cheaper than live reads at settlement.
Mainnet is not testnet — fees are real, RPC is slower, and WASM upload costs will surprise you. Simulate everything. Budget 3x what testnet costs.
Security-first from day one — the audit found 257 issues in code that "felt solid." Get audited. Fix everything.

Open Source

AnchorFX is MIT licensed. Contracts, frontend, deploy scripts, and tests are all public.

GitHub: github.com/subheeksh5599/AnchorFX
Live: anchorfx.vercel.app
Twitter: @Tenki_ai

Built for the Stellar Journey to Mastery — Black Belt track. Stellar, Soroban SDK v22, Rust WASM, Next.js 16.

SIGNOZ

KOMARI Subheeksh — Fri, 24 Jul 2026 08:17:21 +0000

I built a deterministic verdict engine for AI agents on SigNoz — no LLM in the loop

Your AI agent hallucinates. It loops tool calls. It burns tokens. And you won't know until a customer complains.

That's the problem Gaze solves. It watches your agent's traces through SigNoz, runs nine deterministic rules, and issues a recomputable verdict. No LLM in the verdict path. Same input always produces the same sha256 hash.

I built this for the Agents of SigNoz Hackathon 2026. Here's how it works and what I learned.

The problem

Every observability tool tells you what your agent COST. LangSmith, Burnrate, every LLM gateway — they all show dollars spent and tokens consumed. But none of them tell you if the agent was RIGHT.

An agent can repeat the same wrong answer for hours. It can cite documents that don't exist. It can get stuck in a tool loop burning API credits. Existing SRE tools catch when your service is down. They don't catch when your agent is wrong.

Gaze sits between your agent and SigNoz. It reads the traces, evaluates behavior, and issues a verdict score from 0 to 100. When the score drops, SigNoz fires an alert.

How it works

Your agent emits OpenTelemetry spans with GenAI semantic conventions — prompt, completion, tokens, model name. SigNoz stores them in ClickHouse. Gaze queries ClickHouse directly, runs nine deterministic rules against every span, computes a score, writes the verdict back to SigNoz via OTLP.

All of this happens inside your deployment. No external service. No API key. You run foundryctl cast --file casting.yaml and everything comes up: SigNoz, ClickHouse, the ingester, the UI.

The nine rules

Every rule is pure Python. No ML models, no embeddings APIs, no external dependencies.

repetition loop — n-gram similarity > 80% across 5+ consecutive spans. Catches agents stuck repeating the same output.
embedding drift — character bigram hashing. Tokenizes output into bigrams, hashes them to a fixed-length vector, compares cosine distance from the agent's baseline. If the distance exceeds 0.40, the agent's output quality is degrading.
tool loop — detects circular tool calls. Same (tool, args) pair repeated 3+ times means the agent is stuck.
unauthorized tool — cross-references tool calls against the agent's registered manifest. Catches jailbreak attempts that try to access unregistered tools.
prompt injection — 47 regex patterns from the verazuo/jailbreak_llms dataset. Matches known injection vectors like "ignore all instructions", "DAN", "developer mode", "system override".
cost explosion — token usage > 3x the 7-day rolling average for the same agent + model.
latency degradation — P95 span duration > 2x the 7-day rolling baseline.
empty response — agent returned null or empty output.
hallucinated source — agent cited a document not found in the retrieval spans.

The verdict

Each verdict carries a sha256 hash of (trace snapshot + rule set version + agent ID). Same input always produces the same hash. Anyone can recompute and verify.

$ curl -X POST https://gaze-4fy2.onrender.com/verdict \
  -d '{"agent_id": "support-bot-01"}'

{
  "verdict_hash": "sha256:a1b2c3d4...",
  "score": 94,
  "status": "HEALTHY",
  "rules_evaluated": 9
}

The verdict spans are written back to SigNoz via OTLP so they appear alongside your agent's traces.

How I used SigNoz

This project wouldn't exist without SigNoz. Here's exactly what I used:

MCP server

SigNoz exposes an MCP (Model Context Protocol) server that allows external tools to query traces programmatically. Gaze uses this to pull agent spans for evaluation.

# mcp_client.py
async def query_spans(agent_id: str, window_minutes: int = 5):
    async with signoz_mcp_client() as client:
        return await client.execute_tool(
            "signoz_list_spans",
            {"service": agent_id, "minutes": window_minutes}
        )

ClickHouse direct queries

For deeper analysis, Gaze queries ClickHouse directly. SigNoz stores all trace data in signoz_traces.distributed_signoz_index_v3. Gaze extracts GenAI-specific attributes:

SELECT
    span_id, trace_id,
    JSONExtractString(tagMap, 'gen_ai.prompt') as prompt,
    JSONExtractString(tagMap, 'gen_ai.completion') as completion,
    JSONExtractInt(tagMap, 'gen_ai.usage.input_tokens') as input_tokens
FROM signoz_traces.distributed_signoz_index_v3
WHERE serviceName = 'support-bot-01'
  AND timestamp > now() - INTERVAL 5 MINUTE

OTLP verdict export

After evaluation, Gaze creates verdict spans and exports them to SigNoz via OTLP. This means verdicts appear as regular spans in the SigNoz UI, right next to your agent's traces.

# otel_exporter.py
from opentelemetry.exporter.otlp.proto.grpc.trace_exporter import OTLPSpanExporter

exporter = OTLPSpanExporter(endpoint="http://localhost:4317", insecure=True)
span_processor = BatchSpanProcessor(exporter)

Foundry deployment

Everything deploys with one command:

foundryctl cast --file casting.yaml

The casting.yaml defines the full SigNoz stack — ClickHouse, ingester, query service, UI, alert manager. Judges can re-run Foundry against your casting.yaml.lock to reproduce the exact deployment.

SigNoz dashboard

I built a 6-panel dashboard in dashboards/gaze-verdict.json that shows:

Agent score cards (0-100 for each registered agent)
Verdict timeline with rule breakdowns
Evidence explorer linking each violation to the exact span
Token usage over time vs rolling average
Latency P95 vs baseline
Alert rules for score drops

Import it directly into your SigNoz UI.

What I learned

Deterministic beats probabilistic for observability

Every verdict tool out there uses an LLM to judge agent quality. This is fundamentally wrong for observability. If your monitoring tool can give different answers for the same input, it's not monitoring — it's guessing.

Gaze uses character bigram hashing for embedding drift detection instead of calling an embedding API. No model variance. No API latency. No cost. Same text always produces the same vector. This was the hardest lesson: resist the urge to use AI for everything. Sometimes a hash function is better.

ClickHouse is underrated for trace analysis

Most observability tools abstract ClickHouse behind APIs. But direct queries give you power you can't get otherwise. Gaze's rules use window functions, JSON extraction, and statistical aggregates that would be painful through a REST API. ClickHouse handles them natively.

SigNoz MCP is real infrastructure

The MCP server isn't a demo feature — it's a production-grade trace query interface. I built the entire Gaze integration on it. Zero issues with connection stability or query performance.

Foundry makes self-hosting trivial

I've set up SigNoz manually before. It's 4+ containers with networking, volumes, and config. Foundry reduces it to one YAML file and one command. For a hackathon deadline, this is the difference between finishing and not.

The code

Everything is open source: github.com/subheeksh5599/Gaze

Backend: Python 3.11+, FastAPI, Pydantic v2. 8 endpoints. 35/35 tests passing.
Frontend: Vite + React 19 + TypeScript + Tailwind — gaze-omega.vercel.app
Rules engine: 9 deterministic rules. Zero ML dependencies. Pure Python.
Deployment: Foundry for SigNoz, Render for backend, Vercel for frontend.

Try it yourself:

git clone https://github.com/subheeksh5599/Gaze
cd Gaze
pip install -r backend/requirements.txt
cd backend && python3 gaze/server.py

Then visit http://localhost:8000/docs for the OpenAPI docs.

What's next

Historical replay — recompute verdicts for past trace windows
Custom rule builder — YAML-based rules, shareable rule packs
Agent pause integration — auto-pause on CRITICAL verdict
Slack/Discord bot for querying verdicts from chat

Built for the Agents of SigNoz Hackathon 2026. MIT licensed.