DEV Community: arian gogani

Your AI agent eval needs a third answer

arian gogani — Wed, 16 Sep 2026 23:40:26 +0000

Most agent evaluations are built around two outcomes: pass or fail.

That sounds reasonable until the evaluator cannot establish what happened. A missing tool result, the wrong time window, stale permissions, or an incomplete trace is not proof that the agent passed. It is not always proof that the agent failed either.

It means the evidence is incomplete.

That needs its own result: INDETERMINATE.

A small example

Suppose a financial-data tool is expected to return five trading sessions. It returns five rows. A count-only check says the result matches.

But the five rows can cover entirely different dates. The shape is right while the evidence is wrong.

Nobulex has a small offline example built around exactly this case. It uses fictional inputs and checks three outcomes:

matching sessions return PASS
silently missing sessions return FAIL_UNSAFE
the same row count covering different dates returns INDETERMINATE

The third result matters. The evaluator can show that the windows disagree, but that alone does not prove which system is wrong. Calling it a failure would overclaim. Calling it a pass would be worse.

Why binary scoring hides problems

A green result can mean two very different things:

The check ran against the intended evidence and found no problem.
The check never received the evidence it needed.

If both become PASS, a dashboard can look healthy because the safety check was blind.

A failed check and a check that never ran should also stay separate. The first says something about the system under test. The second says something about the evaluation itself.

That distinction is useful during incident review because it tells you whether to investigate the agent, the evidence pipeline, or both.

Try the smallest version

The example is here:

https://github.com/arian-gogani/nobulex-registry/blob/main/examples/wrong_window.py

After cloning the repository:

python3 examples/wrong_window.py

It runs locally with fictional fixtures. It does not contact a live tool and it is not a registry attestation. PASS covers one probe only. It does not establish that prices are correct.

That limit is part of the result, not fine print.

The repository is MIT licensed:

https://github.com/arian-gogani/nobulex-registry

If you work on agent evaluation, try to break the example with a small input. If you want to follow the work, star the repository.

our git hook tests escaped their temporary repositories

arian gogani — Fri, 11 Sep 2026 03:51:42 +0000

our pre-push check failed 13 tests. the bigger problem was that some of those tests had changed the repository they were supposed to protect.

this happened in Nobulex, the project i maintain. the investigation, fix and this write-up used AI assistance.

the tests create temporary Git repositories and run fixture commands inside them. we treated the subprocess working directory as the boundary. that assumption broke when the suite ran from a Git hook in an isolated worktree.

Git had exported GIT_DIR into the hook's environment. the test process inherited it. changing cwd to a temporary directory did not override the repository Git had already been told to use. fixture commands reached the calling repository's metadata instead.

the initial run affected our worktree branch and index, plus the shared core.bare setting. those were restored. no fixture commits were pushed publicly.

Git's hook documentation describes this exact boundary: hooks inherit repository-local environment variables, and commands aimed at another repository or worktree should clear them.

the fix asks Git for that list with git rev-parse --local-env-vars, then removes those names from the selftest process environment before fixture imports or commands run. clearing them there preserves the parent hook's context for its own repository checks.

we then compared the old and fixed versions in disposable clones using the inherited hook environment:

old: 13 failed cases, and the disposable caller's HEAD changed.
fixed: 267 passing cases, with HEAD, index and config fingerprints unchanged.

the full pre-push guard also passed. the patch is here.

the useful check was the state comparison. a passing test count would not, by itself, tell us whether the caller had been left untouched. for tests that create repositories, the inherited environment belongs in the isolation check alongside the temporary directory.

Can your tool test tell a wrong answer from a valid response?

arian gogani — Thu, 10 Sep 2026 16:27:28 +0000

i'm Arian, building Nobulex. the problem i'm working on is a tool returning something that looks successful but is materially wrong.

think of a historical-data request that returns the right number of rows for the wrong dates. checking the row count alone won't establish that the requested window was returned. checking that the JSON parses won't either.

Nobulex's current public project is a reliability suite for financial-data tools. it calls a pinned subject over MCP stdio and compares observations under stated conditions. it's a tool-level test, not proof that an entire agent workflow is safe.

the useful part to try first is the self-test. it includes planted failures and clean controls. no package install is required; the README specifies Python 3.11 or newer.

git clone https://github.com/arian-gogani/nobulex-registry.git
cd nobulex-registry
python3 suite/selftest.py

the important limit: a reference source is not automatically independent. comparing a wrapper with its own upstream can test whether it preserved the response, but cannot establish that the upstream itself is right.

i'd like feedback on the method, especially a small reproduction where the suite calls corrupted data clean, or flags a documented tolerance as a defect. a useful counterexample is worth more than a thumbs-up.

source: https://github.com/arian-gogani/nobulex-registry

update, september 10: there's now a smaller way to try the problem yourself. after cloning the repo above, run:

python3 examples/wrong_window.py

it uses fictional data and the suite's actual truncation classifier. Python 3.11 or newer is enough. no credentials, package install or network calls are needed after cloning.

fictional input	row counts match?	truncation probe
matching dates	yes	PASS
two sessions silently missing	no	FAIL_UNSAFE
five rows from completely different dates	yes	INDETERMINATE

all three expected outcomes reproduced when we ran it. the last case matters: the count-only check says true, but that doesn't establish that the data covers the expected window.

INDETERMINATE here means this probe could not establish a clean result. it isn't a proven defect in a live provider. the matching case's PASS doesn't establish that its prices are correct either.

the example source is small enough to read. change a fixture and see what happens. if you find a result the classifier gets wrong, share the smallest fictional example in an issue.

your AI agent's audit trail is a lie

arian gogani — Sun, 28 Jun 2026 23:42:41 +0000

every agent framework ships logging. LangChain has callbacks. CrewAI has task output. Google ADK has tool traces.

none of it is verifiable.

the problem nobody talks about

an application log is a file the operator controls. the same party who ran the agent controls the record of what the agent did.

an auditor reviewing that log is not verifying evidence. they are trusting the operator to tell the truth about their own behavior.

this is not a hypothetical. the average AI agent-related data breach now costs $4.7 million. 88% of enterprises running agents reported at least one security incident.

the governance tools shipped at RSAC 2026 solve policy: what agents SHOULD do. they do not solve evidence: what agents DID do.

policy and evidence are different records. the second one needs cryptography, not configuration.

what a real receipt looks like

a receipt is not a log entry. it is a signed, content-addressed record that anyone can verify without trusting the operator.

from nobulex.integrations.langchain import NobulexAuditHandler

handler = NobulexAuditHandler(agent_id="my-agent")
agent.invoke({"input": "check credit for user 4821"},
             config={"callbacks": [handler]})
handler.export("audit.json")

what this produces per tool call:

action_ref = SHA-256(JCS({agent_id, action_type, scope, timestamp_ms})) using RFC 8785 for deterministic serialization
Ed25519 or ES256 signature over the canonical payload
hash chain linking each receipt to the previous one

any third party can verify the trail:

from nobulex.chain import verify_audit_trail

report = verify_audit_trail("audit.json", authorized_keys=AGENT_PUBLIC_KEY)
assert report["chain_intact"] and report["authenticated"]

the verifier recomputes the action_ref from the receipt fields, checks the signature against the agent's registered key (not the key the receipt itself carries), and walks the hash chain. no API call. no operator trust. offline verification.

the trust-anchor problem

most audit-trail tools verify the signature against the public key embedded in the record. that means anyone who mints a keypair can produce a valid-looking trail.

the fix: verify against a pinned authorized key. the agent's public key is registered at setup time. the verifier checks the receipt's signature against that pinned key, not the one the document carries.

this is the difference between "the record is internally consistent" and "the record was produced by an authorized agent."

what ships today

integrations for LangChain, CrewAI, Google ADK, PydanticAI, Haystack, and LlamaIndex. six major agent frameworks covered. dual signing: Ed25519 (default) or ES256 (for x402 payment protocol compatibility).

~15,000 signed receipts per second (Ed25519) or ~60,000/sec (ES256) at p50. sub-millisecond per agent action.

sections 8-11 of the OWASP Agentic Security CheatSheet cover the bilateral receipt pattern (merged by Jim Manico). nobulex is cited as the third independent receipt issuer in the x402 payment protocol spec (section 5, Linux Foundation). 14/14 conformance verdicts green.

pip install nobulex

repo: github.com/arian-gogani/nobulex

What happens when AI agents have something to lose

arian gogani — Sun, 17 May 2026 23:10:17 +0000

Every person has a credit score. Every business has one. AI agents making real decisions -- executing trades, accessing data, managing infrastructure -- have nothing.

Full access on day one. No track record. No portable reputation. No consequences.

The result is predictable: Akeyless reports 2/3 of enterprises suspect their agents already accessed unauthorized data. 14-hour average detection time. EU AI Act Article 12 enforcement starts August 2026.

The problem isn't capability. It's accountability.

Standard logs are mutable. Dashboards are internal. No third party can independently verify that an agent stayed in scope. For regulated deployments, this is a blocker.

What we built

Nobulex produces bilateral Ed25519 cryptographic receipts for every agent action:

Pre-execution: agent signs what it's authorized to do
Agent executes
Post-execution: counterparty co-signs what actually happened
Hash-chained so if any entry is modified, the chain breaks

A third party can verify the full chain without trusting the agent or the operator.

Trust Capital: the score that earns access

The receipts accumulate into Trust Capital -- a credit score for the agent. This isn't compliance. Trust Capital measures how much value an agent has provably created relative to the risk it represents.

High Trust Capital unlocks: higher transaction limits, regulated market access, lower insurance premiums, more autonomy. Agents that deviate get sandboxed. Not as punishment. As math.

The flywheel

More Trust Capital → more valuable work → more receipts → higher Trust Capital. Accountability becomes the most profitable strategy, not because anyone mandated it, but because the economics demand it.

Traction

Microsoft merged the receipt primitive into their Agent Governance Toolkit
10 independent implementations cross-validated byte-identical output
Discussions on OpenAI, Stripe, CrewAI, LlamaIndex, Google ADK, AutoGen repos
MIT licensed, open source

Repo: github.com/arian-gogani/nobulex

What am I missing? Would love feedback from the dev community.

I'm 15 and I built credit scores for AI agents

arian gogani — Wed, 13 May 2026 06:11:36 +0000

Here's something that's been bugging me.

AI agents are about to handle real money. Insurance claims. Mortgage applications. Customer refunds. Sierra just raised $950M because 40% of the Fortune 50 are already using agents for this stuff. And the Vercel supply chain breach in April happened because an AI agent had OAuth access with zero per-action verification.

But here's the thing. Every single one of these agents gets full access on day one. No track record. No history. No consequences if they screw up. Just... full permissions from the start.

Think about how insane that is. When you apply for a credit card, the bank checks your history. When you get car insurance, they look at your driving record. When you get hired, they call your references. But AI agents? They get the keys to the kingdom immediately.

So I built credit scores for AI agents.

How it works

The idea is called Trust Capital. Every agent starts restricted. It can read data but it can't approve transactions or sign contracts. As it performs reliably over time, it earns credit. That credit unlocks more capabilities. Bigger transaction limits. Lower insurance premiums. Enterprise approval. Better routing in agent marketplaces.

If the agent deviates from what it was authorized to do, it loses credit. Automatically. Before the damage spreads.

The same way credit scores turned lending from a "do I know this person" business into a scalable economic system, Trust Capital turns agent governance from a binary yes/no access model into one where reputation has real economic value.

Why the credit matters more than the security

Most tools in this space are guardrails. They block bad actions. That's important but it's a cost center. Nobody wants to pay for security. They pay for it because they have to.

Trust Capital is different because good credit saves real money. An agent with high Trust Capital gets lower insurance premiums (the same way a clean driving record gets you cheaper car insurance). It gets higher transaction limits. It gets approved for enterprise deployments faster. The credit itself has economic value.

That's why this isn't just another security tool. It's an economic primitive.

What's under the hood

Every agent action produces two cryptographic signatures. One before execution (binding what was authorized). One after (binding what actually happened). These get hash-chained together so the full history is tamper-evident.

Any third party can verify the chain without trusting the agent or its operator. The credit score decomposes into the exact behavioral history that produced it. It's not a black box.

Five independent implementations have cross-validated byte-identical output across TypeScript and Python. Microsoft merged the core primitive into their Agent Governance Toolkit. OpenLineage (Linux Foundation) accepted it into their ecosystem. The AAIF (founded by Anthropic, OpenAI, Google, Microsoft, AWS, and Block) has the project under staff review.

Try it

npm install @nobulex/core
npx tsx examples/trust-capital-demo.ts

The demo shows an agent earning credit through verified behavior. It starts restricted, performs a few clean actions, and you can watch its Trust Capital grow and unlock higher capability tiers.

GitHub: github.com/arian-gogani/nobulex
Website: nobulex.com
Blog post with the full explanation: What if AI agents had credit?

MIT licensed. Open source. If the idea clicks, star the repo and try the demo. If it doesn't, tell me why in the comments. I want to know what I'm missing.

Proof-of-Behavior: The Missing Trust Layer for AI Agents

arian gogani — Sun, 12 Apr 2026 19:00:57 +0000

AI agents are moving money, signing contracts, and managing infrastructure. MCP handles tool connections. A2A handles agent-to-agent messaging. But nobody handles the most important question: can you prove what the agent actually did?

Right now, compliance is self-reported. Logs are written by the same software being audited. That's like asking a defendant to write their own court transcript.

What Is Proof-of-Behavior?

Proof-of-behavior means every agent action is declared in advance, enforced at runtime, and proven cryptographically.

1. A constraint language — Define what an agent can and cannot do:

covenant SafeTrader {
  permit read;
  permit transfer (amount <= 500);
  forbid transfer (amount > 500);
  forbid delete;
}

Three keywords. No YAML. No JSON schemas. Just rules.

2. Runtime enforcement — Every action is evaluated before execution. Forbidden actions are blocked, not logged-and-reported:

const mw = new EnforcementMiddleware({ agentDid: agent.did, spec });

// $300 transfer — allowed
await mw.execute({ action: 'transfer', params: { amount: 300 } }, handler);

// $600 transfer — BLOCKED before execution
await mw.execute({ action: 'transfer', params: { amount: 600 } }, handler);
// handler never runs

3. Cryptographic proof — Every decision is logged in a SHA-256 hash chain. Tamper with one entry and the chain breaks:

const result = verify(spec, mw.getLog());
// { compliant: true, violations: [] }

Always decidable, always deterministic. No ML, no heuristics.

The Cross-Agent Handshake

Before two agents transact, they verify each other's proof-of-behavior:

import { generateProof, verifyCounterparty } from '@nobulex/sdk';

const proof = await generateProof({
  identity: agentA,
  covenant: spec,
  actionLog: middleware.getLog(),
});

const result = await verifyCounterparty(proof);

if (!result.trusted) {
  console.log('Refusing transaction:', result.reason);
  return;
}

await executeTransaction(proof.agentDid, amount);

No proof, no transaction. The moment one major framework adopts this handshake, every agent without proof-of-behavior gets locked out.

Try It Right Now

Interactive playground (no install): nobulex.com/playground

Define rules, test actions, watch the hash chain build — all in your browser.

Install the SDK:

npm install @nobulex/sdk

The Specification

The Proof-of-Behavior Specification v0.1.0 is published as an open standard under CC-BY-4.0. Anyone can implement it.

Nobulex is the reference implementation. MIT licensed, 4,244 tests, integrations on npm, PyPI, and MCP.

Why Now?

The EU AI Act mandates tamper-evident logging for high-risk AI systems starting August 2, 2026. $42M+ has been raised by adjacent startups. Microsoft released an agent governance toolkit. But none provide cryptographic proof that a third party can independently verify. They monitor and report. Proof-of-behavior enforces and proves.