Explore how Jev and NovaFabric could support verifiable AI agent decision receipts, policy checks, and replayable execution evidence. This is a research proposal, not a shipped integration.
Imagine an AI agent working on a data-processing task.
At some point, it decides to upload a file to an external endpoint. The operation succeeds, and the workflow continues.
Six months later, someone asks:
- Why did the agent decide to upload that file?
- What information was available when the decision was made?
- Which model evaluated the action?
- Was the action authorized?
- Can we verify the original record without trusting the running system?
A successful HTTP response cannot answer those questions.
What if every important AI-agent decision could produce a verifiable receipt?
This is a research direction I'm exploring around two technologies:
- Jev, a System One model from TypeSafe AI designed for fast, structured decisions.
- NovaFabric, an open-source system I've been developing to capture, replay, compare, and verify AI-agent execution evidence.
The idea is not to connect two APIs and call it an integration.
It's to investigate how we might preserve the evidence behind an AI decision so that engineers, researchers, and auditors can inspect it later.
Current status: This is an architectural proposal. The Jev integration described here has not been shipped or benchmarked, and there is no announced partnership with TypeSafe AI.
1. Why Jev is interesting
Many AI-agent decisions don't require an essay.
An agent might need to answer:
- Is this tool call consistent with the assigned task?
- Which risk category best describes an operation?
- Does this answer satisfy a particular evaluation criterion?
- Does a delegated action appear to exceed the original task?
Traditional LLMs can answer these questions, but generating explanatory text is not always the right interface.
TypeSafe AI's Jev exposes three useful primitives:
| Primitive | Purpose |
|---|---|
| Choice | Select from predefined options |
| Score | Evaluate something against a defined rubric |
| Noul | Return a probability for a yes/no proposition |
For example, we might ask Jev whether an agent's proposed file transfer appears unrelated to its assigned task.
But there is a critical distinction:
A model's probability is not proof that an action is safe, unsafe, or authorized.
It is a signal that must be evaluated against real outcomes and combined with independently enforced rules.
That's where the next part becomes interesting.
2. What NovaFabric adds
NovaFabric focuses on something different from model inference: preserving agent execution evidence.
Its central artifact is a Run Capsule, a portable directory containing recorded execution information from supported capture surfaces.
Depending on the workload, that includes model calls, tool calls, execution traces, environment information, and provenance.
NovaFabric supports structural comparison and scoped replay. Capsules can also be cryptographically sealed using an operator-configured key, making later modifications detectable.
If you want to try the verification side today, this tutorial walks through it end to end: Verify a sealed AI agent run for an auditor, offline.
This matters because an AI decision has two distinct questions:
Was the judgment useful or correct?
That requires evaluation against evidence and outcomes.
Is the archived decision record unchanged?
That can be checked using cryptographic integrity mechanisms, subject to their trust assumptions.
These are different problems. We should not confuse them.
For more background, see my arXiv preprint:
NovaFabric: Tamper-Evident, Replayable Evidence for Autonomous AI Agent Runs
3. The idea: a Decision Witness
I'm considering an optional component called a Decision Witness.
Think of it as a versioned record that connects an agent action with the information used to assess and authorize it.
A proposed record could contain:
- An identifier for the tool call or agent action.
- The model and version used for an assessment.
- The question template and relevant input digest.
- The model's structured output and probability distribution.
- The applicable policy and authorization result.
- Human approval information, when relevant.
- A reference to the captured execution evidence.
Here's an illustrative schema:
{
"schema_version": "decision-witness/0.1",
"event_ref": "tool-call-42",
"assessment": {
"provider": "typesafe",
"model_id": "<pinned-jev-version>",
"question_id": "off_task_transfer",
"type": "noul",
"probability": 0.87
},
"authorization": {
"policy_id": "outbound-policy-v3",
"outcome": "review_required"
},
"human_approval": "pending",
"origin": "observed",
"evidence_ref": "sha256:<digest>"
}
This is a hypothetical design example, not an implemented NovaFabric schema or a real Jev response. The probability is illustrative.
The important feature isn't the number 0.87.
It's the ability to revisit the evidence, policy, and decision that accompanied it.
4. How could the architecture work?
A possible architecture:
Conceptual architecture: proposed, not implemented. Policy decides; Jev only advises; the witness records both.
The exact enforcement boundary matters.
Jev would not be allowed to override identity permissions, data policies, or hard authorization constraints.
A policy engine would remain responsible for those checks.
The Decision Witness would preserve the observed assessment and policy outcome. For post-run assessments, a separately linked attestation would avoid silently modifying an already sealed capsule.
5. Four experiments I'd like to explore
Experiment A: Detect unusual agent behavior
Take a collection of recorded agent tool calls, including benign and suspicious scenarios.
Run a Jev-based assessment in shadow mode.
Compare it against deterministic rules and NovaFabric's existing experimental security/provenance anomaly detector.
Research questions:
- Does Jev identify useful semantic anomalies that structural detection misses?
- How many false alarms does it produce?
- How does its performance change across agent tasks and tool types?
Experiment B: A fast evaluation judge
Could Jev act as an additional judge for agent outputs?
For example, use rubric-based scores to evaluate whether an agent's output satisfies specific requirements.
Compare Jev against existing numerical, rule-based, and LLM-based judges.
The interesting question isn't just which judge is faster.
It's whether the scores remain useful and calibrated across different tasks.
Experiment C: Replay recorded decisions
Suppose a decision was captured today.
Three months later, the underlying model has changed.
Can we replay the recorded response without calling the model again, then separately compare it with a new model evaluation?
This would help distinguish the historical decision from a new assessment.
It would not prove that the model itself is deterministic.
Experiment D: Multi-agent delegation
Imagine several agents working across machines or organizations.
One agent delegates a task to another, which then invokes a tool.
Could we record the delegation chain, assess unusual handoffs, and determine whether authorization remained within the delegated scope?
I think this is particularly interesting for distributed and multi-agent AI systems.
6. How should we evaluate these ideas?
Before claiming that Jev improves agent safety or reliability, I'd want to run controlled experiments.
Useful measurements would include:
| Dimension | Measurement |
|---|---|
| Detection quality | Precision, recall, PR-AUC |
| False alarms | False-positive rate |
| Probability quality | Brier score, calibration error |
| Performance | p50/p95/p99 latency |
| Cost | Cost per 1,000 evaluations |
| Robustness | Behavior under model/version changes |
| Privacy | Data exposed to the external assessment provider |
| Reproducibility | Recorded decision playback and verification |
An independent Jev evaluation is also relevant:
Evaluating and Benchmarking the System One Model Jev
The authors benchmark Jev across 37 datasets and report results that depend on the task and, for some binary decisions, the chosen threshold.
That's a useful reminder that structured probabilities still require careful evaluation.
For a research prototype, I'd begin with offline shadow-mode assessment rather than live enforcement.
7. A question for the developer community
I think we're reaching a point where recording only an AI agent's final output will not be enough for many applications.
We need better ways to inspect not just what an agent did, but also the recorded assessments and authorization decisions surrounding its actions.
I'm particularly interested in feedback on three questions:
1. What should a verifiable AI decision receipt contain?
Would you preserve complete inputs, hashes, model versions, policy decisions, confidence distributions, or something else?
2. Where would this be most useful?
AI coding agents, autonomous infrastructure operations, financial workflows, multi-agent systems, scientific pipelines, or another area?
3. Would you start with historical decision analysis or real-time policy assistance?
My current preference is historical analysis and shadow-mode evaluation first, then carefully measured runtime integration.
I'd be interested to hear how other developers would approach the problem, including reasons this architecture might not work.
Projects and research references
NovaFabric
- Website: https://novafabric.ai/
- Open-source repository: https://github.com/MSKazemi/novafabric
- Documentation: https://novafabric.ai/docs/
- Archived software release: https://zenodo.org/records/23652607
Research papers
- NovaFabric preprint: https://arxiv.org/abs/2609.12582
- Independent Jev benchmarking paper: https://arxiv.org/abs/2609.37647
Jev
- TypeSafe AI introduction: https://typesafe.ai/blog/introducing-system-one-models-and-jev
- Documentation: https://docs.typesafe.ai/introduction
About the author
I'm Mohsen Seyedkazemi Ardebili, working on AI-agent systems, reproducibility, infrastructure, and verifiable execution evidence.
GitHub: https://github.com/MSKazemi
AI disclosure: This article was drafted with AI assistance. It presents a research proposal, not a completed integration or experimental result.
Top comments (0)