DEV Community

Cover image for Your AI Audit Trail Proves Less Than You Think
Jb
Jb

Posted on AI-assisted

Your AI Audit Trail Proves Less Than You Think

AI teams increasingly describe their systems as
auditable, traceable, or even verified.

Usually, this means they have logs.

They may record prompts, model names, outputs, tool
calls, timestamps, policy evaluations, and user
identifiers. More mature teams add distributed
tracing, human approvals, and long-term retention.

These records are valuable. But they leave an
uncomfortable question unanswered:

Can someone outside the system verify that the
evidence has not changed?

That is the problem verifiable execution attempts
to solve. It turns selected execution data into a
portable, tamper-evident artifact that can be
checked independently.

But verifiable execution also has important limits.

A cryptographically valid record does not prove
that an AI provider ran a particular model. It does
not prove that the output was correct. It does not
prove that every relevant event was captured.

Understanding these boundaries is essential if we
want AI evidence that engineers, customers,
auditors, and governance teams can actually trust.

Table of Contents

  • What an AI audit trail normally contains

  • From logs to execution evidence

  • The most important trust boundary

  • What each evidence layer proves

  • Sealing and attestation are different

  • What trusted timestamps prove

  • AI Execution V1 and V2

  • Creating a minimal V2 execution record

  • Installation is not instrumentation

  • Tool calls and real-world actions

  • Policy evidence is not authorization

  • Confidential evidence is not encrypted evidence

  • Multi-step agents need workflow-level evidence

  • Questions governance teams should ask

  • A better meaning of verified AI

What an AI audit trail normally contains

An AI application may record:

  • User input
  • System instructions
  • Model and provider identifiers
  • Generation parameters
  • Retrieved documents
  • Tool calls
  • Policy checks
  • Model outputs
  • Human approvals
  • Downstream actions
  • Timestamps and execution identifiers

These records help engineers debug failures,
monitor performance, and reconstruct workflows.

The problem is that conventional logs are normally
controlled by the same organization whose system is
being reviewed.

Administrators may be able to edit, delete,
truncate, or replace them. An external reviewer
must trust both the records and the infrastructure
presenting them.

Logs therefore provide visibility, but visibility
is not automatically proof.

From logs to execution evidence

A stronger approach is to package the important
parts of an execution into a structured record and
calculate a cryptographic hash over its protected
content.

NexArt calls this artifact a Certified Execution
Record, or CER.

Conceptually, the process looks like this:

Captured execution

Structured snapshot

Canonical representation

Cryptographic hash

Portable execution record

Canonicalization is important because independent
verifiers need to hash exactly the same bytes.

Differences in JSON spacing, property order, or
serialization rules must not produce different
hashes for logically equivalent records.

A verifier can then:

  1. Validate the record against its expected schema.
  2. Recompute hashes for protected inputs and
    outputs.

  3. Reconstruct the protected certificate payload.

  4. Recompute the certificate hash.

  5. Compare the result with the stored value.

If a protected field changes, verification fails.

This creates tamper evidence.

It does not create truth.

The most important trust boundary

The producer is the application that captures the
execution.

That application supplies claims such as:

  • Which provider handled the request
  • Which model was requested
  • What input was sent
  • What output was received
  • Which tools were called
  • Which policy evaluation occurred
  • Whether the execution was advisory or operative

Cryptographic hashing protects those claims after
they have been recorded. It cannot determine
whether they were accurate when recorded.

This distinction matters enough to state directly:

Integrity proves consistency with the recorded
evidence. It does not prove that the recorded
evidence was truthful or complete.

Suppose an application creates this record:

{
    "provider": "example-provider",
    "model": "example-model",
    "input": {
      "message": "Should this refund be approved?"
    },
    "output": {
      "decision": "approve"
    },
    "executionRole": "advisory"
  }
Enter fullscreen mode Exit fullscreen mode

After sealing, a verifier may be able to establish
that none of those protected values changed.

The verifier still cannot establish, from the hash
alone, that:

  • The named provider received the request
  • The named model produced the output
  • The complete prompt was captured
  • The role was genuinely advisory
  • No relevant tool call was omitted
  • The refund was actually approved
  • The decision was correct

Those conclusions require additional evidence and
controls.

What each evidence layer proves

Instead of treating “verified” as one universal
state, it is safer to describe the exact claim
supported by each evidence layer.

Claim The record is internally
consistent
Required evidence Schema and hash
verification
What remains unproven Whether its contents are
truthful
───────────────────────────────────────────────────
Claim Protected fields have not
changed
Required evidence Hash verification and a
trusted reference
What remains unproven Whether the entire record
was replaced
───────────────────────────────────────────────────
Claim A party attested to the
record
Required evidence Valid signature and trusted
public key
What remains unproven Whether the recorded
execution really occurred
───────────────────────────────────────────────────
Claim The digest existed by a
particular time
Required evidence Trusted timestamp bound to
the digest
What remains unproven The exact execution time
───────────────────────────────────────────────────
Claim A policy result is bound
into the record
Required evidence Protected policy evidence
What remains unproven Whether the policy was
appropriate or enforced
───────────────────────────────────────────────────
Claim Every required execution
was captured
Required evidence Independent reconciliation
and capture controls
What remains unproven Completeness from hashing
alone
───────────────────────────────────────────────────
Claim Disclosed content matches
an earlier commitment
Required evidence Valid confidential opening
What remains unproven Whether the hidden content
was truthful

This vocabulary is less impressive than saying “the
AI is verified.”

It is also much more useful.

Sealing and attestation are different

A locally sealed record contains a cryptographic
identity derived from its protected content.

Attestation introduces another party.

Application → Producer → Attestor → Verifier

The producer creates and hashes the record.

An attestor can inspect the submitted artifact and
issue a signed receipt for it. A verifier can later
check that signature using the attestor’s public
key.

A valid signature establishes that the
corresponding private key signed the receipt.

It does not automatically establish that the signer
should be trusted.

The relying organization must still decide:

  • Which attestors it accepts
  • How public keys are obtained
  • How keys are rotated or revoked
  • Whether historical keys remain trusted
  • Which receipt fields are protected
  • What happens when attestation is unavailable

Downloading a public key from an untrusted location
and successfully verifying its signature only
proves that the associated private key signed the
data. It does not establish the identity or
authority of the key owner.

What trusted timestamps prove

A local timestamp such as createdAt is part of the
producer’s record.

It is useful metadata, but it remains a producer
assertion.

A trusted external timestamp can provide stronger
evidence that a particular digest existed no later
than a stated time.

That still does not prove the exact moment at which
the AI execution occurred.

The distinction is:

Local timestamp:
“The producer says this happened at this time.”

Trusted external timestamp:
“This digest existed by this time under the
timestamp authority’s trust model.”

Timestamp verification should remain separate from
basic record verification.

A CER can have valid internal integrity while its
timestamp evidence is missing, unsupported,
untrusted, or invalid.

AI Execution V1 and V2

Version terminology is another easy source of
confusion.

NexArt separates several compatibility axes:

  • npm package version
  • CER record schema
  • Canonicalization protocol version
  • Transport package schema
  • Attestation receipt schema
  • External evidence-format version

For example, version 1.4.0 of an SDK does not imply
protocol version 1.4.0.

NexArt AI Execution V2 uses:

Bundle type: cer.ai.execution.v2
Protocol version: 1.3.1

V1 and V2 have separate producer and verifier APIs.

A V1 verifier should not guess how to interpret a
V2 record, and a V2 producer should not silently
downgrade to V1.

When dealing with evidence formats, failing closed
is safer than coercing an unknown record into a
familiar schema.

Creating a minimal V2 execution record

A local integration can explicitly create and
verify a V2 record:

import {
    certifyDecisionV2,
    verifyCerV2,
  } from "@nexart/ai-execution";

  const bundle = certifyDecisionV2({
    executionId: "example-v2",
    timestamp: "2026-09-08T10:00:00.000Z",
    createdAt: "2026-09-08T10:00:01.000Z",

    provider: "example-provider",
    model: "requested-model",

    input: {
      message: "Summarize this support case."
    },

    parameters: {
      temperature: 0,
      maxTokens: 200,
      topP: null,
      seed: null
    },

    output: {
      summary: "The customer requested a refund."
    },

    executionRole: "advisory",
    instruction: "Summarize the case in one
    sentence.",
    instructionKind: "system"
  });

  const result = verifyCerV2(bundle);

  if (!result.ok) {
    throw new Error(
      `${result.code}: ${result.errors.join("; ")}`
    );
  }
Enter fullscreen mode Exit fullscreen mode

This creates and verifies a local evidence
artifact.

It does not contact an AI provider or an
attestation service.

The application remains responsible for calling the
provider and supplying faithfully captured request
and response data.

Installation is not instrumentation

This is one of the most common implementation
mistakes.

Running:

npm install @nexart/ai-execution

does not capture anything.

The application must instrument the function that
actually performs the operation.

Wrapping manually reconstructed data after
execution creates evidence about that
reconstruction—not necessarily about the original
execution.

A reliable integration should cover:

  • Successful responses
  • Provider failures
  • Tool failures
  • Retries
  • Timeouts
  • Streaming responses
  • Human approval boundaries
  • Downstream side effects

It should also include negative tests.

Change one protected field and confirm that
verification fails. Use the wrong signing key.
Remove required evidence. Simulate the attestation
service being unavailable.

These tests help confirm that the failure paths are
genuinely connected to the application.

Tool calls and real-world actions

Agents become consequential when they move beyond
text generation and start using tools.

A wrapper can record:

  • Tool name
  • Supplied arguments
  • Returned result
  • Execution time
  • Errors
  • Tool-call identity

But wrappers have limits.

They may not observe hidden process state, network
activity inside a tool, or real-world side effects.

For example, an agent may produce a tool call
requesting a refund. The audit trail should
distinguish:

  1. The model proposed the call.
  2. The application authorized it.
  3. The tool invocation was attempted.
  4. The target system returned success.
  5. The refund transaction was recorded.

A model output containing "refund": "approved" is
not proof that money moved.

The target system’s transaction identifier,
receipt, or resulting object state may be needed to
support that claim.

Policy evidence is not authorization

A deterministic policy evaluator can produce
results such as:

PASS
FAIL
ERROR

Those states should remain distinct.

  • PASS means the supplied data satisfied the
    supplied rules.

  • FAIL means evaluation completed and a rule was
    not satisfied.

  • ERROR means evaluation could not complete
    reliably.

A PASS does not authorize an action by itself.

The surrounding application must implement the
operational gate.

This distinction becomes particularly important
when policy evaluation occurs after an action.
Binding a policy result into an execution record
proves that the result is part of the protected
evidence.

It does not prove that the policy prevented or
authorized anything.

Confidential evidence is not encrypted evidence

Some workflows cannot expose raw prompts, customer
information, or model outputs in a portable record.

One option is to replace selected values with
cryptographic commitments.

NexArt’s confidential mode uses salted, domain-
separated HMAC commitments and returns opening
material separately.

The public record can contain the commitment while
the plaintext remains private. Later, an authorized
party can present the plaintext and opening salt to
demonstrate that they match the earlier commitment.

This provides selective disclosure, but it is not
encryption.

It does not provide:

  • Plaintext recovery
  • Secret management
  • Automatic deletion
  • Anonymity
  • Protection for uncommitted metadata
  • Proof that the hidden content was truthful

Opening salts must be protected.

Public metadata such as timestamps, workflow names,
actor references, or model identifiers may still
reveal sensitive information.

Multi-step agents need workflow-level evidence

A single CER can represent one execution or
decision.

Agent workflows often involve many related steps.

Project Bundles can group multiple V1 and V2
execution records while preserving their explicit
sequence and relationships.

This gives reviewers a coherent artifact
containing:

  • Individual execution records
  • Step identifiers
  • Parent relationships
  • Sequence information
  • Project-level integrity

The bundle only covers the records included in it.

It cannot prove that omitted executions never
occurred.

Completeness requires operational controls such as:

  • Defining which events are in scope
  • Monitoring capture failures
  • Comparing records with an independent activity
    source

  • Detecting missing sequence numbers

  • Investigating evidence gaps

  • Preventing silent downgrade when evidence
    creation fails

Cryptography can protect captured evidence.

It cannot capture missing evidence.

Questions governance teams should ask

When evaluating an AI evidence system, asking “Does
it use cryptography?” is not enough.

Ask:

  1. Where is the real execution boundary?
  2. Which fields are protected?
  3. Who supplies those fields?
  4. How is capture completeness monitored?
  5. Can the evidence be exported?
  6. Can another party verify it independently?
  7. Which verifier supports each record schema?
  8. How are signing keys trusted and rotated?
  9. Are integrity, attestation, and timestamp results reported separately?
  10. What happens when evidence generation fails?
  11. How are confidential values and openings protected?
  12. Does policy evidence affect application control flow?
  13. Can reviewers distinguish proposed actions from completed side effects?
  14. Which external services remain in the trust chain?
  15. What exactly does a successful verification claim?

Good governance starts by making each claim narrow
enough to defend.

A better meaning of verified AI

“The AI is verified” compresses too many different
questions into one phrase.

A more defensible statement is:

The protected fields in this execution record
match their cryptographic identity, and the
attached evidence has been checked under the
relying organization’s stated trust policy.

That statement does not claim correctness,
fairness, provider execution, completeness, or
compliance.

It says what was checked.

As AI systems become more autonomous, the
distinction between visibility and evidence will
become increasingly important.

Logs help us operate systems.

Structured execution records help us preserve what
those systems reported.

Hashes make later changes detectable.

Signatures add claims from identifiable keys.

Timestamps add external time evidence.

Governance determines whether those layers are
sufficient for a particular decision.

The goal is not to make AI systems sound more
trustworthy than they are.

The goal is to make their evidence precise enough
that trust no longer depends on vague language.

———

NexArt creates portable, tamper-evident execution
records for AI systems, agents, and multi-step
workflows.

Learn more at nexart.io (https://nexart.io/) or
explore the NexArt documentation
(https://docs.nexart.io/).

Top comments (0)