Attestly: Generate EU AI Act Annex IV docs from your agents’ operational traces

From AI Agent Traces to EU AI Act Annex IV Documentation: A More Evidence-Driven Approach

AI agents are becoming more capable, but documenting how they operate is becoming a challenge of its own.

An agent might invoke several tools, call multiple models, retry failed operations, hand control to a human, and interact with services maintained by other teams. Over time, its behavior changes as prompts, models, integrations and deployments evolve.

Much of this activity is already recorded in operational traces.

Yet when a team needs to prepare technical documentation for the EU AI Act, it may still have to reconstruct what happened from raw logs, engineering notes, tickets, configuration files and conversations with developers.

That creates a gap between the evidence generated during normal operations and the documentation needed to explain how an AI system actually behaves.

Attestly is built to help close that gap by turning AI agent runtime traces into structured, evidence-linked EU AI Act technical documentation.

Website: https://www.attestly.online/

1. Why documenting AI agents is different

Traditional software documentation often describes architecture, interfaces, dependencies and expected behavior.

AI agents introduce another layer of complexity: their operational behavior can involve sequences of decisions and actions that are difficult to describe accurately using source code alone.

Consider an AI agent that processes customer refund requests.

During one execution, it might:

1. Receive a customer request.
2. Invoke a language model to interpret the request.
3. Call an internal customer database.
4. Invoke a refund API.
5. Encounter a timeout and retry the operation.
6. Escalate the request to a human reviewer.
7. Continue after approval.
8. Record the final result.

A source-code review may reveal that the agent has access to the database and refund API. It may identify the model integration and the available approval mechanism.

But that doesn't establish which tools were actually called during a particular execution, whether the API failed, whether a human intervened, or what happened after the intervention.

Those details belong to the system's runtime history.

For teams that need to document deployed AI systems, the distinction between what software could do and what it actually did is important.

1. What EU AI Act Annex IV requires

Annex IV of Regulation (EU) 2024/1689 specifies technical documentation requirements for high-risk AI systems. Article 11 establishes the associated technical documentation obligation, subject to the regulation's scope and applicable provisions.

The documentation covers considerably more than operational logs.

Depending on the system and the applicable requirements, it includes:

• A general description of the system, its intended purpose, provider and versions.
• Information about its design, development process and architecture.
• Details of relevant components, integrations and computational resources.
• Information about monitoring, operation, capabilities and limitations.
• Performance metrics, testing and validation.
• Risk-management measures.
• Relevant changes throughout the system's lifecycle.
• Applicable conformity and post-market monitoring information.

A significant portion of this information may already exist in engineering documentation, model evaluations, risk assessments, deployment records and other internal systems.

Runtime traces contribute another valuable source of evidence, particularly for understanding actual execution behavior, tool usage, errors, human intervention and changes observed across deployments.

The practical challenge is connecting these different sources to the appropriate documentation requirements without confusing observed evidence with assumptions.

Official regulation: https://eur-lex.europa.eu/eli/reg/2024/1689/oj

1. Why runtime traces are useful evidence

Modern AI applications frequently emit structured execution data through observability tools and agent frameworks.

Depending on the instrumentation, traces can capture events such as:

• Model invocations and associated execution metadata.
• Tool calls, their status and their sequence.
• Errors, timeouts and retries.
• Human approvals, overrides and other interventions.
• Interactions with external APIs or services.
• Deployment and version changes, when those events are instrumented.
• Timing and other operational measurements.

These events can provide concrete evidence about system behavior.

For example, a trace recording a failed API request followed by a successful retry is more specific than a generic statement that the system supports retry handling.

Likewise, an event recording a human approval can support a description of a particular intervention, provided the event contains sufficient context and its provenance can be established.

There are limitations, however. A trace can only establish what was captured. Missing instrumentation, incomplete context or inaccurate event metadata can leave important questions unanswered.

A model invocation trace does not, by itself, establish model accuracy. A recorded approval does not prove that every required human-oversight measure is effective. A deployment event does not fully explain the rationale behind a design change.

The value comes from using operational evidence appropriately, not treating every log entry as proof of regulatory compliance.

1. How Attestly turns trace events into documentation

Attestly is designed around a runtime-evidence-first workflow.

Step 1: Ingest existing execution data

Instead of requiring teams to begin with a blank compliance template, Attestly accepts trace data from supported sources, including:

• OpenTelemetry
• LangSmith
• AgentOps
• MCP-related logs
• Pre-normalized JSON

This approach is useful for teams working with different agent frameworks or combining their own code with third-party tools.

Generic JSON support also provides a way to work with trace data exported from systems that do not have a dedicated native integration, provided the data is normalized into a supported structure.

Step 2: Normalize events into a usable structure

Operational data from different systems does not necessarily use the same event names, fields or metadata conventions.

A tool call in one tracing system may be represented differently from a corresponding event in another.

Attestly's workflow structures events such as model calls, tool invocations, human interventions, errors and deployment changes so that relevant information can be associated with documentation requirements.

Normalization is important because compliance documentation needs consistent descriptions rather than a collection of unrelated provider-specific log formats.

It also makes it easier to reason about sequences of events across a workflow.

Step 3: Map evidence to documentation requirements

The next step is to connect relevant trace events to the sections they can support.

For example:

• Tool-call events can help describe operational interactions and system behavior.
• Error and retry events can contribute evidence about observed failures and operational monitoring.
• Human-intervention events can support descriptions of recorded oversight activity.
• Deployment and version events can contribute to a record of changes over time.

These mappings provide a starting point for drafting documentation, but they do not eliminate the need for other evidence sources.

Information about intended purpose, architecture, training data, design rationale, testing methodology and risk acceptance may need to come from system owners, engineering documentation, evaluation reports or other records.

A reliable workflow must distinguish what the traces establish from what still needs to be supplied or verified.

Step 4: Generate evidence-linked documentation

Attestly uses the mapped evidence to draft structured documentation.

A central feature is traceability: generated sentences can link back to the specific trace events used to support them.

This changes how a reviewer can examine a draft.

Instead of seeing a statement and having to search through raw logs to discover its origin, the reviewer can follow the evidence link and inspect the supporting event.

The result is a documentation draft with a clearer relationship between claims and their sources.

Step 5: Identify missing information and require human approval

Not every Annex IV requirement can be answered using runtime traces.

Attestly distinguishes between information marked as AI-generated, information provided by the user, and information that is missing.

This distinction matters because an incomplete evidence set should produce an identifiable gap, not a fabricated answer.

Every generated section must be reviewed, edited or rejected by a person before it is approved for export. The product's exported Word document includes requirement-level review status and supporting evidence references.

The aim is to reduce the manual work involved in preparing documentation while retaining human responsibility for its accuracy and completeness.

1. An illustrative example: documenting a refund agent

Imagine a refund agent produces the following normalized events:

[
{
"event": "tool_call",
"tool": "refund_api",
"id": "a91f02c1",
"status": "success"
},
{
"event": "human_intervention",
"id": "a91f02c2",
"action": "approved"
},
{
"event": "error",
"id": "a91f02c3",
"message": "timeout on retry 2"
},
{
"event": "deployment_change",
"id": "a91f02c4",
"from": "v1.4",
"to": "v1.5"
}
]

This is an illustrative example of the kind of normalized event data used to explain the workflow, not a claim about a real production incident.

A documentation draft might summarize the observed sequence:

"The agent invoked the refund API, encountered a timeout during a retry, and recorded a human approval. The system's deployment version subsequently changed from v1.4 to v1.5."

Each part of that statement should be supported by the relevant event, with the sequence and timing checked before the text is finalized.

The reviewer can then ask more specific questions:

• Was the human approval required by the intended workflow?
• Does the trace contain enough information to establish when the approval occurred?
• Is the timeout associated with the same execution as the refund operation?
• What changed between the two deployment versions?
• Is there separate evidence showing that the relevant risk controls were tested?

Some questions may be answerable from traces. Others require additional records or input from the engineering team.

That is precisely why evidence-linked drafting is more useful than automatically generating a polished narrative without showing where its claims came from.

1. Why this is different from static code scanning

Static code analysis and runtime observability solve different problems.

A code scanner examines source code and can identify dependencies, integrations, potentially risky patterns and parts of the implementation.

Runtime traces record observed activity from instrumented executions.

Question| Static code analysis| Runtime traces
Which SDKs or libraries are imported?| Useful| Not its primary purpose
Which tools were invoked in a particular execution?| Cannot establish this from code alone| Useful when instrumented
Which calls failed or timed out?| Can identify some possible failure paths| Records observed failures
Was a human intervention recorded?| May identify the available mechanism| Can record actual intervention events
What changed between deployments?| Can compare code versions| Can capture deployment events when instrumented
What was the intended purpose of the system?| May provide clues| Usually requires other sources

Neither method provides a complete account of regulatory compliance on its own.

Attestly focuses on the narrower problem of turning runtime evidence into structured, traceable documentation. It is not intended to replace source-code analysis, a general governance platform, legal review or a complete risk-management process.

1. Evidence provenance matters as much as the generated text

AI-generated documentation creates a particular risk: plausible language can conceal uncertainty.

A paragraph may sound technically accurate without being supported by the available evidence.

For regulatory documentation, that is a serious weakness.

A useful system needs to make it possible to distinguish:

Observed evidence: What a trace or other source actually records.

User-provided information: Details supplied by someone responsible for the system.

Generated interpretation: A draft description derived from the available information.

Missing evidence: Information that has not been supplied or cannot be established from the sources available.

Attestly is designed to preserve these distinctions and link generated claims to supporting trace events.

The website also describes approval records as cryptographically hashed and evidence trails as tamper-evident. This helps establish whether recorded approval data has been altered, but it does not independently prove that the original evidence was complete, accurate or legally sufficient.

Provenance improves reviewability. It does not remove the need for verification.

1. Keeping documentation useful as systems change

Technical documentation can become outdated when an agent's model, tools, prompts, dependencies or deployment configuration changes.

A document prepared before launch may no longer accurately describe a system several months later.

Operational traces can help teams identify changes in observed behavior, while deployment events and version metadata can provide context for those changes.

Attestly is designed to support ongoing documentation from runtime evidence rather than treating documentation as a one-time exercise.

However, not every change can be understood from traces alone. A new deployment version may tell a reviewer that a change occurred without explaining the design rationale, the validation performed or the associated risk assessment.

Those details still need to be captured from appropriate sources.

The objective is to make documentation easier to maintain as the system evolves, not to assume that logging automatically keeps every Annex IV requirement satisfied.

1. Data handling and integration considerations

Teams evaluating an evidence-processing tool should also understand how their operational data is handled.

According to Attestly's published description, trace data is stored per organization with row-level database access controls. Only the specific events linked as evidence for a documentation section are sent to the AI model used to draft that section.

This is a relevant design detail for teams assessing how sensitive execution records are processed.

It is still important to review the product's privacy documentation and determine whether the traces being uploaded contain personal data, credentials, customer information or confidential business details. Instrumentation and redaction practices remain the responsibility of the system operator.

Teams should evaluate these considerations before connecting production data to any external service.

1. Who is Attestly designed for?

Attestly is aimed at teams that operate AI agents and need a more structured way to prepare technical documentation.

AI startups can use the workflow to build a documentation draft without manually reconstructing every operational detail from the beginning.

Enterprise AI teams can use evidence-linked documentation as a starting point for reviewing systems operating across multiple tools and services.

Compliance and risk professionals can inspect generated claims alongside their supporting events and identify gaps that require further evidence.

AI governance consultancies can use trace-based drafts as a starting point when preparing documentation for multiple systems or clients.

The usefulness of the workflow depends on the quality of the available traces, the completeness of other required information and the review process adopted by the organization.

1. Trying the workflow

Attestly provides a free tier covering one AI system and ten lifetime documentation generations.

The free tier is intended to let users explore the workflow, connect supported trace data and evaluate the resulting documentation before deciding whether a paid plan is appropriate.

There is also a free EU AI Act risk checker for teams that need an initial, directional assessment of whether their system may fall into a high-risk category.

Neither the risk checker nor the generated documents constitute legal advice or a guarantee of regulatory compliance. The actual classification and obligations depend on the specific system, its intended purpose and the applicable legal provisions.

You can explore the platform at:

https://www.attestly.online/

Conclusion

AI agents already generate operational records that can help explain how they behave. The challenge is making those records useful for technical documentation without losing their context, provenance or limitations.

Attestly approaches that problem by ingesting runtime traces, mapping relevant events to documentation requirements, drafting evidence-linked sections and making missing information visible for human review.

The broader principle is worth considering: documentation should not merely describe what a team believes its AI system does. Wherever possible, it should connect those descriptions to evidence that can be inspected and verified.

For teams working on AI systems subject to the EU AI Act, that approach offers a practical starting point for making documentation more traceable and easier to review.

Learn more: https://www.attestly.online/

Discussion: If you operate AI agents in production, how do you currently maintain technical documentation as models, tools and deployments change? Do you rely on manual reviews, internal observability pipelines, compliance platforms, or a combination of these approaches?