AI agents leave traces behind.
They call tools, invoke models, encounter errors, receive human intervention and change behavior across deployments. Yet when teams need to document how an AI system operates, much of that evidence still has to be reconstructed manually.
Someone reviews logs, interviews engineers, searches through tickets and tries to turn operational history into structured technical documentation.
I think there is a better way to approach this problem.
That's the idea behind "Attestly" (https://www.attestly.online/).
The gap between running an AI agent and documenting it
Building an AI system and documenting its behavior are two different tasks.
An agent might successfully complete a workflow, encounter a timeout, retry an operation and eventually require human approval. Those events may already exist in the system's execution traces.
But operational logs aren't the same as organized compliance documentation.
Teams still need to identify relevant evidence, map it to specific documentation requirements, explain what happened and identify what information is missing.
For teams working with the EU AI Act, this becomes particularly important when preparing technical documentation for high-risk AI systems.
Annex IV covers areas including system design, intended purpose, monitoring, validation and risk management. Keeping that documentation accurate as a system evolves can become a substantial ongoing task.
What Attestly does differently
Attestly takes a runtime-evidence-first approach.
Instead of relying only on source-code analysis or manually completed templates, it ingests execution traces that AI systems already produce and uses them to draft structured EU AI Act technical documentation.
Supported inputs include:
- OpenTelemetry traces
- LangSmith runs
- AgentOps sessions
- MCP-related logs
- Pre-normalized JSON trace data
The system structures events such as tool calls, model calls, errors, human interventions and deployment changes, then maps relevant evidence to documentation requirements.
The important detail is traceability.
Generated documentation sections can link back to the specific events that support them. If evidence is missing, that gap should be identified rather than filled with plausible-sounding assumptions.
From trace events to a documentation draft
Consider an illustrative AI agent workflow.
An agent invokes a refund API. The request times out on a retry. A human reviewer subsequently approves the action.
Those events can tell a more useful story than a generic description saying the system supports automated refunds.
They show what the agent did, what went wrong and where a person intervened.
Attestly uses this type of operational evidence to draft relevant documentation sections, including monitoring measures and risk-management summaries.
The generated material is not automatically treated as final. A human must review, edit or reject each section before export.
This distinction matters: automation can reduce the work required to prepare documentation, but it should not remove accountability for what gets approved.
Why runtime traces matter
Source-code scanning has a valuable role in understanding an application. It can identify libraries, SDKs and integrations.
However, code alone cannot establish every detail of what happened during a particular execution.
It may not reveal which tools an agent actually invoked, when a human overrode a decision, which calls failed or how behavior changed between deployments.
Runtime traces provide another layer of evidence.
Attestly focuses specifically on turning that evidence into structured, reviewable documentation rather than attempting to replace every part of an organization's AI governance or compliance process.
Who might find this useful?
Attestly is designed for teams such as:
- AI startups preparing documentation for systems deployed to EU customers.
- Enterprise teams operating multiple AI agents.
- Compliance and risk professionals who currently reconstruct system behavior manually.
- AI governance consultants preparing documentation for multiple clients.
The goal is to give these teams an evidence-linked starting point rather than requiring them to begin every document from a blank template.
A practical starting point
Attestly offers a free tier covering one AI system and ten lifetime documentation generations. You can connect supported trace data, explore the generated documentation and evaluate whether the workflow fits your needs.
There is also a free EU AI Act risk checker for teams that are still trying to understand how their system may be classified.
Attestly does not provide legal advice or guarantee regulatory compliance. Generated documentation requires human review, and organizations remain responsible for determining their actual obligations.
Explore Attestly
Website: https://www.attestly.online/
The broader idea is simple: if AI systems already produce operational evidence, we should be able to use that evidence more effectively when documenting how those systems behave.
I'd be interested in hearing from engineers and compliance professionals working with AI agents:
How are you currently turning runtime logs and execution traces into technical documentation, and which part of that process takes the most time?
Top comments (0)