DEV Community: Swish

Your AI Agent Already Generates the Evidence. Why Reconstruct It Manually?

Swish — Sun, 25 Oct 2026 12:41:53 +0000

AI agents leave traces behind.

They call tools, invoke models, encounter errors, receive human intervention and change behavior across deployments. Yet when teams need to document how an AI system operates, much of that evidence still has to be reconstructed manually.

Someone reviews logs, interviews engineers, searches through tickets and tries to turn operational history into structured technical documentation.

I think there is a better way to approach this problem.

That's the idea behind "Attestly" (https://www.attestly.online/).

The gap between running an AI agent and documenting it

Building an AI system and documenting its behavior are two different tasks.

An agent might successfully complete a workflow, encounter a timeout, retry an operation and eventually require human approval. Those events may already exist in the system's execution traces.

But operational logs aren't the same as organized compliance documentation.

Teams still need to identify relevant evidence, map it to specific documentation requirements, explain what happened and identify what information is missing.

For teams working with the EU AI Act, this becomes particularly important when preparing technical documentation for high-risk AI systems.

Annex IV covers areas including system design, intended purpose, monitoring, validation and risk management. Keeping that documentation accurate as a system evolves can become a substantial ongoing task.

What Attestly does differently

Attestly takes a runtime-evidence-first approach.

Instead of relying only on source-code analysis or manually completed templates, it ingests execution traces that AI systems already produce and uses them to draft structured EU AI Act technical documentation.

Supported inputs include:

OpenTelemetry traces
LangSmith runs
AgentOps sessions
MCP-related logs
Pre-normalized JSON trace data

The system structures events such as tool calls, model calls, errors, human interventions and deployment changes, then maps relevant evidence to documentation requirements.

The important detail is traceability.

Generated documentation sections can link back to the specific events that support them. If evidence is missing, that gap should be identified rather than filled with plausible-sounding assumptions.

From trace events to a documentation draft

Consider an illustrative AI agent workflow.

An agent invokes a refund API. The request times out on a retry. A human reviewer subsequently approves the action.

Those events can tell a more useful story than a generic description saying the system supports automated refunds.

They show what the agent did, what went wrong and where a person intervened.

Attestly uses this type of operational evidence to draft relevant documentation sections, including monitoring measures and risk-management summaries.

The generated material is not automatically treated as final. A human must review, edit or reject each section before export.

This distinction matters: automation can reduce the work required to prepare documentation, but it should not remove accountability for what gets approved.

Why runtime traces matter

Source-code scanning has a valuable role in understanding an application. It can identify libraries, SDKs and integrations.

However, code alone cannot establish every detail of what happened during a particular execution.

It may not reveal which tools an agent actually invoked, when a human overrode a decision, which calls failed or how behavior changed between deployments.

Runtime traces provide another layer of evidence.

Attestly focuses specifically on turning that evidence into structured, reviewable documentation rather than attempting to replace every part of an organization's AI governance or compliance process.

Who might find this useful?

Attestly is designed for teams such as:

AI startups preparing documentation for systems deployed to EU customers.
Enterprise teams operating multiple AI agents.
Compliance and risk professionals who currently reconstruct system behavior manually.
AI governance consultants preparing documentation for multiple clients.

The goal is to give these teams an evidence-linked starting point rather than requiring them to begin every document from a blank template.

A practical starting point

Attestly offers a free tier covering one AI system and ten lifetime documentation generations. You can connect supported trace data, explore the generated documentation and evaluate whether the workflow fits your needs.

There is also a free EU AI Act risk checker for teams that are still trying to understand how their system may be classified.

Attestly does not provide legal advice or guarantee regulatory compliance. Generated documentation requires human review, and organizations remain responsible for determining their actual obligations.

Explore Attestly

Website: https://www.attestly.online/

The broader idea is simple: if AI systems already produce operational evidence, we should be able to use that evidence more effectively when documenting how those systems behave.

I'd be interested in hearing from engineers and compliance professionals working with AI agents:

How are you currently turning runtime logs and execution traces into technical documentation, and which part of that process takes the most time?

I Built EOBCheck to Help People Understand Medical Bills and Insurance Statements

Swish — Sun, 25 Oct 2026 12:38:06 +0000

Medical bills can be confusing, especially when the amount a healthcare provider says you owe doesn't match the amount your health insurer says is your responsibility.

For someone trying to understand what happened, figuring out the difference can mean navigating insurance terminology, comparing statements and contacting a provider to ask questions.

That's the problem I built EOBCheck to help with.

Website: https://www.eobcheck.online/

The problem: Your medical bill and EOB don't always agree

After processing an insurance claim, your insurer sends an Explanation of Benefits (EOB). It describes the claim, the amount the insurer allows, what it paid and what it calculates as your responsibility.

An EOB is not itself a bill.

However, comparing it with the bill from your healthcare provider can sometimes raise questions.

For example, imagine your provider's bill says you owe $1,842, while your EOB lists your patient responsibility as $1,126.

That's a difference of $716.

There may be a legitimate explanation. Perhaps the claim was processed at a different time, multiple claims were involved, or a payment hasn't been reflected yet. But it's reasonable to want to understand the discrepancy before proceeding.

The difficulty is knowing where to start.

What is EOBCheck?

EOBCheck is a browser-based tool that compares medical billing figures with the corresponding insurance EOB.

The goal is to make the comparison easier to understand and help people prepare a clear question for their healthcare provider when something doesn't add up.

Here's how it works.

Enter your medical bill details

Start with the amount your provider says you owe. You can also enter the total billed amount and other relevant details.

Enter the figures from your EOB

Add the insurer's patient responsibility, allowed amount, insurance payment and other requested figures.

For the free check, you only need the main totals. Itemized details are optional.

Review the reconciliation

EOBCheck compares the amount owed on the provider's bill with the patient responsibility shown on the EOB.

It flags the difference and provides a plain-English explanation rather than expecting you to interpret every number yourself.

A flagged difference is not automatically proof of a billing error. It is a reason to investigate and ask questions.

Generate a dispute letter

EOBCheck prepares a letter based on the figures you entered. You can review and edit it before downloading or sending it to your provider.

The purpose is to make it easier to ask for a written explanation or correction when appropriate.

Privacy was an important part of the design

Medical billing information is sensitive. People should be able to examine their numbers without unnecessarily sharing them with another service.

EOBCheck processes the figures entered into its reconciliation tool locally in the browser. The free tool does not transmit those entries to a server.

The paid document-processing features also process uploaded bill and EOB files locally in the browser.

There's no account required to start the free reconciliation.

I wanted the experience to be straightforward: bring your documents, enter the relevant information, review the results and decide what to do next.

What can you do with EOBCheck?

The product currently offers different options depending on the complexity of the billing situation.

Free: Compare one bill with one EOB, review the reconciliation flags and generate a dispute letter.
Full Report: Get itemized reconciliation, a Good Faith Estimate check, document-assisted amount entry and a more detailed editable letter.
Complex Bill: Reconcile multiple bills or providers and produce a combined variance summary with separate letters where needed.

For uninsured or self-pay patients, EOBCheck also includes a check related to the federal $400 Good Faith Estimate dispute threshold. Meeting that threshold alone does not establish eligibility; additional requirements apply.

The intention is to provide a useful starting point for understanding billing discrepancies, not to replace professional advice or guarantee that a bill will be reduced.

Why I built it

I wanted to make a frustrating administrative task more manageable.

People shouldn't have to become experts in insurance terminology just to compare two statements and identify a question worth asking.

A simple comparison, a clear explanation and a letter they can edit may help them approach that conversation with more information.

That's the idea behind EOBCheck: make medical billing reconciliation more accessible, keep the process simple and give people a practical next step when their numbers don't match.

The product is available now at:

https://www.eobcheck.online/

You can try the free reconciliation without creating an account.

I'd be interested in hearing from people who have dealt with medical billing discrepancies: What was the hardest part—understanding the EOB, finding the discrepancy, or getting a clear answer from the provider?

I Built Kiver: A Discovery Marketplace for Telegram Bots and Mini Apps

Swish — Sun, 18 Oct 2026 05:03:23 +0000

Building a Telegram bot is one thing. Getting people to discover it is another.

That problem is what led me to build Kiver.

"https://www.getkiver.com" (https://www.getkiver.com)

The problem I wanted to solve

Telegram has an ecosystem of bots and Mini Apps for almost everything: AI assistants, productivity tools, finance, education, gaming, automation and more.

Developers spend time building these products, improving their features and trying to attract users. But once a bot is ready, discovery becomes a challenge of its own.

A Telegram link is useful, but it doesn't always give a product the broader visibility it needs. Potential users may not know the bot exists, and developers need more ways to put their work in front of relevant audiences.

I wanted to build something that could help bridge that gap.

What is Kiver?

Kiver is a discovery marketplace for Telegram bots and Mini Apps.

The idea is straightforward: help users find useful Telegram tools and give makers a public place to showcase what they've built.

On Kiver, users can:

Browse Telegram tools across different categories.
Search for bots by name, username or use case.
Explore individual product pages.
Upvote tools they find useful.
Open listed bots directly in Telegram.

For developers and makers, Kiver provides a way to turn a Telegram link into a dedicated, shareable product listing.

Giving Telegram tools a public home

One part of Kiver I'm particularly interested in is discoverability beyond Telegram itself.

A maker can submit a public Telegram link, review the listing details, select a category and publish a product page.

That page gives the tool a dedicated URL that can be shared in communities, included in documentation or discovered through search engines.

The goal is to create another discovery channel for Telegram products instead of relying entirely on someone already knowing a bot's username or receiving its link from another person.

Of course, a public page doesn't guarantee search rankings or new users. The challenge is building a useful directory that people actually want to visit.

I'm still growing it

Kiver is an early-stage project, and I'm starting to see traffic coming to the website.

It's not massive traffic, and I'm not going to pretend it is. But seeing people visit a product you're building is useful feedback. It gives me a reason to keep improving the experience, expanding the directory and thinking more carefully about how users discover and evaluate tools.

The next challenge is turning visits into something more meaningful: people finding useful products, makers getting discovered, and visitors returning because the directory helps them.

That's the kind of growth I want to build toward.

What I'd like to improve

There are several areas I'm interested in developing further:

Better discovery: Helping users find relevant tools without having to know their names beforehand.
More useful listings: Making it easier to understand what a bot does before opening it.
More makers: Bringing more developers into the directory so users have a wider selection of tools.
A stronger community: Making product feedback and voting more useful for both users and builders.

The important part is making the directory valuable to both sides. More listings are useful, but only if people can find tools that genuinely solve their problems.

Try Kiver

If you've built a Telegram bot or Mini App, you can explore the directory and submit your own project.

Website: https://www.getkiver.com

Browse the directory, see how listings work, or launch your own Telegram tool on the platform.

I'm building Kiver in public and would appreciate feedback from developers, indie hackers and anyone working in the Telegram ecosystem.

If you build Telegram tools, what is the biggest challenge you've faced when trying to get your first users?

I'd be interested to hear what has worked for you.

Attestly: Generate EU AI Act Annex IV docs from your agents’ operational traces

Swish — Sat, 12 Sep 2026 07:14:34 +0000

From AI Agent Traces to EU AI Act Annex IV Documentation: A More Evidence-Driven Approach

AI agents are becoming more capable, but documenting how they operate is becoming a challenge of its own.

An agent might invoke several tools, call multiple models, retry failed operations, hand control to a human, and interact with services maintained by other teams. Over time, its behavior changes as prompts, models, integrations and deployments evolve.

Much of this activity is already recorded in operational traces.

Yet when a team needs to prepare technical documentation for the EU AI Act, it may still have to reconstruct what happened from raw logs, engineering notes, tickets, configuration files and conversations with developers.

That creates a gap between the evidence generated during normal operations and the documentation needed to explain how an AI system actually behaves.

Attestly is built to help close that gap by turning AI agent runtime traces into structured, evidence-linked EU AI Act technical documentation.

Website: https://www.attestly.online/

Why documenting AI agents is different

Traditional software documentation often describes architecture, interfaces, dependencies and expected behavior.

AI agents introduce another layer of complexity: their operational behavior can involve sequences of decisions and actions that are difficult to describe accurately using source code alone.

Consider an AI agent that processes customer refund requests.

During one execution, it might:

Receive a customer request.
Invoke a language model to interpret the request.
Call an internal customer database.
Invoke a refund API.
Encounter a timeout and retry the operation.
Escalate the request to a human reviewer.
Continue after approval.
Record the final result.

A source-code review may reveal that the agent has access to the database and refund API. It may identify the model integration and the available approval mechanism.

But that doesn't establish which tools were actually called during a particular execution, whether the API failed, whether a human intervened, or what happened after the intervention.

Those details belong to the system's runtime history.

For teams that need to document deployed AI systems, the distinction between what software could do and what it actually did is important.

What EU AI Act Annex IV requires

Annex IV of Regulation (EU) 2024/1689 specifies technical documentation requirements for high-risk AI systems. Article 11 establishes the associated technical documentation obligation, subject to the regulation's scope and applicable provisions.

The documentation covers considerably more than operational logs.

Depending on the system and the applicable requirements, it includes:

A general description of the system, its intended purpose, provider and versions.
Information about its design, development process and architecture.
Details of relevant components, integrations and computational resources.
Information about monitoring, operation, capabilities and limitations.
Performance metrics, testing and validation.
Risk-management measures.
Relevant changes throughout the system's lifecycle.
Applicable conformity and post-market monitoring information.

A significant portion of this information may already exist in engineering documentation, model evaluations, risk assessments, deployment records and other internal systems.

Runtime traces contribute another valuable source of evidence, particularly for understanding actual execution behavior, tool usage, errors, human intervention and changes observed across deployments.

The practical challenge is connecting these different sources to the appropriate documentation requirements without confusing observed evidence with assumptions.

Official regulation: https://eur-lex.europa.eu/eli/reg/2024/1689/oj

Why runtime traces are useful evidence

Modern AI applications frequently emit structured execution data through observability tools and agent frameworks.

Depending on the instrumentation, traces can capture events such as:

Model invocations and associated execution metadata.
Tool calls, their status and their sequence.
Errors, timeouts and retries.
Human approvals, overrides and other interventions.
Interactions with external APIs or services.
Deployment and version changes, when those events are instrumented.
Timing and other operational measurements.

These events can provide concrete evidence about system behavior.

For example, a trace recording a failed API request followed by a successful retry is more specific than a generic statement that the system supports retry handling.

Likewise, an event recording a human approval can support a description of a particular intervention, provided the event contains sufficient context and its provenance can be established.

There are limitations, however. A trace can only establish what was captured. Missing instrumentation, incomplete context or inaccurate event metadata can leave important questions unanswered.

A model invocation trace does not, by itself, establish model accuracy. A recorded approval does not prove that every required human-oversight measure is effective. A deployment event does not fully explain the rationale behind a design change.

The value comes from using operational evidence appropriately, not treating every log entry as proof of regulatory compliance.

How Attestly turns trace events into documentation

Attestly is designed around a runtime-evidence-first workflow.

Step 1: Ingest existing execution data

Instead of requiring teams to begin with a blank compliance template, Attestly accepts trace data from supported sources, including:

OpenTelemetry
LangSmith
AgentOps
MCP-related logs
Pre-normalized JSON

This approach is useful for teams working with different agent frameworks or combining their own code with third-party tools.

Generic JSON support also provides a way to work with trace data exported from systems that do not have a dedicated native integration, provided the data is normalized into a supported structure.

Step 2: Normalize events into a usable structure

Operational data from different systems does not necessarily use the same event names, fields or metadata conventions.

A tool call in one tracing system may be represented differently from a corresponding event in another.

Attestly's workflow structures events such as model calls, tool invocations, human interventions, errors and deployment changes so that relevant information can be associated with documentation requirements.

Normalization is important because compliance documentation needs consistent descriptions rather than a collection of unrelated provider-specific log formats.

It also makes it easier to reason about sequences of events across a workflow.

Step 3: Map evidence to documentation requirements

The next step is to connect relevant trace events to the sections they can support.

For example:

Tool-call events can help describe operational interactions and system behavior.
Error and retry events can contribute evidence about observed failures and operational monitoring.
Human-intervention events can support descriptions of recorded oversight activity.
Deployment and version events can contribute to a record of changes over time.

These mappings provide a starting point for drafting documentation, but they do not eliminate the need for other evidence sources.

Information about intended purpose, architecture, training data, design rationale, testing methodology and risk acceptance may need to come from system owners, engineering documentation, evaluation reports or other records.

A reliable workflow must distinguish what the traces establish from what still needs to be supplied or verified.

Step 4: Generate evidence-linked documentation

Attestly uses the mapped evidence to draft structured documentation.

A central feature is traceability: generated sentences can link back to the specific trace events used to support them.

This changes how a reviewer can examine a draft.

Instead of seeing a statement and having to search through raw logs to discover its origin, the reviewer can follow the evidence link and inspect the supporting event.

The result is a documentation draft with a clearer relationship between claims and their sources.

Step 5: Identify missing information and require human approval

Not every Annex IV requirement can be answered using runtime traces.

Attestly distinguishes between information marked as AI-generated, information provided by the user, and information that is missing.

This distinction matters because an incomplete evidence set should produce an identifiable gap, not a fabricated answer.

Every generated section must be reviewed, edited or rejected by a person before it is approved for export. The product's exported Word document includes requirement-level review status and supporting evidence references.

The aim is to reduce the manual work involved in preparing documentation while retaining human responsibility for its accuracy and completeness.

An illustrative example: documenting a refund agent

Imagine a refund agent produces the following normalized events:

[
{
"event": "tool_call",
"tool": "refund_api",
"id": "a91f02c1",
"status": "success"
},
{
"event": "human_intervention",
"id": "a91f02c2",
"action": "approved"
},
{
"event": "error",
"id": "a91f02c3",
"message": "timeout on retry 2"
},
{
"event": "deployment_change",
"id": "a91f02c4",
"from": "v1.4",
"to": "v1.5"
}
]

This is an illustrative example of the kind of normalized event data used to explain the workflow, not a claim about a real production incident.

A documentation draft might summarize the observed sequence:

"The agent invoked the refund API, encountered a timeout during a retry, and recorded a human approval. The system's deployment version subsequently changed from v1.4 to v1.5."

Each part of that statement should be supported by the relevant event, with the sequence and timing checked before the text is finalized.

The reviewer can then ask more specific questions:

Was the human approval required by the intended workflow?
Does the trace contain enough information to establish when the approval occurred?
Is the timeout associated with the same execution as the refund operation?
What changed between the two deployment versions?
Is there separate evidence showing that the relevant risk controls were tested?

Some questions may be answerable from traces. Others require additional records or input from the engineering team.

That is precisely why evidence-linked drafting is more useful than automatically generating a polished narrative without showing where its claims came from.

Why this is different from static code scanning

Static code analysis and runtime observability solve different problems.

A code scanner examines source code and can identify dependencies, integrations, potentially risky patterns and parts of the implementation.

Runtime traces record observed activity from instrumented executions.

Question| Static code analysis| Runtime traces
Which SDKs or libraries are imported?| Useful| Not its primary purpose
Which tools were invoked in a particular execution?| Cannot establish this from code alone| Useful when instrumented
Which calls failed or timed out?| Can identify some possible failure paths| Records observed failures
Was a human intervention recorded?| May identify the available mechanism| Can record actual intervention events
What changed between deployments?| Can compare code versions| Can capture deployment events when instrumented
What was the intended purpose of the system?| May provide clues| Usually requires other sources

Neither method provides a complete account of regulatory compliance on its own.

Attestly focuses on the narrower problem of turning runtime evidence into structured, traceable documentation. It is not intended to replace source-code analysis, a general governance platform, legal review or a complete risk-management process.

Evidence provenance matters as much as the generated text

AI-generated documentation creates a particular risk: plausible language can conceal uncertainty.

A paragraph may sound technically accurate without being supported by the available evidence.

For regulatory documentation, that is a serious weakness.

A useful system needs to make it possible to distinguish:

Observed evidence: What a trace or other source actually records.

User-provided information: Details supplied by someone responsible for the system.

Generated interpretation: A draft description derived from the available information.

Missing evidence: Information that has not been supplied or cannot be established from the sources available.

Attestly is designed to preserve these distinctions and link generated claims to supporting trace events.

The website also describes approval records as cryptographically hashed and evidence trails as tamper-evident. This helps establish whether recorded approval data has been altered, but it does not independently prove that the original evidence was complete, accurate or legally sufficient.

Provenance improves reviewability. It does not remove the need for verification.

Keeping documentation useful as systems change

Technical documentation can become outdated when an agent's model, tools, prompts, dependencies or deployment configuration changes.

A document prepared before launch may no longer accurately describe a system several months later.

Operational traces can help teams identify changes in observed behavior, while deployment events and version metadata can provide context for those changes.

Attestly is designed to support ongoing documentation from runtime evidence rather than treating documentation as a one-time exercise.

However, not every change can be understood from traces alone. A new deployment version may tell a reviewer that a change occurred without explaining the design rationale, the validation performed or the associated risk assessment.

Those details still need to be captured from appropriate sources.

The objective is to make documentation easier to maintain as the system evolves, not to assume that logging automatically keeps every Annex IV requirement satisfied.

Data handling and integration considerations

Teams evaluating an evidence-processing tool should also understand how their operational data is handled.

According to Attestly's published description, trace data is stored per organization with row-level database access controls. Only the specific events linked as evidence for a documentation section are sent to the AI model used to draft that section.

This is a relevant design detail for teams assessing how sensitive execution records are processed.

It is still important to review the product's privacy documentation and determine whether the traces being uploaded contain personal data, credentials, customer information or confidential business details. Instrumentation and redaction practices remain the responsibility of the system operator.

Teams should evaluate these considerations before connecting production data to any external service.

Who is Attestly designed for?

Attestly is aimed at teams that operate AI agents and need a more structured way to prepare technical documentation.

AI startups can use the workflow to build a documentation draft without manually reconstructing every operational detail from the beginning.

Enterprise AI teams can use evidence-linked documentation as a starting point for reviewing systems operating across multiple tools and services.

Compliance and risk professionals can inspect generated claims alongside their supporting events and identify gaps that require further evidence.

AI governance consultancies can use trace-based drafts as a starting point when preparing documentation for multiple systems or clients.

The usefulness of the workflow depends on the quality of the available traces, the completeness of other required information and the review process adopted by the organization.

Trying the workflow

Attestly provides a free tier covering one AI system and ten lifetime documentation generations.

The free tier is intended to let users explore the workflow, connect supported trace data and evaluate the resulting documentation before deciding whether a paid plan is appropriate.

There is also a free EU AI Act risk checker for teams that need an initial, directional assessment of whether their system may fall into a high-risk category.

Neither the risk checker nor the generated documents constitute legal advice or a guarantee of regulatory compliance. The actual classification and obligations depend on the specific system, its intended purpose and the applicable legal provisions.

You can explore the platform at:

https://www.attestly.online/

Conclusion

AI agents already generate operational records that can help explain how they behave. The challenge is making those records useful for technical documentation without losing their context, provenance or limitations.

Attestly approaches that problem by ingesting runtime traces, mapping relevant events to documentation requirements, drafting evidence-linked sections and making missing information visible for human review.

The broader principle is worth considering: documentation should not merely describe what a team believes its AI system does. Wherever possible, it should connect those descriptions to evidence that can be inspected and verified.

For teams working on AI systems subject to the EU AI Act, that approach offers a practical starting point for making documentation more traceable and easier to review.

Learn more: https://www.attestly.online/

Discussion: If you operate AI agents in production, how do you currently maintain technical documentation as models, tools and deployments change? Do you rely on manual reviews, internal observability pipelines, compliance platforms, or a combination of these approaches?