DEV Community: Guillaume Marchand

The coaching agent that can’t fake a citation, built on AWS (part 2 of 2)

Guillaume Marchand — Wed, 19 Aug 2026 14:42:41 +0000

This is part 2 of a two-part series. Part 1 turned a book into a citable graph on AWS: a repaired Markdown split by DoCO role, a chunk per book division, a Method_Model that keeps only what quotes the book word for word, and an MCP server that exposes 21 tools, 7 resources and 4 prompts against it. Two vocabulary points from part 1 carry into this one: a chunk is a division of the book (a chapter, a section) named by the table of contents, and a statement is a sentence the extraction model has written about the book, useful for search, never citable. This part builds the agents that read from that substrate, and the five mechanisms that stop them from inventing a citation.

How the agents are built

There are five families of agents, not five agents : assess, coach, writer, one agent per persona the book describes, and three generic reviewers. On a book with three personas, that gives nine actual agents. All families come out of a single function, build_agent, that takes the book's name, a few booleans, and returns a Agent from Strands ready to run. The booleans are what distinguishes one family from another: enforce_grounding decides whether the verification chain applies, retrieval whether the agent receives tools, allow_work_writes whether it can modify your files.

A second function, agent_wiring, decides the composition without building an agent. It makes review possible, and I return to it at the end of this chapter.

What a layer is

A Strands agent is built from four lists: a system prompt, tools, plugins and hooks. What I call a layer is an entry into one of those lists. agent_wiring fills them and returns the argument dictionary that Agent will receive as is:

return AgentWiring(
    kwargs={
        "system_prompt": system_prompt,
        "tools": [
            *(AGENT_TOOLS if retrieval else []),
            *(WORK_WRITE_TOOLS if allow_work_writes else []),
        ],
        "plugins": plugins,
        "hooks": [hook],
        "callback_handler": None,
    },
    skills=existing,
)

Six layers are possible, and no agent has all six. The seventh row in the table is not a layer I place: it is the SDK default, and I name it because what follows depends on it.

Layer Strands class Present when Skill AgentSkills A SKILL.md has been rendered for this book Steering Subclassed LLMSteeringHandler By default, unless disabled Verification GoalLoop enforce_grounding=True Memory MemoryManager A reader identifier is supplied Tools The book and document registries retrieval=True Evidence ledger Homegrown HookProvider Always (SDK default) SlidingWindowConversationManager Always, because none is passed

Steering: surveillance before a tool call

Steering is an SDK plugin. Each time the agent is about to call a tool, it submits the situation to a model with its own system prompt, which can wave it through or return an instruction the agent reads before acting. It is therefore preventive, where the output checks are corrective: they judge a finished answer.

Ours is deliberately narrow: it does not judge the quality of a finding, only whether it is sourced.

You keep an assessment honest about its sources. You judge only whether a claim is sourced, never whether it is good. […] Do not intervene on the substance of the assessment. The craft judgement belongs to the author and to the book, not to you. […] Never choose interrupt. This runs unattended, so there is no human to answer; guide is how you say something is wrong.

The last sentence is asked for in the prompt and enforced in the code, because a prompt is not a guarantee. The SDK offers three actions before a tool call: Proceed, which waves through and is the nominal path, Guide, which cancels the call and reinjects an instruction, and Interrupt, which suspends for human intervention. A subclass demotes the third into the second:

class NonInteractiveSteering(LLMSteeringHandler):
    async def steer_before_tool(self, **kwargs: Any) -> Any:
        action = await super().steer_before_tool(**kwargs)
        if isinstance(action, Interrupt):
            return Guide(reason=action.reason)
        return action

This subclass is required for two reasons. First, there is no human behind an MCP tool call. Second, an active interrupt makes the GoalLoop retry fail with a type error, because the invocation-end event carries a text string where the interrupt state expects a list of blocks.

Memory: distilled facts, not preserved turns

There is no Strands session management here, and the execution path does not need one: an MCP tool builds the agent, invokes it once, clears its memory and returns a document. The day coaching becomes conversational, this is the first mechanism to wire in.

What survives across executions is something else: distilled facts, through the MemoryManager plugin. The store deliberately omits add_messages (the method that would write the raw turns) to force the SDK to distil the conversation into facts with a model before writing. No memory search tool is exposed (search_tool_config=False), because an additional retrieval tool would compete with the citation discipline.

The skill: a retrieval manual, not a copy of the book

The obvious design is to inline the method into the skill. I tried it: inlining every element cost 93,000 characters and prevented assessment from completing. A references directory containing the same material was written, then never read.

The SKILL.md is therefore generated by render_skill, from the Method_Model, at the end of ingestion. It contains five things: a frontmatter, a tools table, an order of operations, a retrieval discipline, and a self-detection table. It contains no book content, only the names of the method steps.

The frontmatter is the part that acts on the SDK:

---
name: charpente-fr
description: How to interrogate <method> — the tools, the order, and the retrieval discipline.
  Activate when assessing, reviewing or coaching a work against this book.
allowed-tools: book_search book_semantic_search book_passages book_passage book_chapters book_status
metadata:
  book: charpente_fr
  book_id: charpente.fr
---

allowed-tools names the six book tools, excluding book_ask and the tools for your document. Beware of what this field actually does: the documentation marks it as experimental and not enforced at runtime. The field is a human-readable declaration of intent, not a restriction. If you rely on it as an access control, you have the wrong mechanism.

What enters the system prompt is a block listing the available skills with, for each, its name, its description and the path of its SKILL.md. The body of the file is not there. The description therefore says when to activate the skill, and the agent will read the rest if it decides to use it. The block is refreshed before every invocation.

The last part of the file is the most reusable. It lists concrete signals the agent can spot in its own draft (“I just wrote a sentence about the book without having called a tool”) and the corrective action to take. These are review rules that a model knows how to apply to its own text, whereas an abstract instruction dilutes fast.

Signal in your draft Corrective action A claim about the book with no retrieval behind it retrieve it, or remove the claim A quotation you have not read in a passage retrieve the passage, or drop the quotation marks A chapter title not obtained from book_chapters list the chapters, use the real title Advice that would fit any book it is generic, anchor it here or cut it "typically" or "generally" about the book the book says it or it does not

The five families, layer by layer

assess coach writer persona generic Skill the book's the book's none none none Steering yes yes no no no GoalLoop 2 attempts 2 attempts none none none Memory if reader known if reader known if reader known none none Tools book + document read same same + write none none Verification applied yes yes no no no Returns the assessment the revisions the prose a reaction a finding

The assess system prompt carries ten non-negotiable rules (not to be confused with the SOP: a SOP describes the steps to follow, these rules impose what the agent is not allowed to do). The coach has seven, the writer nine, the persona seven. Two of the ten assess rules are why a citation is a citation:

You MUST NOT state what the book requires, recommends or forbids without having retrieved it in this conversation. Recall is not retrieval, and your training data is not this book.

WHERE you put the book’s words in quotation marks, they MUST be the book’s words exactly, character for character. […] Do not shorten a sentence by ending it early at a comma. WHERE you drop anything from the middle, mark the gap with [...]. A tidied quotation is a misquotation.

coach receives a book-specific prompt, because the SOP generated for that book forms its opening. It does not receive your document a second time: the prompt carries a reference to the manifest, while the document and the assessment still enter the evidence ledger, since it is against the ledger that a citation is verified.

When the SOP contains more than five steps (which is the case for any serious book), coaching is delegated to sub-agents. An orchestrator written in code (a loop, not a model) cuts the worklist into slices of five and instantiates a fresh Agent per slice. Each sub-agent does its own retrievals, produces its revisions, and shares nothing with the others. Parallelism falls out naturally, three sub-agents run at once. On a book with 51 steps, 11 sub-agents produce 269,000 characters of coaching in ~20 minutes. A single agent, on the other hand, hits the model's output ceiling at 37,000 characters and does not finish. This is the sub-agent delegation pattern, tracked in the issue strands-agents/harness-sdk#911. The SDK already provides Graph and Swarm for multi-agent. Graph runs the ready nodes of one wave in parallel but fixes its topology at construction. Swarm chains sequential handoffs. Here, the number of sub-agents equals the number of worklist slices. It is decided at runtime from the book, so the code orchestrator stays a fit.

Figure 4: sub-agent delegation. The orchestrator, a loop in code, cuts the worklist into slices of five steps and instantiates a fresh-context agent per slice. The agents run in parallel, share nothing, and their outputs are concatenated.

writer is the only agent allowed to write, and the only one whose verification is disabled. Writing a scene is not a claim about the book, and a citation guardrail would reject a good scene. Its prompt opens by naming the relationship ("You are the hand, not the head") and its second rule protects your structure:

The heading is not yours. You rewrite what is under it […] do not renumber, retitle or re-level anything the author named.

Personas and generic reviewers: verifiable isolation

No persona agent definition exists in the code. Extraction produces the list of audiences the book describes, and a persona agent is a function call that fills a prompt template from one of those audiences. A book that describes no audience produces no persona. The first point of the prompt is what makes it useful:

React. Do not assess. “This violates the rule about X” is not something you would ever say.

Six further instructions follow: say where you dropped off and at which passage, say what it would take to say yes, stay in character, write in the language of the work, in the first person.

Three generic reviewers sit next to the personas (structure, clarity, audience fit) with static prompts and no dependency on the Method_Model. They therefore work on a book that was never distilled.

Both kinds of reviewer receive retrieval=False, which has two effects: the tool list is empty, and build_agent does not bind the graph. The book is therefore not just kept quiet in a persona's context. There is no tool through which to reach it. No prompt instruction carries the load, and this is verifiable by reading six lines rather than rereading a prompt. These reviewers are also not subject to the check chain: a reaction or a craft observation claims nothing about the book, and submitting them to it would make them fail for not having cited a book they are forbidden to read.

How to prove the agent really reads the book

Here is the measurement that made me build everything that follows. On an earlier version of the agent, one answer contained fourteen citations attributed to the book. Seven were absent from it. The agent had written them from memory, and the evidence ledger, the list of what it had actually retrieved, still claimed to hold all fourteen. The verifier, that is the code that confronts the citations in the answer with that ledger, therefore validated them. I call this measurement the seven-out-of-fourteen run.

The cause was not the model but the meeting of two mechanisms. Figure 5 shows them together: the ledger on the left, the check chain in the middle, the retry path on the right.

Figure 5: how a citation is verified before the answer leaves the agent. Three columns in English: the evidence ledger that classifies each tool result and pins the message that carries it, the five checks chained from cheapest to most expensive, and the retry path that reruns the agent at most twice when a check fails.

The two mechanisms that collide

The first is the conversation manager. I pass none, and that is the point: every agent therefore inherits the SDK default, a SlidingWindowConversationManager whose window is 40 messages. It intervenes through two paths. Routinely, at the end of every invocation, if the history exceeds the window, it calls reduce_context and trims the oldest messages. And when the model returns a ContextWindowOverflowException, the agent calls the same reduce_context passing it the exception. On that reactive path, the manager first tries to truncate the oldest tool results, and only cuts messages otherwise.

The oldest tool results: these are precisely the passages the agent retrieved at the start of its work. The mechanism meant to save the invocation is therefore exactly the one that removes the evidence.

The second is the evidence ledger, an instance of our GroundingHook class. Concretely, two Python lists in memory for the duration of a call: passages for what comes from the book, given for what comes from your document. It is registered as a hook provider with the agent and subscribes to two SDK events:

def register_hooks(self, registry: Any, **_kwargs: Any) -> None:
    from strands.hooks import AfterToolCallEvent, MessageAddedEvent

    registry.add_callback(AfterToolCallEvent, self._record)
    registry.add_callback(MessageAddedEvent, self._pin_evidence)

AfterToolCallEvent fires after every tool call: _record classifies the result into one of the two channels. The hook documentation is in the Strands Agents SDK guide.

The collision is here. The SlidingWindowConversationManager drops a tool result message from the conversation, but the ledger keeps its copy of the text. The ledger then vouches for a passage the model no longer sees. The model keeps writing, cites the book from memory, and the verifier accepts the citation because its ledger says it arrived through a tool.

Mechanism 3: the fix, pin the evidence

_pin_evidence is the other subscription. On every message added, it looks at whether that message carries a tool result the ledger has recorded, and if so it pins it:

from strands.agent.conversation_manager.compression.pin_message import pin_message
pin_message([message], 0)

The SlidingWindowConversationManager accepts a pin_first parameter, which protects the first messages in the window. It does not cover evidence that arrives mid-conversation, after the tool calls. pin_message fills that role, and it is the primitive that the SDK's own test suite uses to pin a message in the middle of the history. The import path goes through the submodule because the compression package does not expose a public alias. pin_message sets a flag in the message metadata, and reduce_context drops only unpinned messages. Trimming can therefore let any conversation turn go, and it keeps the pinned evidence.

The general principle, if you take only one line from this chapter: the component that decides “this is evidence” must be the one that protects it.

Two channels, because reading your document is not reading the book

The ledger holds two separate lists: passages for the graph tool results, given for what the agent read of your document. Both are citable, only one counts as evidence, passages answers retrieved_anything. Citing yourself can therefore never satisfy the check. A generic file-reading tool once figured among the evidence tools, and because the ledger indexes on the tool name, reading any file satisfied "the agent consulted the book". This separation closes the hole.

Mechanism 4: a verifier that costs nothing until it must

Verification is carried by GoalLoop, a plugin supplied by Strands: it submits the agent's answer to a goal, and if the goal is not met, it injects feedback and makes the agent try again, up to a cap on attempts.

A goal can be expressed in two ways: a string, which the SDK has judged by a model, or a Python function (a callable) that renders the verdict itself. Here it is a function. No judge agent is built, no token is spent, and verification is therefore free. It chains five checks in order, from cheapest to most expensive, so that an answer that retrieved nothing costs no guardrail call.

The answer is not empty.
Something arrived through the evidence tools. Reading your document does not count.
At least two chapter attributions are present.
Every citation credited to the book appears word for word in the retrieved material, after typographic normalisation.
The Amazon Bedrock Guardrails contextual-grounding check agrees.

Check 3 is deliberately coarse: it counts occurrences of the word “chapter” and of the real titles returned by book_chapters, with a length filter that excludes three-letter titles liable to be found inside ordinary words.

Check 4 is the one that decides that a citation is a citation. It judges nothing, it compares: each citation credited to the book must appear word for word in the retrieved material, after typographic normalisation. Two details matter. The comparison runs before any paid call, and it reads the untruncated material. Having run it against the truncated copy sent to the guardrail flagged fourteen citations out of nineteen as missing when they were in fact present.

The fifth check in depth: Amazon Bedrock Guardrails

The first four checks are our own code and cost nothing. The fifth asks a managed service for a second opinion: Amazon Bedrock Guardrails’ contextual grounding check, which scores how well a text is supported by a provided source. The call is made with ApplyGuardrail and takes three content blocks: the grounding source (qualified grounding_source), the question (qualified query), and the text to evaluate with no qualifier. The last block is the one that gets forgotten, because attached to a Converse call the guardrail would see the model's response for free.

This check comes with two scoping caveats. The docs cover summarisation, paraphrasing and question-answering, but exclude conversational: an assessment triggered by a single tool call falls on the right side, coaching that has become a dialogue would no longer fit. And the guardrail merges all grounding_source values before evaluating them, which erases the notion of "which division" by construction and mechanically prevents it from catching a chapter misattribution.

Mechanism 5: citations the model does not compose

The four preceding checks work after the fact, on prose: the citation is a string the model has written that has to be found in the material, and its chapter attribution is read from the words around it. Check 4 catches the invented citation, but not the misattribution. A passage from chapter 3 credited to chapter 7 passes. Mechanism 5 changes the nature of the guarantee: the citation text no longer comes from the model, it becomes structurally incapable of it.

The flow runs in four steps:

We send the Method_Model excerpts to Bedrock as a citable document. A DocumentBlock from the Converse API accepts citations: {"enabled": true} and a source whose content member is a list of text blocks (DocumentContentBlock[]). Each block is an independent citable unit. Our excerpts are already verified word for word against the division they name and carry its identifier and chapter title.
The model answers with CitationsContentBlock attached to the passages it produces. Each citation carries two decisive fields: sourceContent[].text (a verbatim slice of the block sent , not a string composed by the model) and location.documentChunk.start (the index of the source block in the sent list).
The citation resolves by index, no longer by text search. Since the source block is identified by an index and the quoted text is a slice of it, verification becomes an array lookup: read documentChunk.start, retrieve the matching Method_Model excerpt, and read its chapter title. A wrong chapter is structurally impossible.
An attribution filter decides which citations to submit. Verified agents legitimately put their own words in quotation marks (a need they articulate, a line they propose). A citation counts as a claim about the book only if an attribution cue (“according to”, “the book”, “chapter”, the author’s name, and about fifteen others) appears within the 120 characters that precede it. Without this filter, the agent is punished for doing its job.

This flow is not a homegrown construction. The Converse API citations are the feature Bedrock provides for verifiable attribution. The work specific to this project fits in one decision: send as citable blocks only excerpts already verified word for word.

Two artefacts survive the process. The delivered document carries a “Verification note” section that names in plain text every unverified citation, and the results.json evaluation record tags rejected citations invented or not retrieved. The evidence ledger itself writes nothing. It lives in memory and dies with the call. A rejection is costly, it destroys a finished answer, so the log line is what distinguishes a justified rejection from an unjustified one.

The retry budget, and a timeout that cannot help

The loop allows two attempts, not three. Each retry appends the previous answer and the feedback to the conversation, which grows on every turn. A measured run degraded exactly at the third: seven unverified citations out of twenty-nine, then one out of twenty-six, then a collapse to 1,252 characters. max_attempts=2 is measured, not chosen.

GoalLoop also accepts a timeout, but it checks it in AfterInvocationEvent, so after the model's work is finished and billed. A timeout at that point cannot cancel an in-flight invocation, only skip verification of an already-paid response. It is max_attempts that really bounds the loop. And GoalLoop returns its last attempt rather than its best one, so the final selection happens outside the loop over the preserved verdicts, with a rule that ranks "has at least one attribution" above every citation count. Otherwise a collapsed retry that asks for the document back (zero unverified citations) would beat a real assessment that has one.

When the window loses anyway, nothing ships

Pinning protects the citations, but it can also make an overflow unrecoverable: if all the messages old enough to be dropped are pinned, the conversation manager has nothing left to cut and raises an exception. This is an accepted trade-off, and the failure is loud rather than silent. The agent raises GroundingLostError with a message that says exactly why: "nothing was written : a document produced past this point quotes the book from the model's memory rather than from the passages, and reads exactly like one that does not".

Measuring the agents: evaluations and optimiser

Two Strands packages carry this part and answer different questions. The evaluation SDK asks whether an answer is good. The Harness Optimizer asks whether a prompt is better.

Why evaluation cases come in pairs

Each evaluation case comes with a twin. The first carries a known violation of the method and tests that the agent finds it. The second is a compliant case and tests that the agent does not fabricate a finding. A suite made only of violations can never catch that second failure mode, which is the most costly one: an invented reproach on a correct text.

The optimiser proposes, a human applies

The optimiser wires up the real agent, the real evaluators and a ContrastiveReflectionOptimizer that reads execution traces and proposes a rewrite of the prompts. The reward is the evaluation suite itself.

Now comes the decision that matters most: the optimiser never writes into the sources. It writes a before, an after and an evaluation record into a timestamped directory.

Conclusion

The architecture turns a methodology book into a coaching agent, and five mechanisms defend the citation. A pinned evidence ledger prevents a context trim from letting the verifier vouch for a text the conversation dropped. One chunk per book division makes a citation name a place you can open. Word-for-word extraction or nothing keeps the unintelligible away from the model. Native Converse citations make a citation non-composable by the model and its attribution non-forgeable, because the quoted text is a slice of the block we supplied. And a two-stage output check runs an exact test for free before asking Amazon Bedrock Guardrails for a second opinion.

The failure that all five prevent is an unsourced answer that reads exactly like a sourced one. You cannot detect it on a read, and that is why it must be made impossible upstream. You get feedback on your own document with a citation you can open, and an auditable trail behind each one.

If you take only five engineering lessons from this architecture, take these. Pin the tool results of your evidence ledger, because a sliding window will otherwise let it vouch for text the conversation dropped. Register that ledger as a HookProvider and not as a plugin, because the plugin registry indexes on a name it does not have. Bound a GoalLoop by attempts rather than by clock, since its timeout is evaluated after the model's work is billed. Let an optimiser propose rather than apply, because an evaluation suite that scores documents cannot catch a prompt quietly relaxed. And when an agent's output exceeds what the model can produce in one generation, delegate to sub-agents with a fresh context rather than raising the ceiling: the orchestration is code, the work is in the agent, and the parallelism falls out naturally.

Editing a Markdown corpus without loss, with an Amazon Bedrock AgentCore agent

Guillaume Marchand — Wed, 19 Aug 2026 14:40:31 +0000

Context

A 23 KB document in our corpus came back at 1,364 bytes. Every tool call had returned a success, the file was still valid Markdown, and the agent had reported its work done. Nothing in the session said anything was wrong. What had gone was everything the skeleton the agent had just written did not contain.

That is the failure this article is about: an agent writing into a corpus of linked documents can succeed at every write and damage the corpus, and what it removed is only visible at the scale of the whole. It is not specific to one kind of corpus. A documentary workspace where every document cites its neighbours describes an on-call runbook base, a library of contractual clauses, a set of architecture decisions, a regulatory file and a corpus of research notes equally well. The on-call engineer, the lawyer, the technical writer and the screenwriter have the same exposure here: the value is not in the files taken one by one, it is in what they presuppose of one another.

The corpus lives in Markdown for a simple reason. An agent with text tools can read, write and circulate a Markdown file with the same primitives a human uses in an editor. A .docx document requires a zipped binary format and a dedicated library, which few agent stacks carry natively. The same text therefore becomes the source of truth, the reading rendition and the WYSIWYG editing format. Changes across a set of Markdown documents are traceable in a version control tool like Git.

As our corpus grew, we decided to adopt frontMatter and wikilinks. FrontMatter in a Markdown document is a header section placed at the very top of the file, holding structured metadata meant to be read by processing tools rather than displayed in the final rendition. It carries contextual information about the document — the title, the date, the tags, the author, or custom variables such as status. In practice it is a standardised mechanism that separates control data from narrative content, which is what makes automation, reuse and large-scale management of Markdown documents possible in technical or editorial workflows.

That continuity holds in the browser too. We edit these files with the same libraries the agent reads and writes. @mdxeditor/editor handles WYSIWYG editing. react-markdown renders the reading view, with remark-gfm for tables and checkboxes and remark-frontmatter for the YAML frontMatter. mermaid renders the diagrams. The author sees the same document as the agent, with no intermediate conversion.

The customer stake comes down to this: what an agent degrades silently is authored work already produced and already paid for, along with the confidence that made delegating the writing possible. A regression found three weeks later does not cost what a visible error at call time costs — you first have to establish when it started, then what it has touched since.

The problem

An editorial agent writes into a workspace mounted on AgentCore Runtime, versioned on every turn in git through AWS CodeCommit. Two families of tools reach this corpus with different scopes. The editing tools change one section of one file. The corpus tools read the hundreds of documents and answer questions no file-by-file tool can ask.

We hit two distinct failures, months apart. The second was caused by the fix for the first, and it was quieter than the one it repaired. Both remove information the agent cannot see when it looks at an isolated document, which is why they lasted.

First failure: a whole document travels through a tool argument

Our first write tool was the one everybody writes first: a path, a content, a commit message.

The trap is in the signature, not in the body. content is a tool-call argument, so the model does not forward it. It generates it, token by token, out of the same output budget as its visible answer. A 23 KB document is roughly 7,000 tokens of escaped JSON string, to be produced in one go before the call even exists.

When that budget runs out while the arguments are being generated, the call block stays incomplete. A tool call only executes once its block is finished. With no execution, no result comes back. And the agent loop hands the floor back to the model by re-injecting that result. With no result, the loop does not hand back, and the stream ends with no message and no error.

The tool-call event counters show it in one line. A failing turn records five calls started and only four results.

Our system-prompt workaround made things worse. The rule asked the model to write a skeleton first. Then it asked it to fill that skeleton in section by section, through the same tool that replaced the entire file:

The skeleton is short, so writing it succeeds. The filling is a large call, so it does not get through. And write_text on an existing path replaces the whole file.

We had written a destructive primitive. Neither its name, create_or_update, nor its description "Save a deliverable" announced that it could lose work. The system prompt carries no rule of that kind: writing goes through the per-section tools described in the second failure.

The tool, renamed create_document, refuses the dangerous case in its description by naming the replacement tools. A model only chooses well among descriptions that say what a tool destroys.

Second failure: per-section editing wipes the frontMatter

The fix was to move to per-section editing. A tool that changes one section leaves the rest of the file intact. Each call carries only one section, so the arguments stay small and the output-budget wall disappears. That part worked.

It introduced a data loss quieter than the one it repaired. Our parser splits the document on its headings, and the rebuild reassembles the sections.

Everything before the first heading belongs to no section. The frontMatter is therefore absent from the rebuild. The four write tools — update, insert, delete, move — share this parse-and-rebuild pair, and each one deleted the frontMatter of the file it touched.

The fix captures that preamble and re-emits it. The parameter is required, with no default: a caller that omits it deletes the frontMatter silently, which is exactly the failure the parameter exists to prevent.

The solution: a graph of the corpus

In a linked corpus a document is not self-supporting: a runbook presupposes a topology, a clause presupposes a definition, an analysis presupposes a statement of intent. An agent handed a single document produces an answer that contradicts the others, because it cannot see what that document presupposes.

Two signals already exist in the files to make those links explicit to the agent. The frontMatter gives the agent a document’s type and status without reading it: it knows a sheet is locked or in progress before changing it. The [[target]] wikilinks are relations written by the author, not inferred by a model. That is the correlation signal the agent needs in order to gather the relevant context instead of loading everything.

For the author, the graph answers three questions that neither a read-through nor a per-document test asks. What does this document presuppose, and what therefore has to be re-read before changing it. Which documents are load-bearing, meaning the ones where a change propagates furthest.

For the agent, the graph does two things. It gives it the relevant context to gather instead of loading everything, which is the difference between an answer that accounts for what the document presupposes and an answer that contradicts it. And it acts as a guard rail: it detects damage the agent itself caused, at a scale the agent does not look at.

We built a MultiDiGraph from NetworkX that reads both signals across every document and exposes three tools to the agent, plus an audit tool.

document_links returns a document's neighbourhood at depth N — who cites it and who it cites.
document_path returns the shortest chain of links between two documents.
cluster_report measures betweenness centrality around a topic to find the pivot documents.
check_frontmatter audits frontMatter hygiene and reports broken titles, unknown tags, and outgoing links that point nowhere.

The graph therefore serves two purposes:

giving the agent the relevant context during an edit,
detecting damage at the scale of the corpus.

This is a neuro-symbolic architecture in the ordinary sense of the term: a deep learning model on one side, a symbolic representation and symbolic reasoning on the other, with the tool call as the interface. The symbolic half owes nothing to a model: the relations are not extracted, they are written by the author. The traversal is not learned, it is a shortest path and a betweenness centrality. No step introduces model error, so when the agent is wrong the graph is not a suspect. Many stacks called neuro-symbolic have a symbolic half built by extraction from a model, which offers no such guarantee.

Figure 1 shows where all of this is deployed. Figure 2 shows what is inside the container.

Figure 1. The deployed architecture. The call to the agent goes through neither CloudFront nor API Gateway: the browser reaches the AgentCore data plane directly. A single execution role carries every access, and the partition between authors is not in IAM — which allows the whole aicd-* prefix — but in the code, from the token's sub_._

Figure 2. The software architecture. The SDK provides the model, the conversation window, the turn replay and context offloading; ours are the turn lifecycle, the deterministic closing frame and the tools.

How all of this is wired to Strands

The agent itself fits in six arguments.

Six arguments, two of them plugins the SDK already offers. The SDK brings the model, the conversation window, the replay of a turn that stopped at an announcement, the offloading of bulky results, the persistence of the conversation, and the @tool decorator that turns a Python function into a tool whose description the model reads.

That brevity is the SDK’s doing and not the sign of a minimal agent: each of these arguments carries a whole mechanism. The two sections that follow cover the two plugins, because they are the ones that decide something about the conversation.

The turn replay belongs to the SDK

The problem: the model announces an action — “Let me check the real state before writing” — and its turn ends there, with no tool call. The user sees one sentence and nothing else. In a real session, the person typed “Retry” three times before giving up.

The mechanism that answers this is in the SDK. AfterInvocationEvent carries a resume field which the hooks documentation describes as making the agent re-invoke itself automatically with the supplied input, and the GoalLoop plugin wraps it: it validates the response after each invocation and, if the validator refuses, re-injects its comment as a user message and re-enters the loop, under a max_attempts and a timeout.

What we supply is the validator, and it is a predicate over the turn that has just finished.

The point of this split is not brevity. The second attempt is part of the same run and the same stream: there are no two START/FINISHED pairs to stitch together, no concatenation of answers, and the attempt counter is the SDK’s. The test that matters drives a real Strands agent with a scripted model and checks that the model is called twice — what is measured is the SDK’s mechanism, not our confidence in it.

Bulky research leaves the window

For example, a real working turn reads nineteen sections, two method chapters and three web search results. Every result stays in the context window until the sliding window pushes it out — and what it pushes out first is what came in first, which is often the instruction.

The SDK provides ContextOffloader for this: a tool result that is too bulky goes off to a storage backend, leaves only a preview and a reference in the context, and the agent pulls the whole thing back with retrieve_offloaded_content when it genuinely needs it. The plugin is wired like the previous one.

The default behaviour is to offload every result above the threshold. Adopted as such, it would have recreated the failure this article opens with. get_document_section feeds update_document_section: give the model a thousand-token preview of a section it is in the middle of rewriting, and it rewrites the preview — it truncates the document. The should_offload parameter exists for this, and our predicate is an allowlist, not a denylist. The direction is the heart of the matter: whatever is not in it stays whole, so a tool added later is on the right side by default.

One last trap, and it holds for any list of this kind. _OFFLOADABLE_RESULTS is written in registered names — the ones the model sees. But the @tool decorator accepts an explicit name, so a tool does not always register under the name of its Python function: read_file_tool registers as read_file. Writing the function name there therefore raises no error: the entry designates no tool, and that tool stops being offloaded without anything reporting it. A test compares the set against the registry of the built agent, which is the only way to make that silence audible.

That registry holds forty-four tools: our forty-three, plus retrieve_offloaded_content, which the plugin adds to pull back what left the window.

What stays ours, and why

The workspace lifecycle, and it is not in the agent loop. A turn registry launches the work in a detached task, so that a client disconnect does not cancel it, and a second call on the same session attaches to the first instead of opening a second workspace on the same repository.

That last point is not theoretical, and it is the sharpest illustration of the whole problem. Two concurrent turns shared a directory. The second did a fetch and a reset --hard underneath the first. Eight sections out of eighteen ended up empty — and both tools reported success. Nothing in either turn had failed.

No hook could have prevented it, because the conflict is not about the conversation but about a working directory we chose to mount and clone. Neither Strands nor AgentCore knows that /mnt/workspace/ is a Git clone whose two simultaneous writers destroy each other.

The HTTP layer, on the other hand, belongs to the SDK. AGUIApp provides the routes, the RunAgentInput validation, an encoder built from the request's accept header, a context carrying the Authorization header we take the identity from, and a WebSocket transport into the bargain.

Two details of that wiring look like obstacles and are not. The heartbeat first: a raw SSE comment frame does not pass through an event encoder, but AG-UI defines a CustomEvent carrying a free name and value, so the heartbeat is an event like any other. Then, AGUIApp's generator is consumed by the HTTP response, so a client disconnect cancels it — with no effect here, because the work does not live in that generator: it lives in a task the turn registry has detached, and the generator only relays.

The structuring point fits in one sentence: the turn carries AG-UI events, not encoded frames. The SDK does the encoding, with the request’s encoder.

One constraint resists, and that one was read in the SDK’s source. The task tracking that computes the busy status automatically — add_async_task,complete_async_task, and a /ping that answers HealthyBusy for as long as a task is registered — belongs to BedrockAgentCoreApp, not to AGUIApp: the latter has no trace of that registry and offers only a ping decorator. And BedrockAgentCoreApp does not speak AG-UI. So the choice is between the protocol our interface speaks and the automatic tracking. We keep the protocol and write the ping handler, which fits in one line since the registry already knows whether a turn is in flight.

On that /ping, the documentation carries a warning. The time_of_last_update field is optional, and filling it with the current time on every call signals a continuous state change: the idle timeout then never fires, sessions live to their maximum lifetime and the quota runs out. We do not emit it, which is the recommended conduct.

Our application-level refusals — repository absent, repository not owned by the caller — are RunErrorEvent events in the stream and not HTTP 400 and 403 codes. That is the AG-UI rule, and the protocol contract states it unambiguously: every error is serialised as a RUN_ERROR event, whether it happens before or during the stream. What changes is not the shape but the HTTP code accompanying it — its real code for a connection-level error, 200 once the stream has begun. The client surfaces them instead of retrying them, which is the right behaviour for an authorisation refusal.

The detached work, on the other hand, is not a mechanism the platform provides. The documented pattern is that the developer launches the background task and the SDK merely tracks its health. Our turn registry is therefore the expected shape, not a reinvention. What it adds — a replay buffer for a client that comes back, and the refusal to open a second workspace on the same repository — has no documented equivalent.

The lesson fits in one sentence: “the platform already does it” is verified mechanism by mechanism, in the source, and never layer by layer.

The conversation is a session, and it lives in a table

The conversation is not an object the runtime holds: it is a Strands session. Session management persists agent.messages as the turn goes and restores it on AgentInitializedEvent.

Strands provides four managers — file, S3, repository, snapshot. None of them suits here, because this conversation is read back by a Lambda that serves it to the interface, and not by the agent alone. But the persistence sits behind an interface: SessionRepository, nine methods, which RepositorySessionManager consumes. We write one over DynamoDB, one partition per session:

The index is zero-padded, so lexicographic order is numeric order and the conversation window is a Query bounded by a key range. That is where the implementation counts: the S3 repository the SDK provides lists every object under a prefix and sorts in memory, which grows with the whole conversation. The reads are strongly consistent — the previous turn's writes are seconds old, and an eventually consistent read that misses the last message loses context without saying anything.

That choice has an effect which is the real benefit. A hand-rolled writer called with a single text block persists only that block: the reasoning and the tool calls are lost, whatever its documentation claims. SessionMessage carries a whole Strands Message, so they are persisted, and the interface displays them.

And the client stops carrying the conversation. The bridge only replays RunAgentInput.messages into an agent without a session manager; with one, it forwards the latest user message and leaves the history to Strands.

Two traps are silent, and we fell into both. The first: StrandsAgent treats the agent it is given as a template and rebuilds one per thread. A session manager set on that template is deliberately discarded — otherwise every thread would share one session id — and the SDK requires StrandsAgentConfig.session_manager_provider instead. Our two hundred and sixty-five offline tests passed with the manager ignored; a real turn is what revealed it.

The second: the rebuild forwards every constructor parameter it finds as an attribute on the template, and Strands does not retain plugins. The tool registry, on the other hand, is forwarded. The rebuilt agent would therefore have kept retrieve_offloaded_content visible to the model while the offloading and the replay stopped firing: a behaviour that disappears leaving its tools on display. No test that looks at the tool list sees that.

That leaves AgentCore Memory, which is not a competitor: it carries the cross-session semantic recall, injected into the system prompt, and not a conversation replay.

What AgentCore brings, declared in CDK

AgentCore is a hosting contract: it validates the token, mounts the workspace, relays the stream. Declaring it is enough to say so.

Two details of this declaration cost a whole turn when they are missing. Without request_header_allowlist, AgentCore does not forward the Authorization header and the container fails closed. And mount_path only mounts the workspace at invocation, never at container initialisation, so nothing may read the corpus at import time.

GET /ping answers HealthyBusy for as long as a turn is in flight. Without that, AgentCore judges the session idle during a long silent generation and recycles it mid-write.

Finally, the workspace root travels to the tools through contextvars, but in a mutable dictionary rather than by assignment. Strands runs each tool in its own task, where a ContextVar.set() stays invisible to the parent.

Why not GraphRAG

The author’s corpus changes on every turn, since the agent writes into it, so an ingestion per section edit would be both permanent and billed. And its symbolic half is already written, as said above, so the extraction would have nothing to produce. The graph is built in memory, with no embedding, and sees an edit immediately. GraphRAG remains the right choice on a purchased, frozen corpus: the extraction is paid for once and the embedding gives a semantic search that wikilinks do not allow.

An explicit link weighs less than a shared tag

An explicit link and a shared tag do not say the same thing. A wikilink is the author asserting that two documents belong together. A shared tag is generic and connects otherwise unrelated documents. The graph keeps two edge kinds, with different weights.

The MAX_TAG_GROUP ceiling comes from a measurement. Without it, a cross-cutting tag such as research connected every document to every other and made any distance unusable.

Those four numbers are the part of this article that does not transfer. The two bounds and the 1.0/3.0 weights were calibrated on one corpus of 90 documents, with its own tagging habits and its own ratio of explicit links to shared tags. A base of two thousand runbooks with a dozen tags in circulation will not have the same ceiling, and a corpus where authors tag more than they cite will not have the same weights. What transfers is the shape — two edge kinds, the weaker one bounded so a generic tag cannot form a clique — and the method for setting them, which is to measure when the distances stop being usable. Adopting the constants themselves is the one thing to avoid.

Self-remediation at the end of a turn

The validator from the previous section rests on a threshold, and that threshold comes from a measurement. It requires three simultaneous conditions: the turn called no tool, it wrote nothing, and its answer is under 400 characters. A day of real traffic gave the number. The three faulty announcements were 87, 99 and 108 characters. The shortest legitimate answer with no tool was 3,080.

Under that threshold and with nothing written, the turn is an announcement; above it, it is an answer and we leave it alone.

A symmetric case exists, and that one stays outside: a turn uses a tool, the tool succeeds, and the model hands back without a closing sentence. The user sees a silent screen. The _closing_summary function composes a deterministic closing message from the files that were written, with no second call to the model. It is free and predictable, and it covers the case where the output budget ran out while the tool was finishing its work.

The final split is therefore the one in the yellow frame of Figure 2, and it has a logic. The replay decides something about the conversation, so it belongs in the agent loop and lives in the plugin. The closing frame decides nothing: it builds an AG-UI frame the client expects, out of files already written. It stays in the HTTP layer, where it costs zero model calls.

Results

Per-section editing removes the output-budget wall. The same merge work that used to lose a document now produces a richer one, with as many results as calls.

The graph gives the state of the corpus after repair. We rewrote 15 links across 9 documents to catch up with an unfinished rename, then restored the 8 frontMatter blocks from the Git history.

Two rows of this table need a clarification, because their units have already produced a false alarm. “Citation relations” counts distinct pairs; “Written links” counts the edges, one per wikilink actually written. Both numbers are right, and their gap measures the corpus’s repetition: every extra link is a second mention of a document already cited. The field was originally called wikilinkEdges, a name that admitted neither reading; comparing the tool with the graph gave 469 against 808 and the reasonable conclusion that one of the two was wrong. Establishing that neither was cost a full investigation, for a defect that was in the counter's name and nowhere else.

Conclusion

You now know how to edit a Markdown corpus without loss, with an agent deployed on Amazon Bedrock AgentCore. Never pass a whole document through a tool argument. Edit per section, preserve the frontMatter at every reassembly, and keep a graph of the corpus to see what per-section editing does not show. None of this is specific to a creative corpus: the same architecture holds for runbooks, clauses or architecture decisions, as soon as the documents presuppose one another.

Two pieces of work remain open. A partial section patch, avoiding the retransmission of a long section’s whole body. And passing documents by reference in the other direction: ContextOffloader handles the bulky results coming back into the context, but the work still travels whole to the sub-agents and the MCP servers. It is the same lesson, on a layer we have not revisited yet.

Two things to take away:

A tool argument is generated by the model. It consumes its output budget. A call whose arguments are truncated is not executed, so it does not fail: it disappears.
A tool that replaces a whole file is a destructive primitive. Its name and its description must say so, and it must refuse the dangerous case by naming the replacement tool.

Identifying speakers by voice in live streaming with AWS

Guillaume Marchand — Thu, 06 Aug 2026 17:41:37 +0000

Introduction

During a televised debate, a false claim travels faster than its correction. Newsrooms that analyze live need two things at once: what was said, and which speaker said it. Amazon Transcribe answers the first question well. Identifying the speaker by voice is harder than it looks, and it is the subject of this post.

Live processing imposes constraints that batch processing ignores. Amazon Transcribe finalizes a speech segment about every five seconds, while an automated analysis runs several AI agents for two to five minutes. Naming the author of each segment has to fit in a budget of a few hundred milliseconds. Speaker turns sometimes last under a second.

Three failures are possible, and they do not cost the same. Not knowing who speaks is inconvenient. Answering too late is useless. Attributing a sentence to the wrong person is worse than attributing nothing.

Attributing a claim to someone

A verified claim without an author loses most of its journalistic value. “Unemployment fell by three points” carries different weight depending on whether a candidate, a minister, or an invited economist says it. The journalist needs to attach each claim to a person. They also need to follow that person from one broadcast to the next, to measure how consistent their statements are.

Amazon Transcribe provides diarization: the service segments the stream and labels speaker turns. That information is useful, and it is not sufficient for this use case.

Three limits of diarization alone

Diarization assigns local labels. It tells you that two passages come from two different people within one stream. It does not tell you who they are. Nothing makes a given speaker keep the same label for the whole stream either, since a label can be reassigned or split mid-session. And those labels do not survive the end of the stream, because they are renumbered on the next one.

The three limits compound. A participant becomes “speaker 1” on Monday and “speaker 3” on Thursday, with nothing linking the two. Comparing what one person said across several shows then takes manual reconciliation, which is what live processing rules out.

The first approach: inferring names from text

The first version of the identification named speakers without analyzing their voices. An AI agent “Context” on Bedrock AgentCore Runtime kept every segment of the session in Amazon Bedrock AgentCore Memory. It exploited a regularity of talk shows: at the opening, each participant introduces themselves or is introduced by the host.

From those sentences, the “Context” agent linked the anonymous diarization label to a name, then propagated that link across the rest of the broadcast. The mechanism is still readable in that sub-agent prompt. It records facts of the form “speaker 0 in session S is the first candidate”, then tries to relate any new label to the facts already stored.

That approach worked often, and that is the problem. The name came from a language model inference over text, not from a measurement on the signal. Nothing made the same conclusion reproducible from one run to the next, and no value quantified the confidence.

Failure was also silent. A missing introduction, an ambiguous phrasing, or a guest arriving mid-show, and the agent produced a plausible name rather than no name. A journalist cannot publish an attribution on that basis, because they cannot know which one to verify. The decision needed to be reproducible, to carry a score, and to be computed on the voice itself.

Solution overview

A voice fingerprint answers all those limits. It describes the timbre as a vector, independent of the show where it was computed. It is the principle of a fingerprint, applied to the speech signal. Two nearby vectors designate the same voice, across shows and across weeks. The platform compares each finalized speech segment against a registry of known voices, and attaches either a name, a stable anonymous identifier, or an explicit abstention.

That choice moves the problem rather than removing it. It introduces persistent biometric data, with the obligations that come with it, and a new risk: attributing a sentence to the wrong person. For a newsroom, that error is worse than no attribution, and the two rules in the next section come from it.

Two design rules before any optimization

The system prefers silence to error. When the resemblance to a known voice is insufficient or ambiguous, it assigns “no name”. It produces a stable anonymous label instead, such as “voice 12”, which stays the same for the whole show and for later shows. The journalist sees that the same person is speaking, without being told who. An operator can name that voice afterwards, and the history already produced becomes named.

Recognition never blocks verification. If identification fails, exceeds its time budget, or could not start, the segment still goes to the AI agents. It then carries an identity marked as degraded, along with the reason. A pipeline whose optional link can interrupt the main link is not usable live.

The architecture

The diagram that follows traces one audio stream from the sender through the streaming server to the registry and the verification agents. Read it left to right: the sender feeds a WebSocket connection, the server splits the audio between transcription and identification, and the identity joins the segment before dispatch.

Figure 1. Speaker identification architecture, from audio capture to the fingerprint registry.

An FFmpeg process decodes the source and emits signed 16-bit little-endian pulse-code modulation (PCM) audio, 16 kHz mono. It is pushed over a WebSocket connection in 100 millisecond frames.

The streaming server is a container running on AWS Fargate, in an Amazon Elastic Container Service (Amazon ECS) cluster. That container does four things. It relays the audio to Amazon Transcribe Streaming and receives partial then finalized results, with their native time boundaries. It keeps recent audio in a 40 second rolling buffer, one per session. It runs two Open Neural Network Exchange (ONNX) models on CPU, and it compares the resulting fingerprint to the registry of known voices.

The buffer holds 40 seconds for a reason. That covers the 30 second cap observed on finalized results, plus the two second widening described in the section on going from signal to identity, plus delivery lag. The cost is bounded: 40 seconds of 16 kHz 16-bit mono audio is 1.28 MB per session.

Amazon Transcribe Streaming is used, and it is the only transcription engine in the system. It produces the text, the partial results displayed to the journalist immediately, the segmentation of speaker turns, and their time boundaries. What the system does not use is the diarization the same service offers alongside transcription.

The platform needs to follow a speaker from one broadcast to the next and to attach a name. Diarization labels, whether from Amazon Transcribe or from any other model, are anonymous identifiers local to a single inference. They distinguish speakers within one stream but do not name them and do not persist across streams. The fingerprint registry solves both: it carries names, and it survives sessions.

Two models run inside the container, and neither provides identity. Segmentation uses pyannote segmentation-3.0 under the MIT license. It signals where there is voice, where the speaker changes, and where two people speak at once. Those are segmentation signals: they tell the fingerprint extractor where to cut, not who speaks. The pyannote speaker classes are just as anonymous and local as the Transcribe spk_N labels, and the system reads none of them. The fingerprint itself uses WeSpeaker ResNet34-LM under the Apache 2.0 license, which produces a 256 dimension vector. Identity comes from comparing that vector against the registry, and from nowhere else.

Both artifacts are published to a versioned Amazon Simple Storage Service (Amazon S3) bucket, and the model version is recorded with every registry entry. A fingerprint computed by one model is comparable only to fingerprints from the same model, and that constraint has to stay verifiable months later.

Running inference inside the container avoids an extra network hop on a time-constrained path. Inference is CPU-bound, so it runs in a dedicated thread pool, never on the event loop that serves the WebSocket connections.

The registry lives in Amazon DynamoDB, in an infrastructure stack separate from the server. That separation is deliberate: biometric data has its own lifecycle, independent of redeployments of the compute that produces it. An AWS Lambda function handles enrolment of reference voices.

There is no synchronization, and that is the design

One question follows from the architecture. The ONNX models run inside the container, and Amazon Transcribe is a remote service. The identity has to land exactly on the boundaries of the text. How are the two clocks kept in step?

They are not, because there are not two streams to align. There is one stream of PCM audio, and both consumers count the same samples. A single call site pushes each frame to the transcription queue and to the rolling buffer, in that order, from one coroutine.

The two clocks are in fact the same clock. The rolling buffer is addressed in bytes, not in time. Its position in milliseconds is the number of bytes received divided by 32, because 16 kHz at two bytes per sample gives 32 bytes per millisecond. Its origin advances only when the buffer trims, so it is a sample counter rather than a wall clock.

Amazon Transcribe returns start and end times in seconds since the beginning of its own stream. That stream was fed by those same bytes. Byte N therefore sits at the same millisecond on both sides. The audio is the clock, nothing needs reconciling, and no correlation between concurrent tasks is required.

In practice, a finalized result arrives with boundaries such as 78787 to 85500 milliseconds. The server cuts its own buffer at those indices and runs the ONNX models on those bytes. The remote service never returns audio.

The principle is worth stating once. The remote automatic speech recognition (ASR) service is used as a function from audio to text plus time coordinates. The reference audio stays local, always.

From signal to identity

The next diagram shows the decision path a single finalized segment travels. It runs from the Transcribe boundaries through the quality tiers to one of three outcomes: a name, an anonymous group, or a degraded identity.

Figure 2. From audio signal to speaker identity, including every abstention path.

Each segment finalized by Amazon Transcribe triggers the same sequence, and the two models play distinct roles. Amazon Transcribe supplies the segment time boundaries, which trigger the processing and frame the window cut from the buffer.

The pyannote segmentation gives voice activity at millisecond resolution, which measures the speech actually usable in the window rather than its raw duration. It gives the instants where the speaker changes, and the passages where two people speak at once. The Transcribe boundaries say when a sentence starts and ends, but not where the silence sits inside it, or where the voice changes.

A short sentence supplies little material, so the system widens the window by up to two seconds before and after. Two rules bound that widening. It never crosses a speaker change boundary that the segmentation detected, otherwise the fingerprint would mix two voices. And it uses only audio already in the buffer, never waiting for audio still to come, which would hold up the verification.

The usable speech duration then determines what the system allows itself to do. Below 30 seconds it extracts no fingerprint and returns a degraded identity.

Between 30 and 75 seconds it extracts a low quality fingerprint. That fingerprint can attach the segment to an already known voice, but it never creates a new voice and never enriches an existing one.

Above 75 seconds the fingerprint is high quality and does both.

This asymmetry protects the registry. A fingerprint computed on a speech fragment is noisy. Using it to label one segment costs at most one wrong label, visible with its score. Using it to create or update a registry entry makes that noise permanent, and every later comparison inherits it. The system therefore lets a weak fingerprint read the registry, never write to it.

Two distinct thresholds govern what follows. Above 0.60 cosine similarity, the segment attaches to the nearest voice and inherits its stable identifier. Above 0.70, and only if that voice already carries a name, the identity is named. Between the two, the segment joins the group without carrying a name.

A third safeguard applies. If the two highest-scoring candidates are separated by less than 0.05, no name is assigned. Two voices too close to tell apart produce an abstention, not a bet. Every branch converges on sending the segment to the verification agents, including the degraded branches.

Concurrency: two invariants to hold

Several Fargate tasks can process sessions in parallel. Two properties of the registry have to hold under that concurrency, and neither comes from a naive write.

one new voice must never create two entries. A lock held in a DynamoDB item serializes the critical section that compares then creates. The lock is acquired by conditional write, and carries a time to live that releases it if its holder disappears. The holder rereads the registry before deciding, and therefore sees any entry created in the meantime.
two simultaneous enrichments of the same voice must both be applied. The aggregate fingerprint of a voice is therefore not a vector rewritten on every contribution, which would lose one write in two. Each contribution is an immutable item on a distinct sort key, and the aggregate fingerprint is a derived value recomputed from the full set of contributions. Two concurrent writes land on two different keys and cannot overwrite each other.

That recomputation happens off the critical path. It cost 95 to 153 milliseconds per segment when it ran synchronously, for a value that is only a cache of contributions already made durable.

Keeping up with live: a session round robin

The dispatch design follows from one piece of arithmetic, which the next diagram lays out alongside the pool it produces.

Figure 3. Session round robin across Amazon Bedrock AgentCore session identifiers.

Amazon Transcribe finalizes a segment about every five seconds, while a full verification runs multiple sub-agents and takes minutes. Processing segments one after another therefore opens a backlog that never closes.

The first version of the POC framed that imbalance with a semaphore: one segment at a time, the others dropped. It lost about 80 percent of the stream. A debate verified at 20 percent has no editorial value.

Amazon Bedrock AgentCore Runtime offers the lever that solves this. It isolates execution by session identifier, so two calls carrying two distinct identifiers run in parallel, in separate environments. Having K identifiers is therefore enough to obtain K concurrent verifications, with no containers to provision or manage.

The server builds that pool at the start of each broadcast. The K identifiers are derived deterministically from the broadcast identifier, which keeps them stable across reconnections, and AgentCore requires each one to be at least 33 characters long. Segments are then distributed across the pool in round robin. The sizing follows from the arithmetic above: the segment rate multiplied by the worst case latency.

The call is made without waiting for the response, and that is the second point holding the whole thing together. The streaming server does not need the verdict. The agent publishes it to AWS AppSync thanks the hook feature of Strands Agent SDK, and the journalist interface receives it through a GraphQL subscription. The server drops the segment on a thread pool and moves to the next one.

This inversion removes backpressure instead of managing it. There is no semaphore, no reading of the response stream, and no segment dropped for lack of room. The server sends 12 segments per minute whatever a verification costs.

The pool is prewarmed. When the broadcast opens, the server sends K warmup calls in parallel, so the first real segments meet environments that are already active rather than a cold start.

Two properties of the system make giving up session affinity acceptable. The AI sub-agents are stateless from one segment to the next. And the facts accumulated about speakers live in the long-term memory of Amazon Bedrock AgentCore Memory, whose namespace is partitioned by broadcast identifier rather than by session identifier. The executions therefore read and write the same set of facts, while short-term memory stays per session.

Two tradeoffs come with this choice. Verdicts arrive out of order, because verifications started in sequence do not finish in the same sequence, so the interface reorders them on their original timestamp. And the shared facts are eventually consistent: a fact written by one execution can take a few seconds to become visible to another. That is acceptable here because a speaker identity stays stable during a show.

One implementation detail is worth calling out, because it connects this section to the concurrency invariants. The container uses two distinct thread pools. One is sized on K for the calls to the agents, which wait on the network. The other is limited to four threads for model inference, which consumes CPU. Mixing them would place voice identification behind 24 in-flight network calls, and would blow its time budget on every segment.

Security and GDPR

A voice fingerprint vector used to uniquely identify a person is biometric data. GDPR places it in the special categories, whose processing is prohibited unless an explicit exception applies. What follows describes technical measures, not a legal qualification. Security on AWS rests on a shared responsibility model. AWS is responsible for security of the cloud, and provides tools such as encryption and fine-grained access management. The customer remains responsible for security in the cloud: configuration, access control, and the compliance of their own processing. AWS helps customers work toward their compliance goals, and no AWS service by itself makes a GDPR processing activity compliant.

Raw audio is never retained. The buffer holds 40 seconds of signal, is trimmed continuously, and is released in full when the session ends or the connection drops. Only the 256 dimension vector crosses the boundary into durable storage. That is data minimization applied as close as possible: the system keeps what it needs to compare, and nothing more.

The registry is encrypted at rest with an AWS managed key, and point-in-time recovery is enabled. Access is restricted by AWS Identity and Access Management policies to the Fargate task role and the enrolment function alone. No other component of the platform reads that data.

Every entry carries its traceability : provenance, indicating whether an operator enrolled the voice or the system built it automatically, creation date, last update date, and model version. That traceability serves accountability under Article 5, and makes it possible to invalidate a batch of entries if the model changes.

Deletion is a tested code path , not an intention. A voice can be deleted by identifier or by person name, which erases the aggregate entry, all of its contributions, and the references to its enrolment clips. After deletion, the system no longer produces the corresponding name or identifier.

What this prototype does not handle deserves to be stated as plainly. It does not manage consent collection or its traceability. It applies no automatic retention policy, so a voice stays in the registry until explicit deletion. It produces no record of processing activities. Those elements are required for a production deployment and sit outside the technical scope described here.

Measured results, and what is not achieved

On an extract of a televised debate used as a demonstration dataset, the system processed 24 speaker turns and told the two speakers apart, with no wrong attribution observed. On a measurement set of 11 annotated segments, 8 are named correctly and 3 stay anonymous because those voices are absent from the registry, which is the expected behavior. These samples are small, and they validate behavior rather than establish general performance.

Measurement on Fargate gives about 470 milliseconds on average and 991 milliseconds at the 95th percentile.

The breakdown explains why, and it is the most useful lesson of this work. The time splits into three parts of comparable weight: inference of the two models, signal preparation ahead of the models, and round trips to DynamoDB. There is no isolated hot spot. The fingerprint comparison itself, which you might suspect first, costs a tenth of a millisecond, because the registry holds a few hundred entries and the vectors are normalized.

I removed what could be removed without a tradeoff. The aggregate fingerprint recomputation moved off the critical path, and the read cache is no longer invalidated in full after each write. That gained about 25 percent on the average, and it is where the cheap wins stopped.

Conclusion

A newsroom that analyzes live can now attach each verified claim to a named speaker, and follow that speaker from one broadcast to the next. That is the benefit the voice fingerprint delivers over the diarization labels the platform started from, and the measurement in this post shows where the two diverge.

Recognizing a voice in a real-time stream asks less for algorithmic sophistication than for rigor on edge cases. The decisions that mattered most are not the choice of models. They are three tradeoffs: preferring abstention to error, preventing identification from blocking processing, and forbidding low quality fingerprints from writing to the registry.

The sensitivity of the data imposes a framework from the design stage. Keeping only the vector, encrypting it, restricting access, tracing provenance, and making deletion effective are measures to build in from the start, not to add afterwards.

To go further, read the Amazon Transcribe Developer Guide, and the Amazon DynamoDB Developer Guide on the conditional writes that make the concurrency invariants holdable. The Amazon Bedrock AgentCore documentation covers the session isolation the round robin relies on.

If you work on an adjacent problem, the first question to settle is not technical: what does your system do when it is not sure?

Accelerate Step Functions Development with LocalStack: Testing Workflows Locally While Connecting…

Guillaume Marchand — Tue, 03 Feb 2026 08:28:09 +0000

Accelerate Step Functions Development with LocalStack: Testing Workflows Locally While Connecting to Real AWS Services

Developing complex Step Functions workflows often creates a frustrating bottleneck for development teams. Every code change requires a full deployment to AWS, turning what should be a quick iteration into a 30-minute wait. This slow feedback loop hampers productivity and makes debugging workflows unnecessarily difficult.

I want to share how you can accelerate your Step Functions development by using LocalStack to test workflows locally while maintaining connections to real AWS services. This approach reduces your feedback loop from 30 minutes to just few minutes, enabling rapid iteration without sacrificing the reliability of testing against real AWS services.

The Traditional Development Challenge

Step Functions workflows orchestrate multiple AWS services through complex state machines. A typical data ingestion workflow might coordinate multiple different steps using parallel states, map operations, and Lambda integrations with services like Amazon DynamoDB, Amazon Bedrock, and Amazon S3.

Testing these workflows traditionally requires deploying the entire stack to AWS. This creates several problems. Development cycles become slow and expensive. Debugging requires sifting through CloudWatch logs across multiple services. Team members often step on each other when sharing development environments.

The core challenge lies in balancing local development speed with testing authenticity. You want the rapid feedback of local testing, but you also need confidence that your workflow will behave the same way in production when interacting with real AWS services.

Solving the LocalStack Credentials Challenge

LocalStack provides an excellent foundation for local AWS service emulation, but it presents a specific challenge when you want to connect to real AWS services. LocalStack automatically injects fake AWS credentials into Lambda containers, preventing them from accessing actual AWS resources.

When LocalStack starts Lambda containers, it injects environment variables that override your real AWS credentials. Variables like AWS_ACCESS_KEY_ID receive fake values, AWS_ENDPOINT_URL gets redirected to LocalStack's internal endpoints, and AWS_SESSION_TOKEN contains invalid tokens. These injected values prevent your Lambda functions from connecting to real AWS services.

The solution involves understanding how LocalStack manages Lambda execution and configuring it to allow real AWS access. LocalStack runs Lambda containers with a specific user account called sbx_user1051, not the root user. This detail becomes crucial when mounting AWS credentials files.

Implementing the Solution

The implementation requires careful configuration of both LocalStack and your Lambda functions. You configure LocalStack using Docker Compose with specific environment variables that control credential injection and container behavior.

Setting network_mode to host allows LocalStack to mount volumes from the host system. The DISABLE_TRANSPARENT_ENDPOINT_INJECTION flag partially disables LocalStack's automatic endpoint redirection. Most importantly, LAMBDA_DOCKER_FLAGS mounts your AWS credentials file into the correct location within Lambda containers.

The critical insight involves mounting credentials to /home/sbx_user1051/.aws rather than /root/.awsbecause LocalStack's Lambda containers run with the sbx_user1051 user account. This ensures that when your Lambda code calls boto3, it can find and use your real AWS credentials.

Your Lambda functions need modification to handle both local and AWS execution environments. You create a session management module that detects LocalStack execution and removes the injected fake credentials. This allows boto3 to fall back to reading credentials from the mounted file.

The session management code checks for the LOCALSTACK_HOSTNAME environment variable to detect local execution. When running locally, it removes AWS_ACCESS_KEY_ID, AWS_SECRET_ACCESS_KEY, AWS_SESSION_TOKEN, and AWS_ENDPOINT_URL from the environment. This forces boto3 to reload credentials from the mounted AWS credentials file.

Results and Trade-offs

This approach delivers significant productivity improvements. Your development feedback loop accelerates from 10s minutes to some minutes. You test real Lambda code against actual AWS services, ensuring high confidence in your testing.

The solution does involve some trade-offs. You still incur costs for AWS services like Amazon Bedrock and DynamoDB during testing. Temporary credentials require periodic regeneration during development sessions. The LocalStack user account path could potentially change between versions, though this rarely occurs in practice.

Conclusion

LocalStack provides a powerful foundation for accelerating Step Functions development when configured properly to work with real AWS services. By understanding and overcoming the credential injection challenges, you can achieve rapid local iteration while maintaining confidence through testing against production services.

This approach transforms Step Functions development from a slow, deployment-heavy process into a fast, iterative experience. Teams report significant productivity gains and reduced debugging time when implementing this local testing strategy.

The investment in setting up this local development environment pays dividends quickly through faster iteration cycles and more reliable deployments. Your development team can focus on building great workflows rather than waiting for deployments and debugging in production environments.

Standardizing AI-Developer Collaboration

Guillaume Marchand — Mon, 06 Oct 2025 10:09:56 +0000

Development teams across industries adopt AI coding assistants (Amazon Q Developer, Cursor, Kiro.dev, Cline, RooCode) to accelerate productivity and improve code quality. However, maintaining consistent standards between AI assistants and human developers presents significant challenges. Teams struggle with varying coding standards, undocumented architectural decisions, and complex knowledge transfer processes as their development efforts scale.

This article introduces a comprehensive sample that addresses these collaboration challenges by establishing standardized development practices, implementing proven architecture patterns, and integrating AI coding assistant configuration. This approach creates a unified development experience that benefits both AI assistants and human developers across all projects.

Addressing AI Development Complexity

Developers want to focus on creative aspects of building applications while staying in their workflow without compromising quality or dealing with repetitive boilerplate code. AI coding assistants promise increased flow, better productivity, and a more enjoyable development experience. As these assistants evolve toward more agentic workflows, developers need effective ways to provide appropriate context to these AI tools.

Many developers express excitement about emerging approaches like “vibe coding,” where they chat with AI agents that guide them step by step through application development. These approaches work well for small prototypes but often break down as projects increase in complexity. Developers frequently report that vibe-coding works for side projects but not for professional work where code quality and correctness are paramount.

Current agentic development approaches often require developers to spend significant time guiding the agents and fixing problems — sometimes as much time as writing code from scratch. As teams leverage agents for more complex tasks, they need to provide more precise project plans to reduce ambiguity and ensure the agent’s work meets quality standards.

Introducing Kiro Integration

Kiro integrates four key features that create structured workflows to maintain quality standards while enabling AI automation. Spec-driven development provides a formalized approach to building features through iterative design and implementation processes. Agent hooks enable automatic AI execution when specific development events occur, operating autonomously based on developer-defined prompts to maintain quality practices.

Agent steering provides additional context and instructions that influence AI assistant behavior throughout development interactions through specialized steering files. Model Context Protocol (MCP) servers extend AI assistant capabilities with specialized tools for various development tasks, making the assistants more capable and contextually aware.

Establishing Shared Development Standards

The sample creates shared understanding between AI assistants and development teams through standardized development rules across projects. This unified approach enables AI assistants to access the same project knowledge that guides human developers, ensuring consistent architectural decisions and coding practices throughout the development lifecycle.

The sample includes pre-configured Model Context Protocol servers for AWS services, documentation generation, diagram creation, and code analysis. Comprehensive development rules guide both AI assistants and developers, eliminating knowledge gaps and ensuring consistent implementation patterns across your projects.

These standardized rules cover architecture patterns, coding standards, testing strategies, and development practices. This approach creates a shared vocabulary between AI assistants and development teams, improving collaboration effectiveness and maintaining consistency as projects evolve.

Implementing Enterprise-Grade Architecture

The sample implements a hexagonal architecture pattern that ensures clean separation between domain logic, ports, and adapters. This pattern makes applications more resilient to changes in external dependencies while enabling AI assistants to understand clear boundaries between business logic and infrastructure concerns.

The hexagonal approach allows your teams to focus on core business logic while keeping implementation details at the edges of your application. This separation of concerns makes your applications easier to test, maintain, and evolve over time, providing long-term benefits for both development velocity and code quality.

Scaling Across Enterprise Teams

At enterprise scale, the template maintains consistency across multiple teams and projects through standardized approaches including design patterns, coding rules, software lifecycle management, and testing strategies. Lead developers can initialize projects with proven patterns while teams receive updates to maintain consistency with evolving standards.

The template uses Cruft and Cookiecutter for project generation and ongoing synchronization with upstream improvements. This approach ensures projects generated from the template receive updates to development standards, security improvements, and new features without manual intervention.

Your teams benefit from centralized standard management while maintaining autonomy in their specific implementations. The template approach scales organizational knowledge and best practices across all your development efforts, creating consistency without sacrificing team flexibility.

Getting Started with Immediate Benefits

Using the template requires minimal setup and provides immediate productivity gains.

Prerequisites

Install the required tools:

Install Task for task automation
Install uv for Python environment management
Install Cruft for template management:

pip install cruft

Creating a New Project

Generate a new project with a single command:

cruft create https://github.com/aws-samples/sample-ai-coding-standards-template.git --directory template/

This command creates a complete project structure with:

Hexagonal architecture implementation
AWS CDK infrastructure setup
Integration tests with real AWS resources
AI assistant configurations for multiple AI Coding Assistants
Comprehensive documentation system
Build and deployment automation

Project Setup and Development

Once your project is generated, follow these steps:

Set up development environment :

task setup

2. Set up infrastructure environment and build Lambda functions :

task cdk:setup 
task cdk:build

3. Deploy to AWS :

task cdk:deploy

Set up test environment and run tests :

task test:setup 
task test:all

AI Assistant Integration

Each generated project includes pre-configured AI assistant support:

Amazon Q Developer
Roo Cline
Kiro AI
Cursor AI
Cline

The steering files and hooks configure AI agent behavior for specific project contexts, providing comprehensive organizational context that enables AI assistants to understand architectural patterns, coding standards, and specific requirements without extensive human guidance.

Template Synchronization

Every project includes automated template update capabilities:

# One command for seamless template updates
task cruft:update

This automation provides automatic conflict resolution, requires no user interaction, maintains a clean workspace, and never blocks your workflow. The system applies updates where possible and gracefully handles conflicts.

Documentation System

Projects include a comprehensive documentation system built with MkDocs:

# Set up and serve documentation locally
task docs:setup
task docs:build
task docs:serve

The documentation system automatically generates API documentation from Python docstrings, organized by hexagonal architecture layers, and includes examples demonstrating proper usage patterns.

Transforming Development Team Productivity

This standardized approach delivers significant advantages for teams building applications with AI assistance. AI coding assistants receive comprehensive organizational context, enabling them to understand architectural patterns, coding standards, and specific requirements without extensive human explanation. Structured development rules and steering files provide clear guidance on implementation patterns, making AI suggestions more accurate and contextually appropriate for your projects.

Developers eliminate the cognitive overhead of maintaining consistency between AI-generated code and established project standards. Teams can trust AI suggestions because they follow the same architectural patterns and coding conventions that guide human development decisions. Comprehensive documentation and reference implementations enable developers to quickly understand and extend AI-generated code while maintaining project consistency.

The hexagonal architecture creates loosely coupled systems where application components can be tested independently, with no dependencies on data stores or user interfaces. This pattern helps prevent technology lock-in while providing a clear structure for both AI assistants and human developers to follow.

AI assistants become more effective contributors through better project context understanding. Developers spend less time reviewing and correcting AI-generated code because it follows established standards from creation. Standardized patterns simplify team member onboarding and maintain consistency across projects and contributors.

The combination of proven architectural patterns, comprehensive automation, and shared standards creates a powerful foundation for modern cloud native development. Your teams can leverage AI assistance effectively while maintaining the consistency and quality standards required for successful applications, accelerating development velocity while ensuring code quality and architectural integrity across your organization.

Process millions of media assets with an open source tool “FFmpeg” on AWS

Guillaume Marchand — Wed, 03 Sep 2025 07:36:52 +0000

Process millions of media assets with FFmpeg on AWS

I need to process over 3 million multi-modal files for training a large language model (LLM) that can understand and generate audio in order to launch generative artificial intelligence based customer experiences. Training an audio LLM requires massive amounts of high-quality audio data to learn and understand acoustic patterns. The team has access to millions of audio files stored in Amazon S3, but processing them sequentially on an Amazon EC2 instance does not scale.

To efficiently process the audio for training LLMs , I improved the AWS Batch with FFmpeg sample code, an open audio/video processing sample code using AWS Batch and an Open Source tool FFmpeg.

In this article, I provide technical details on building a reliable, scalable processing workflow using AWS Step Functions and AWS Batch. The workflow utilizes the open source tool “FFmpeg”, to process large volumes of media assets. I describe how to configure AWS Step Functions to orchestrate AWS Batch jobs, handle job failures gracefully, and work within service limits. This architecture shows how you can leverage AWS services like Step Functions and Batch together with open source tools like FFmpeg to create a robust and managed processing pipeline.

The architecture

At AWS re:Invent 2022, AWS announced the availability of a distributed map for AWS Step Functions. This new state type extended support for orchestrating large-scale parallel workloads.

This state is ideal for processing workflows, where many assets can be processed in parallel. I can compose any AWS service API supported by Step Functions into the workflow. In our use case, AWS Batch is invoked directly from the Amazon States Language to parallel process assets without writing new code. This is achieved through Step Functions Service Integrations which allow users to call supported services directly in the Resource field of a Task state.

The following code describes a Step Function state submitting a new job to Batch:

"SubmitJob": {
            "Type": "Task",
            "Resource": "arn:aws:states:::batch:submitJob.sync",
            "Parameters": {
              "JobName.$": "$.name",
              "JobDefinition.$": "States.Format('arn:aws:batch:<region>:<account>:job-definition/batch-ffmpeg-job-definition-{}',$.compute)",
              "JobQueue.$": "States.Format('arn:aws:batch:<region>:<account>:job-queue/batch-ffmpeg-job-queue-{}',$.compute)",
              "Parameters.$": "$"
            },
            "End": true,
          }

I build upon the existing “AWS Batch and FFmpeg“ sample by encapsulating it in a Step Functions state machine. The state machine uses a distributed Map state task optimized for Amazon S3 inputs. By configuring the S3 bucket and prefix directly in the map, the state machine processes assets in parallel.

For each map task, the state machine invokes a Batch job to leverage FFmpeg tool and perform audio encoding.

The following design shows how AWS Batch handles compute provisioning and scheduling, while Step Functions orchestrates the workflow — all in a serverless model.

Beyond the limit

Retry mechanism

Following the Step Functions Quota documentation, the Step Functions distributed map supports a maximum concurrency of up to 10,000 executions in parallel, which exceeds the concurrency limits of AWS Batch. When integrating Step Functions with other services, I must consider the downstream service’s quotas and limits, to avoid errors, as you can read in the following screenshot.

AWS Batch has a quota of 1 million jobs in Submitted state and a limit of 50 transactions per second (TPS) for SubmitJob API calls. To avoid exceeding the TPS limit, I could configure the Step Functions distributed map’s “maximum concurrency” to 50. However, at this rate, it would take over 16 hours to submit 1 million jobs, excluding processing time.

A better solution is to use Step Functions’ “enhanced error handling” capabilities. This allows us to set a max limit on retry intervals to prevent excessive delays. Adding jitter introduces randomness into the retries, avoiding a retry storm that could overwhelm Batch. The combined error handling controls retry rates appropriately during failures while still allowing the high concurrency of distributed maps for normal operation.

When configuring AWS Step Functions, it’s important to set the maximum concurrency appropriately for the workload. Here, I set it to 5,000. Instead of retrying immediately and aggressively, the Step Functions waits some amount of time between tries. The most common pattern is an exponential backoff, where the wait time (IntervalSeconds = 180 sec.) is increased exponentially (BackoffRate = 3) after every attempt. Exponential backoff can lead to very long backoff times, because exponential functions grow quickly. To avoid retrying for too long, implementations typically cap their backoff to a maximum value ( MaxAttempts = 10). This is called, predictably, “capped exponential backoff “, the blog post ”Timeouts, retries, and backoff with jitter“ from Amazon Builders’ Library explains in detail the concept . If all the failed calls back off to the same time, they cause contention or overload again when they are retried, Jitter adds some amount of randomness to the backoff to spread the retries around in time (JitterStrategy = “FULL”).

The following code describes a Step Function state part with this retry mechanism:

"Retry": [
              {
                "ErrorEquals": [
                  "States.ALL"
                ],
                "BackoffRate": 3,
                "IntervalSeconds": 180,
                "MaxAttempts": 10,
                "Comment": "retry because of AWS Batch Quotas Issue",
                "MaxDelaySeconds": 300,
                "JitterStrategy": "FULL"
              }
            ]

When the Step Functions workflow starts, the initial burst of AWS Batch SubmitJob API calls throttles due to exceeding TPS limits. But the built-in retry policy with capped exponential backoff and jitter allows all jobs to eventually succeed without failing. The backoff provides time for the throttling to clear, while jitter spreads out the retries to avoid more throttling. This shows how Step Functions’ retry policies can gracefully handle temporary throttling or failures.

Application state data

AWS Step Functions store application state data for each workflow invocation. The maximum size limit for this application state data is 256 kilobytes per workflow invocation. This means the total size of all data loaded into the state machine and passed across transitions must be less than 256KB for each invocation. Exceeding this 256KB limit will result in an exception and aborted execution as described in the following screenshot.

Fortunately, AWS Step Functions provide a solution to consolidate large amounts of data from child workflow executions. It aggregates all child workflow execution data, including execution inputs, outputs, and status. Step Functions export executions with the same status to their respective files in the specified Amazon S3 location. The “ResultWriter” field specifies the S3 bucket and prefix where Step Functions will write the aggregated results of all child workflows started by a Distributed Map state.

The following code describes a Step Function state part with this application state data export configuration:

"ResultWriter": {
        "Resource": "arn:aws:states:::s3:putObject",
        "Parameters": {
          "Bucket.$": "$.input.s3_bucket",
          "Prefix": "batch-ffmpeg-state-machine/results-output/"
        }

How to use it

Prerequisites

You will need the following prerequisites to set up the solution:

An AWS account
Latest version of AWS Cloud Development Kit (AWS CDK) with bootstrapping already done
Latest version of Task
Latest version of Docker
Latest version of Python 3.

Deploy the sample code

Deploy the “AWS Batch with FFMPEG“ sample code following the README file in the GitHub repository : https://github.com/aws-samples/aws-batch-with-ffmpeg#deploy-the-solution-with-aws-cdk

Use the solution

A Step Functions execution is triggered with a JSON file as an input. In our case, here is the JSON “input.json” designed for the solution:

{
  "name": "pytest-sdk-audio",
  "compute": "intel",
  "input": {
    "s3_bucket": "my-input-bucket",
    "s3_prefix": "media-assets/",
    "file_options": "null"
  },
  "output": {
    "s3_bucket": "my-output-bucket",
    "s3_prefix": "output/",
    "s3_suffix": "",
    "file_options": "-ac 1 -ar 48000"
  },
  "global": {
    "options": "null"
  }
}

Parameters of this JSON Step Function Execution input are:

$.name: metadata of this job for observability.
$.compute: Instances family used to compute the media asset : intel, arm, amd, nvidia, xilinx.
$.input.s3_bucket and $.input.s3_prefix: List of Amazon S3 Objects to be processed by FFmpeg.
$.input.file_options: FFmpeg input file options described in the FFmpeg official documentation.
$.output.s3_bucket and $.output.s3_prefix: S3 bucket and prefix where all processed media assets will be stored.
$.output.s3_suffix : Suffix to add to all processed media assets which will be stored on a Amazon S3 Bucket
$.output.file_options: FFmpeg output file options described in the official documentation.
$.global.options: FFmpeg global options described in the official documentation.

I submit the Step Function execution with the AWS CLI

aws stepfunctions start-execution — state-machine-arn arn:aws:states:.<region>:<account_id>:stateMachine:batch-ffmpeg-state-machine —name <execution-name> —input "$(jq -R . input.json —raw-output)

When the execution of this Step Functions completes, the processed media assets become available within the S3 bucket configured as “output”. The S3 path to access these media files is: s3://{$.output.s3_bucket}{$.output.s3_suffix}{Input S3 object key}{$.output.s3_suffix}.

Now I have access to millions of properly processed audio files and can proceed to train its audio LLM (Large Language Model).

Cost

AWS Batch enables optimizing compute costs by only paying for the resources used. Leveraging Spot instances allows customers to take advantage of unused EC2 capacity to achieve significant cost savings compared to On-Demand instances. It’s important to benchmark different instance types and sizes to find the optimal configuration for the workload. Testing options like GPU vs CPU helps strike the right balance between performance and cost as described in the following blog post “Optimizing video encoding with FFmpeg using NVIDIA GPU-based Amazon EC2 instances”.

Clean up

To avoid incurring unnecessary charges after testing this solution, I have to clean up the resources I created by following these steps:

Delete all objects in the Amazon S3 bucket used for testing. Remove these objects from the S3 console by selecting all objects and clicking “Delete.”
Destroy the AWS CDK stack that was deployed for testing. Open a terminal in the Git repository and run: task cdk:destroy
Verify that all resources have been removed by checking the AWS console. This ensures no resources are accidentally left running, which would lead to unexpected charges.

Summary

I leveraged an Open Source tool FFmpeg and multiple AWS services (AWS Batch, AWS Step Functions, and Amazon S3) to process millions of audio files in parallel. This serverless architecture overcame scalability and service quota challenges by combining AWS services with an open source technology:

AWS Step Functions’ distributed map enabled large-scale parallel processing of assets stored in S3.
Integrating AWS Batch into the Step Functions workflow provided scalable compute while Step Functions handled orchestration.
Error handling strategies like retries and jitter in Step Functions helped avoid overloading downstream AWS Batch when executing high volumes of jobs.

In summary, the combination of AWS Batch, Step Functions, S3, and Open Source tool like FFmpeg allowed efficient, scalable, parallel processing of millions of assets.

The following screenshot illustrates the item status processing of 2 million audio files accomplished by the team in nearly 2 days. A sequential execution would have taken several weeks to complete the same task.

Process millions of media assets with FFmpeg on AWS

Guillaume Marchand — Wed, 03 Sep 2025 07:36:52 +0000

Process millions of media assets with FFmpeg on AWS

To efficiently process the audio for training LLMs , I improved the AWS Batch with FFmpeg sample code, an open audio/video processing sample code using AWS Batch and an Open Source tool FFmpeg.

The architecture

At AWS re:Invent 2022, AWS announced the availability of a distributed map for AWS Step Functions. This new state type extended support for orchestrating large-scale parallel workloads.

The following code describes a Step Function state submitting a new job to Batch:

"SubmitJob": {
            "Type": "Task",
            "Resource": "arn:aws:states:::batch:submitJob.sync",
            "Parameters": {
              "JobName.$": "$.name",
              "JobDefinition.$": "States.Format('arn:aws:batch:<region>:<account>:job-definition/batch-ffmpeg-job-definition-{}',$.compute)",
              "JobQueue.$": "States.Format('arn:aws:batch:<region>:<account>:job-queue/batch-ffmpeg-job-queue-{}',$.compute)",
              "Parameters.$": "$"
            },
            "End": true,
          }

For each map task, the state machine invokes a Batch job to leverage FFmpeg tool and perform audio encoding.

The following design shows how AWS Batch handles compute provisioning and scheduling, while Step Functions orchestrates the workflow — all in a serverless model.

Beyond the limit

Retry mechanism

The following code describes a Step Function state part with this retry mechanism:

"Retry": [
              {
                "ErrorEquals": [
                  "States.ALL"
                ],
                "BackoffRate": 3,
                "IntervalSeconds": 180,
                "MaxAttempts": 10,
                "Comment": "retry because of AWS Batch Quotas Issue",
                "MaxDelaySeconds": 300,
                "JitterStrategy": "FULL"
              }
            ]

Application state data

The following code describes a Step Function state part with this application state data export configuration:

"ResultWriter": {
        "Resource": "arn:aws:states:::s3:putObject",
        "Parameters": {
          "Bucket.$": "$.input.s3_bucket",
          "Prefix": "batch-ffmpeg-state-machine/results-output/"
        }

How to use it

Prerequisites

You will need the following prerequisites to set up the solution:

An AWS account
Latest version of AWS Cloud Development Kit (AWS CDK) with bootstrapping already done
Latest version of Task
Latest version of Docker
Latest version of Python 3.

Deploy the sample code

Deploy the “AWS Batch with FFMPEG“ sample code following the README file in the GitHub repository : https://github.com/aws-samples/aws-batch-with-ffmpeg#deploy-the-solution-with-aws-cdk

Use the solution

A Step Functions execution is triggered with a JSON file as an input. In our case, here is the JSON “input.json” designed for the solution:

{
  "name": "pytest-sdk-audio",
  "compute": "intel",
  "input": {
    "s3_bucket": "my-input-bucket",
    "s3_prefix": "media-assets/",
    "file_options": "null"
  },
  "output": {
    "s3_bucket": "my-output-bucket",
    "s3_prefix": "output/",
    "s3_suffix": "",
    "file_options": "-ac 1 -ar 48000"
  },
  "global": {
    "options": "null"
  }
}

Parameters of this JSON Step Function Execution input are:

$.name: metadata of this job for observability.
$.compute: Instances family used to compute the media asset : intel, arm, amd, nvidia, xilinx.
$.input.s3_bucket and $.input.s3_prefix: List of Amazon S3 Objects to be processed by FFmpeg.
$.input.file_options: FFmpeg input file options described in the FFmpeg official documentation.
$.output.s3_bucket and $.output.s3_prefix: S3 bucket and prefix where all processed media assets will be stored.
$.output.s3_suffix : Suffix to add to all processed media assets which will be stored on a Amazon S3 Bucket
$.output.file_options: FFmpeg output file options described in the official documentation.
$.global.options: FFmpeg global options described in the official documentation.

I submit the Step Function execution with the AWS CLI

aws stepfunctions start-execution — state-machine-arn arn:aws:states:.<region>:<account_id>:stateMachine:batch-ffmpeg-state-machine —name <execution-name> —input "$(jq -R . input.json —raw-output)

Now I have access to millions of properly processed audio files and can proceed to train its audio LLM (Large Language Model).

Cost

Clean up

To avoid incurring unnecessary charges after testing this solution, I have to clean up the resources I created by following these steps:

Delete all objects in the Amazon S3 bucket used for testing. Remove these objects from the S3 console by selecting all objects and clicking “Delete.”
Destroy the AWS CDK stack that was deployed for testing. Open a terminal in the Git repository and run: task cdk:destroy
Verify that all resources have been removed by checking the AWS console. This ensures no resources are accidentally left running, which would lead to unexpected charges.

Summary

AWS Step Functions’ distributed map enabled large-scale parallel processing of assets stored in S3.
Integrating AWS Batch into the Step Functions workflow provided scalable compute while Step Functions handled orchestration.
Error handling strategies like retries and jitter in Step Functions helped avoid overloading downstream AWS Batch when executing high volumes of jobs.

In summary, the combination of AWS Batch, Step Functions, S3, and Open Source tool like FFmpeg allowed efficient, scalable, parallel processing of millions of assets.

Automate Cloud Resource Management for Scheduled Events

Guillaume Marchand — Tue, 02 Sep 2025 09:58:26 +0000

Organizations across industries face operational challenges when managing planned high-traffic events. Business teams must create IT support tickets for each event, requiring DevOps teams to manually provision and scale resources. This process creates bottlenecks that impact customer experience and operational efficiency.

The "Event Scheduling on AWS" implementation sample addresses these challenges by automating resource provisioning and scaling for planned events. This sample enables organizations to deliver exceptional customer experiences during high-demand periods while reducing operational costs and eliminating manual coordination between business and technical teams.

Understanding the Challenge

Enterprises across multiple industries face similar operational obstacles when managing planned events. A media company planning a live sports broadcast must coordinate with DevOps teams weeks in advance, creating support tickets and requiring technical staff presence during events for manual scaling operations.

Retail organizations preparing flash sales encounter comparable challenges. E-commerce platforms must provision additional compute capacity, configure content delivery networks (CDNs), and scale database resources to handle sudden traffic spikes during promotional events. Manual coordination between marketing and technical teams creates delays and increases the risk of failures.

Gaming companies launching new titles or hosting e-sports tournaments face infrastructure scaling complexities. They must coordinate server provisioning across multiple regions, configure matchmaking services, and ensure backend systems can handle concurrent player loads. The manual nature of these operations often results in poor player experiences during peak gaming events.

Financial services organizations managing trading platform events encounter similar operational bottlenecks. Market events, earnings announcements, and regulatory changes require rapid infrastructure adjustments to handle increased trading volumes. Manual provisioning processes create delays that can impact trading performance and customer satisfaction.

These manual approaches increase operational overhead and extend planning timelines across industries. The risks of human error increase, IT resource utilization becomes inefficient, and organizations struggle to scale their operations effectively. These factors directly impact service quality during business-critical moments.

Solution Architecture

The "Event Scheduling on AWS" implementation sample uses an event-driven architecture built on AWS serverless technologies. AWS Step Functions orchestrates workflows through preparation, provisioning, configuration, and cleanup phases. Amazon EventBridge manages event scheduling and message coordination.

The Step Functions workflow implements a complete event lifecycle through preroll and postroll phases. The preroll phase executes before the event, managing resource provisioning, configuration validation, and system preparation tasks. During this phase, the platform deploys AWS Service Catalog products or executes Systems Manager documents for infrastructure scaling, and performs checks to ensure readiness.

The postroll phase activates after event completion, managing resource cleanup, and operational reporting. This phase terminates temporary resources, reduces infrastructure to baseline levels. The preroll and postroll approach ensures consistent event execution while optimizing resource utilization and operational costs.

AWS Service Catalog manages infrastructure deployments while AWS Systems Manager executes automation workflows. AWS AppSync provides the GraphQL API layer and Amazon CloudWatch delivers comprehensive monitoring and alerting capabilities.

The platform integrates with existing AWS services through standardized tags and IAM policies. Resources tagged with application=event-scheduling-platform become available for orchestration, enabling seamless integration with current infrastructure.

Key Organizational Benefits

Business teams gain self-service capabilities to schedule events without dependencies on IT teams. They work with familiar business metrics such as audience size and performance targets while tracking event status in real-time. This approach removes bottlenecks and accelerates event planning cycles.

Operational teams maintain visibility and control despite increased business team independence. The integration with AWS Chatbot delivers real-time notifications to Microsoft Teams and Slack channels, ensuring operational staff remain informed of all event activities. Teams receive alerts for event scheduling, resource provisioning status, execution progress, and completion notifications.

This notification system allows operational teams to monitor business-initiated events without requiring direct involvement in routine operations. They can respond quickly to issues while enabling business teams to operate independently for standard event scenarios. The integration preserves operational oversight while eliminating manual coordination bottlenecks.

DevOps teams create reusable infrastructure templates through AWS Service Catalog products and Systems Manager documents. Automation reduces manual intervention, allowing technical personnel to focus on platform improvements rather than routine operational tasks.

The implementation sample optimizes costs through automatic resource cleanup after events.

Open Architecture with AWS Service Integration

The "Event Scheduling" implementation sample provides an open framework that packages industry-specific use cases through established AWS services. This approach ensures organizations can leverage existing AWS capabilities while maintaining operational consistency and security standards.

AWS Systems Manager Integration

AWS Systems Manager documents encapsulate operational procedures for different industries. Gaming companies can create documents that scale Amazon EC2 Auto Scaling groups and configure Amazon ElastiCache clusters for tournament events. Financial services organizations can develop documents that adjust Amazon RDS replicas and modify AWS Lambda concurrency limits during trading events.

Each Systems Manager document includes built-in audit capabilities through AWS CloudTrail integration. Execution history, parameter changes, and resource modifications are automatically logged, providing complete traceability for compliance requirements. The service manages document versioning, rollback capabilities, ensuring reliable automation across distributed environments.

Kubernetes Integration for Containerized Workloads

Organizations running containerized applications can implement SSM documents that integrate with Amazon EKS clusters. These documents address the technical challenge of authenticating with EKS clusters and interacting with the Kubernetes API from SSM automation workflows.

The SSM document would enable resource pre-warming via Kubernetes Horizontal Pod Autoscaler (HPA) or Deployment for existing EKS clusters. This document would support both pre-roll actions that scale deployments before events and post-roll actions that reduce resources afterward. The integration leverages AWS Lambda functions within the SSM document to execute Kubernetes client operations, ensuring secure authentication and reliable API interactions.

AWS Service Catalog Product Packaging

AWS Service Catalog products package complete infrastructure solutions for specific event types. Service Catalog provides governance through launch constraints, template constraints, and notification constraints. Product portfolios enable different organizational units to access appropriate infrastructure templates while preventing unauthorized resource creation.

Live Video Streaming with SRT Source

Media organizations could deploy complete live video streaming infrastructure using a Service Catalog product that provisions AWS Media services. This product would address the technical challenge of provisioning live video workflows with predefined SRT input endpoints and HLS/DASH output endpoints that integrate with existing information systems.

The product would include MediaConnect for SRT source ingestion, MediaLive for video processing, MediaPackage for content packaging, and CloudFront for content delivery. Route 53 would provide predictable domain names while AWS Certificate Manager would manage TLS certificates tied to CloudFront distributions. This comprehensive approach could ensure reliable video delivery with minimal manual configuration.

Existing AWS Solutions Integration

Organizations can leverage existing AWS architectures by integrating CloudFormation or Terraform templates into Service Catalog products.

This approach allows organizations to build upon proven AWS architectures while adding custom configurations and governance controls. The integration maintains the benefits of AWS architectures while providing the automation and scheduling capabilities of the "Event Scheduling" implementation sample.

Operational Reliability

AWS Step Functions provides reliable workflow orchestration with built-in error handling, retry logic, and state management. The service automatically manages transient failures and provides detailed execution history for troubleshooting. The integration with Amazon EventBridge ensures event scheduling remains accurate even during service interruptions.

Amazon CloudWatch monitors all platform components with custom metrics, alarms, and dashboards. Organizations can track event success rates, resource provisioning times, and cost optimization metrics. AWS X-Ray provides distributed tracing capabilities for complex multi-service event workflows.

Getting Started

Follow the README.md in the GitHub project for implementation guidance.

Resource Integration

Existing AWS Service Catalog products and Systems Manager documents integrate with the platform through proper tagging. Resources must include the application=event-scheduling-platform tag to become discoverable by the orchestration system.

The platform provides automated registration scripts that discover and register properly tagged resources. This eliminates manual registration requirements and ensures consistent integration across existing infrastructure.

An audit tool validates configuration and identifies common integration issues. The tool checks IAM permissions, Service Catalog configuration, Systems Manager configuration, and resource tagging to ensure proper platform operation.

Cost Considerations

The solution uses pay-per-use AWS services including DynamoDB, Lambda, Step Functions, AppSync, and CloudWatch. Costs scale with usage patterns and event frequency rather than requiring fixed infrastructure investments.

Automatic resource cleanup after events prevents unnecessary charges from orphaned resources. The architecture design minimizes state transitions and optimizes service usage to control operational expenses.

Next Steps

The "Event Scheduling" implementation sample enables organizations to transform their event management operations by eliminating manual processes and reducing operational overhead. Organizations benefit from faster event deployment cycles, improved resource utilization, and enhanced customer experiences during business-critical moments.

Explore the open-source implementation to understand integration patterns and architectural decisions. Comprehensive documentation includes deployment guides, troubleshooting resources, and extension samples.

Start with the sample resources to understand platform capabilities, then progressively integrate existing infrastructure through proper tagging and registration processes. Audit tools help ensure successful integration and identify optimization opportunities.