Building an Autonomous AI News Agent on Step Functions and AgentCore

TL;DR

• A daily pipeline fetches 24 curated RSS feeds, clusters items describing the same real-world story across sources, categorizes survivors into nine fixed buckets with one LLM call, and writes a ranked digest — no watchlist, no web search, no human review.
• The first design tracked 77 named companies and investors and searched the web for each one. Measurement killed it: only 8 of 77 ever had a working official RSS feed, and the curated feed list surfaced the same funding rounds and incidents anyway, with no targeting at all.
• One "synthesizer" invocation that deduped, categorized, extracted funding data, and wrote prose all in a single call hit AgentCore Runtime's real synchronous ceiling — not the 8-hour session limit the container's own error message implied. Splitting into analyze → write → digest fixed it.
• A Step Functions Map fanning out per-category writers hit a second real bug: $$.Map.Item.Value only resolves inside the Map's own ItemSelector, not inside a nested Task's Payload. Neither bug showed up in 79 passing unit, contract, and integration tests — only a real deployed execution surfaced them.
• The whole system holds state in flat JSON and Markdown files in one S3 bucket. No database. A story-thread index is the only thing that persists across days, and it's checked and pruned on every run, not on a schedule.

---

The wrong unit of work

The first version of this pipeline tracked entities. A watchlist of AI labs, chip makers, and investors — 77 of them — each with its own scan session, each hitting AgentCore Web Search to ask "what happened with this company today." The architecture made sense on paper: bounded, per-entity context; a clean map-reduce shape; isolation so one company's noisy news couldn't bleed into another's summary.

Then the watchlist got measured. Of 77 tracked entities, 8 had a working official RSS feed. The other 69 depended entirely on web search returning something relevant on a given day, for a company that might not have shipped anything that week. Meanwhile, a small set of broad industry-news aggregators — the outlets that already do the work of deciding what's newsworthy — covered the same funding rounds and product launches, correctly attributed, with no targeting at all.

The fix wasn't a better watchlist. It was dropping the watchlist entirely. Twenty-four curated feeds — eight official company blogs, sixteen aggregators, replace a per-entity scan with one fetch that reasons across everything at once. AgentCore Web Search came out with it; there was no longer an entity-scoped gap for it to fill. Coverage went up because attribution was already being done for me, by outlets whose job is to do it.

This is the general shape of the mistake: modeling the problem after the tracked entity, when the actual unit of coverage was the story, and the story doesn't care which company you're watching.

What the pipeline does, once a day

flowchart TD
    A[EventBridge Scheduler<br/>00:30 daily] --> B[Scanner<br/>fetch 24 RSS feeds]
    B --> C[Analyze<br/>dedup, categorize, rank]
    C --> D{Map over categories<br/>with surviving items}
    D --> E1[Writer: category 1]
    D --> E2[Writer: category 2]
    D --> E3[Writer: category N]
    E1 --> F[Digest<br/>top stories + assembly]
    E2 --> F
    E3 --> F
    F --> G[Publish<br/>kill-switch check]
    G --> H[(published/ in S3)]

Scanner fetches every feed unconditionally — no per-source filtering — normalizes each item's date into a comparable ISO-8601 UTC string, drops anything older than 30 days as a cheap staleness guard, and writes the raw batch to S3. No LLM call happens here at all.

Analyze does the actual reasoning, in five steps, none of which write a sentence of prose:

1. Cross-feed dedup. Canonicalize URLs, then cluster items across every feed describing the same real-world event — URL match first, embedding similarity as a fallback. The number of independent sources covering a story becomes a corroboration signal, carried forward rather than discarded once dedup is done.
2. Funding extraction. Clusters that look like funding rounds get a structured extraction call — company, investors, amount, stage — and get force-categorized into a Funding & Capital bucket regardless of what general categorization would pick.
3. Categorization. One LLM call reads every surviving cluster and assigns it to exactly one of nine fixed categories. Closed vocabulary, not open-ended — a genuinely irrelevant cluster gets dropped, not stretched into a category it doesn't belong in.
4. Story-thread classification. Each item gets checked against a global, 14-day trailing index of everything already published. Below a similarity threshold, it's classified directly as new, duplicate, or update; the ambiguous middle band gets one LLM adjudication call. Duplicates are dropped; updates get a reference appended, not a rewritten paragraph.
5. Rank, cap, and select top stories. Per category, take the top N ranked items into a primary section and the next few into an overflow section. Separately, one call reads across every surviving category and picks the handful of stories that matter regardless of which bucket they landed in.

Writer and Digest are where prose gets written, and they're deliberately split from Analyze and from each other — the reason why is the whole second half of this post.

The ceiling that wasn't the one I expected

The first working version of this pipeline had one "synthesizer" stage: dedup, funding extraction, categorization, story-threading, ranking, and prose-writing for every category, all inside a single AgentCore Runtime invocation. It ran fine in testing. It failed on a real deployed run with:

BedrockAgentCore.RuntimeClientErrorException: Runtime initialization time
exceeded. Please make sure that initialization completes in 120s.

That error message is about cold starts. It was not a cold start. The actual cause: AgentCore Runtime runs a /ping health-check thread alongside the entrypoint handler, and a long synchronous sequence of LLM calls — dedup, then five or six structured extractions, then a prose loop over every category — blocks that thread long enough for the platform to conclude the container is unhealthy and kill it.

The number that actually matters isn't a session lifetime measured in hours. AgentCore Runtime's own service quotas list two separate limits: a 15-minute request timeout for synchronous requests, and an 8-hour maximum for asynchronous jobs — neither adjustable. A session can live for hours. One synchronous invocation inside it cannot run past 15 minutes and still return a response.

The fix was architectural, not a bigger timeout: split the one long invocation into three chained stages, each well clear of the ceiling.

flowchart LR
    A[Analyze<br/>dedup + categorize + rank<br/>no prose] --> B[Writer x N<br/>one category per call<br/>≤5 prose calls each]
    B --> C[Digest<br/>top stories + assemble<br/>reads back every draft]

Analyze does everything that doesn't require writing sentences — the fast half of what used to be one call. Writer renders exactly one category's Markdown per invocation, fanned out by a small Step Functions Map over however many categories actually survived that day (at most nine, discovered at run time, not a static config list). Digest runs once after every writer invocation finishes, writes the cross-category "top stories" prose, and reassembles the full digest by reading back each category's already-written draft from S3 — because nothing survives in memory across separate invocations; if the next stage needs it, it has to be a file.

The general pattern: a monolithic long-running call scales badly against a synchronous platform ceiling. Splitting into bounded, single-purpose stages that each stay comfortably under the ceiling — and pushing state through a shared object store instead of holding it in memory — is the same shape AWS's own reference samples use for multi-agent Step Functions pipelines. It's a more boring architecture. It's also the one that doesn't get killed mid-run.

The bug that 79 passing tests didn't catch

Splitting the pipeline into three stages meant fanning out per-category writers with a Step Functions Map. The obvious way to pass the current category into each Map iteration's nested Task is to reference the Map's own context variable directly:

{
  "Payload": {
    "category.$": "$$.Map.Item.Value"
  }
}

This synthesizes cleanly. It passes cdk synth. It passes every unit test, because unit tests exercise the Python handler, not the state machine's actual JSON. It fails at runtime with States.Runtime: The JSONPath '$$.Map.Item.Value' specified for the field 'category' could not be found in the input — because $$.Map.Item.Value is only resolvable inside the enclosing Map state's own ItemSelector, not inside a Task nested in ItemProcessor. The fix is to resolve it one level up and forward it as a plain reference:

// In the Map's own ItemSelector:
{ "category.$": "$$.Map.Item.Value" }

// In the nested Task's Payload, one level down:
{ "category.$": "$.category" }

Neither this bug nor the timeout ceiling above showed up in 79 passing unit, contract, and integration tests before deployment. Both required a real Step Functions execution against a real AgentCore Runtime container to surface. Tests verify that code does what the code says. They don't verify that a platform's actual behavior matches its documentation, or that a JSONPath scoping rule holds the way the mental model assumes. For infrastructure this deep in a managed platform's own execution semantics, a green test suite is a necessary check, not a sufficient one — the deployed system still has to run for real, and the output still has to get read, not merely checked for non-emptiness.

What actually persists

There's no database anywhere in this pipeline. Every handoff between stages is a flat file in one S3 bucket:

flowchart TD
    RSS["config/rss-feeds.yaml<br/>24 feeds"] --> Scanner
    Scanner -->|writes| Raw["signals/{date}/raw-items.json<br/>ephemeral, one run"]
    Raw --> Analyze
    Index[("state/stories-index.json<br/>global, 14-day lookback, forever")] <-->|read + write| Analyze
    Analyze -->|writes| Ranked["state/{date}/ranked-sections.json<br/>no prose, one run"]
    Ranked --> Writer
    Writer -->|writes, one file per category| Drafts["drafts/{category}/{date}.md"]
    Drafts --> Digest
    Digest -->|writes| DailyDigest["drafts/daily-digest/{date}.md"]
    DailyDigest --> Publish
    Publish -->|copies| Published[("published/<br/>terminal output, forever")]

The one piece of state that outlives a single run is state/stories-index.json — a global, append-only record of every story published in the last 14 days, checked on every run so the same event doesn't get written up twice as it develops across days. Everything else — raw feed items, the day's ranked sections, individual drafts — is scoped to one run and never read again once the digest is assembled.

The reasoning for staying file-based instead of introducing a database: at this pipeline's actual volume — 150 to 300 raw items a day, collapsing to 40 to 80 clusters after dedup — full-file reads aren't a bottleneck, and flat files in S3 are free to version, cheap to store, and trivial to inspect with a plain aws s3 cp. A database adds schemas, migrations, and connection handling to solve a query pattern that "read the whole file" already serves at this scale. The rule isn't "never use a database" — it's don't introduce one preemptively, and revisit only when a specific, measured limitation of the flat-file pattern actually gets hit.

Model tiering, and the model that didn't get replaced

Every LLM-calling stage uses one frontier-tier model for both structured extraction and prose — categorization, funding extraction, story-thread adjudication, and every "write this section" call. Nothing in this pipeline runs on a cheap tier today; the scan stage that would have used one makes no LLM call at all post-pivot, since fetching RSS feeds is pure mechanism, not judgment.

The obvious move, once this was running inside AWS infrastructure, was assuming a Bedrock-native model would be cheaper. It wasn't. Checked directly against OpenRouter's own management API and AWS's Bedrock pricing API: the frontier model already in use runs at roughly $0.44 per million input tokens and $0.87 per million output, against $1.00 and $5.00 for Bedrock's equivalent-tier Claude model on the same account. Same story for embeddings — cost parity between the OpenAI embedding model in use and Bedrock's Titan embedding model, at a scale of dozens to low-hundreds of vectors per run with a brute-force linear scan and no vector database — meaning a swap wouldn't change anything measurable either way.

The lesson isn't "OpenRouter beats Bedrock." It's that "we're already on AWS, so the AWS-native option must be cheaper" is a real assumption worth checking against actual current pricing before acting on it, not a fact.

So what

The design that survived contact with real deployment isn't the one that looked cleanest on a whiteboard. It's the one that got measured — a watchlist that turned out to be 8-of-77 useful, a synchronous ceiling that turned out to be 15 minutes rather than 8 hours, a JSONPath scoping rule that only a live execution would surface, a pricing assumption that didn't hold once checked. None of those were visible from the architecture diagram alone.

The pattern underneath: build the pipeline to be cheap to run end-to-end for real, then let real executions — not test suites, not synthesized templates — tell you where the design's assumptions were wrong. A green build says the code does what it claims to do. Only a deployed run with real output, read and judged rather than merely checked for non-emptiness, tells you the architecture was actually right.