DEV Community: Amit

The AI Chips After NVIDIA

Amit — Wed, 05 Aug 2026 05:44:59 +0000

NVIDIA does not have one competitor. It has a queue of startups and hyperscaler platforms attacking different parts of the AI compute problem.

That distinction matters. A wafer-scale processor, a transformer-specific ASIC, a RISC-V accelerator, and a photonic interconnect are not interchangeable products. They solve different constraints, sell into different parts of the stack, and require different kinds of proof before a customer can trust them with production workloads.

The useful question is not, “Which startup beats NVIDIA?” The useful question is, “Which bottleneck is each company removing, and what does that make possible?”

The comparison model

The market map asks five questions: what workload the product targets, which bottleneck it attacks, whether it replaces or complements a GPU, how much software travels with it, and what evidence exists beyond a chip announcement.

The fifth question is the filter. AI hardware companies can raise large rounds before their products reach volume production. A private valuation measures investor expectations at a point in time; it does not prove performance or customer retention. For performance, MLPerf Inference is a useful reference because it separates latency, throughput, quality targets, and system power.

The market is splitting around inference

Training builds a model. Inference serves it to users. Training rewards parallel compute and software flexibility; inference adds time to first token, tokens per second, memory bandwidth, power per request, and cost per query.

That creates room for specialized architectures. A chip does not need to replace every GPU if it wins a workload where the GPU carries more flexibility than the customer needs.

The companies below take different positions on that trade.

The hyperscaler platforms

The startup map is incomplete without the other large-scale accelerator platforms. NVIDIA Vera Rubin NVL72 is the current NVIDIA reference point, while Google Ironwood and Amazon Trainium3 represent cloud providers building their own alternatives.

NVIDIA Vera Rubin NVL72: the rack is the product

Vera Rubin is not a single GPU release. The NVL72 is a rack-scale system with 72 Rubin GPUs, 36 Vera CPUs, and NVLink 6 for high-speed GPU-to-GPU communication. Each Rubin GPU carries 288 GB of HBM4 with 22 TB/s of memory bandwidth. Its NVLink 6 fabric provides 3.6 TB/s of all-to-all GPU-to-GPU scale-up bandwidth per GPU, or about 260 TB/s across the rack. These are different numbers: HBM bandwidth moves data between a GPU and its local memory; NVLink moves data between GPUs.

The rack scales across systems through NVIDIA's Quantum-X800 InfiniBand and Spectrum-X Ethernet. That networking lineage came through NVIDIA's acquisition of Mellanox, whose InfiniBand and Ethernet products became the foundation of NVIDIA's networking business. Quantum-X800 and Spectrum-X are newer NVIDIA product families; NVLink remains NVIDIA's separate GPU scale-up fabric.

The architecture extends NVIDIA's existing advantage: general-purpose accelerators, a tightly integrated scale-up fabric, networking, CPUs, system software, and CUDA compatibility. For inference, NVIDIA also positions the Groq 3 LPX rack beside Vera Rubin for low-latency decode and large-context workloads. Rubin GPUs provide general-purpose compute and attention processing, while Groq-derived LPUs in the LPX rack accelerate latency-sensitive feed-forward decode. NVIDIA Dynamo coordinates the two, combining broad GPU flexibility with specialized low-latency inference in one system. CUDA remains the foundation for the broader platform.

NVIDIA claims one-fourth the GPU count for training some mixture-of-experts models and one-tenth the cost per million tokens versus Blackwell. Those are vendor comparisons, not neutral benchmark results. The strategic point is more durable: NVIDIA is competing at rack and data-center level, where a startup must replace an entire system relationship rather than one accelerator card.

Google Ironwood: a TPU pod built around co-design

Google Ironwood, also called TPU7x, is Google's seventh-generation TPU and its first generation explicitly focused on inference. Each chip has 192 GiB of HBM3E and about 7.4 TB/s of memory bandwidth. A pod scales to 9,216 chips through Google's Inter-Chip Interconnect, optical circuit switching, data-center networking, and liquid cooling.

Ironwood's differentiator is system-level co-design. The hardware, XLA compiler and Pallas kernel system are built together, and Google presents the pod as one large computer rather than a loose collection of accelerators. The trade is portability: Google Cloud access, JAX and PyTorch support, and a programming model that differs from CUDA.

Amazon Trainium3: cloud-native scale with Neuron

Trainium3 is the fourth-generation AWS AI chip. A Trainium3 chip provides 2.52 FP8 petaflops, 144 GB of HBM3e, and 4.9 TB/s of memory bandwidth. Trn3 UltraServers scale to 144 chips using NeuronSwitch-v1, and UltraCluster 3.0 connects much larger deployments through a non-blocking, petabit-scale network.

Trainium3's architectural advantage is not only the chip. It is the combination of Trainium, the AWS Neuron SDK, and cloud integration across services such as Amazon Bedrock, SageMaker, EKS, and Batch. Neuron supports PyTorch and JAX, while the Neuron Kernel Interface provides a lower-level path for custom kernels.

Trainium3 is therefore a cloud-native alternative rather than a general market card. Its strength is cost and scale inside the AWS environment. Its trade is that a customer adopting it is also adopting the Neuron software path and the operating model of that cloud.

Platform	Primary advantage	Main trade
NVIDIA Vera Rubin NVL72	Broadest accelerator, networking, and software platform	Highest system complexity and dependence on NVIDIA's stack
Google Ironwood TPU7x	Tight hardware, compiler, and pod co-design	Google Cloud and XLA-oriented programming model
Amazon Trainium3 / Trn3	Cloud-native scale and Neuron economics	AWS-specific hardware and software path
Startup accelerators	Narrow optimization for latency, memory, power, or openness	Smaller ecosystem and less proven deployment scale

1. Cerebras: put the model on one enormous processor

Cerebras takes the most visible architectural departure from the GPU cluster. Its Wafer-Scale Engine uses an entire silicon wafer as a single processor, with a large on-chip memory system and a fabric connecting the compute elements.

The bet is that moving data between many separate chips creates a large part of the latency and system complexity. A wafer-scale system keeps more of the working set close to the compute and reduces the need to coordinate a large collection of discrete accelerators.

Cerebras is one of the strongest examples of a startup moving beyond a chip into a complete infrastructure product. Its OpenAI agreement describes a planned 750 megawatts of wafer-scale systems for high-speed inference. The company also announced a partnership with AMD that combines AMD rack-scale systems for throughput with Cerebras systems for token generation.

Recent announcements make the scale-up challenge more concrete. Cerebras says it will add 200 megawatts of European AI compute capacity by the end of 2027, while third-party coverage places the OpenAI agreement above $20 billion over multiple years. The official announcement describes the deal as a multi-year 750-megawatt deployment; the larger figure is a reported commercial value, not a published contract document.

The trade is extraordinary local memory and low latency in exchange for a less conventional manufacturing, programming, and deployment model. Cerebras is not a drop-in GPU. It is a different computer.

2. SambaNova: sell the system, not only the silicon

SambaNova packages custom processors, memory, networking, and software into enterprise AI systems. Its pitch is less about selling a component to an engineering team and more about providing a system that can be deployed for controlled inference workloads.

The company raised $1 billion at an $11 billion valuation in July 2026. The same announcement describes expansion across enterprises, neoclouds, sovereign customers, and service providers. JPMorgan Chase selected SambaNova as an inference infrastructure partner, according to the company.

SambaNova's architecture is aimed at the part of inference where memory movement and serving cost dominate. Alternative accelerators often enter the market as part of a heterogeneous system rather than replacing every GPU.

3. d-Matrix: move compute into memory

d-Matrix is attacking the memory wall directly. Its digital in-memory computing approach places more of the required computation near the data instead of repeatedly moving model weights between separate memory and compute units.

The company raised $275 million in Series C funding at a $2 billion valuation. In June 2026, it announced that its Corsair platform entered full production, with volume shipments planned for selected hyperscale, neocloud, and frontier-lab customers.

d-Matrix is also explicit about an architecture that works alongside GPUs. Its company-reported testing describes a heterogeneous configuration in which Corsair handles part of inference beside GPU infrastructure.

That position became more concrete in 2026. d-Matrix acquired GigaIO's data-center business to add rack-scale systems and interconnect expertise, then announced a Parasail deployment pairing Corsair with NVIDIA Hopper and Blackwell systems. The company is no longer presenting only a chip; it is assembling the deployment layer around a heterogeneous rack.

The product becomes useful when its compilers, runtimes, and fallback behavior cover the model graphs customers actually run.

4. Tenstorrent: make the stack more open

Tenstorrent is taking a different route. Its Blackhole processors combine Tensix AI cores with RISC-V cores, and the company publishes an open software stack that gives developers access to lower layers of the system. Its Blackhole cards are listed for purchase, including a model priced at $999 with 120 Tensix cores and up to 32 GB of GDDR6 memory.

The architectural bet is not only about raw throughput. It is also about control: developers can inspect and tune more of the machine rather than treating the accelerator as a sealed appliance. That creates a software burden because NVIDIA's advantage includes the accumulated compatibility of CUDA libraries, frameworks, profilers, kernels, and developer habits.

Tenstorrent therefore competes on a different axis: lower entry cost, RISC-V flexibility, and a more inspectable stack. The latest product update is larger than a card launch. Tenstorrent announced general availability of Galaxy Blackhole, a 32-chip air-cooled system starting at $110,000, with a four-system supercluster starting at $440,000. The company also says Galaxy is shipping in volume and has been deployed in multi-server configurations.

Tenstorrent is moving from an openness thesis toward a systems thesis: standard Ethernet, RISC-V-based processors, public software, and rack-scale deployment.

5. Etched: specialize for transformers

Etched is making the narrowest bet in this group. The company is building an ASIC optimized for transformer models, the architecture behind many current large language models.

The benefit of specialization is straightforward: remove general-purpose features and use the saved area and power budget for a targeted workload. The cost is equally straightforward: model architectures change.

TechCrunch reported that Etched closed a $300 million Series C at a $10.3 billion valuation in July 2026. The same report says Etched has manufactured its chips, is testing full systems with clients, and has booked $1 billion in orders. Those are stronger maturity signals than a funding announcement, but they remain company-reported claims rather than independently audited revenue.

6. Fractile: treat memory as the compute surface

Fractile is another memory-centric architecture, but with a different implementation path. The London startup is building in-memory computing hardware intended to perform more of the inference arithmetic where the model weights are stored.

The company raised $220 million in Series B funding in May 2026. Recent trade coverage puts the first-chip target in the second half of 2026, earlier than the 2027 launch window cited in earlier coverage. That timeline needs confirmation from Fractile before it becomes a firm milestone.

The architecture is attractive because inference repeatedly reads model weights. Its risk is software coverage across changing operators, quantization schemes, and model families.

7. Positron and FuriosaAI: win on efficiency and deployment

Positron is building accelerator and memory products around high-throughput inference. TechCrunch reported a $230 million Series B and a plan to bring its Asimov chip into production in early 2027. The company's current product and company pages distinguish the products clearly: Atlas is already shipping, while Asimov is the future custom-silicon product with more than 2 TB of memory per chip.

FuriosaAI is taking a similar efficiency-oriented position from South Korea. Its RNGD inference accelerator is aimed at reducing the power and system cost of serving models, and the company announced a $125 million funding round to scale production. In May 2026, FuriosaAI announced a partnership with Broadcom for a third-generation inference platform while stating that RNGD had entered mass production and had been validated by Samsung SDS and LG AI Research.

The relevant comparison is whether a customer can meet its latency and quality target with fewer racks, lower power, lower cost, or a better deployment footprint.

8. Lightmatter: attack the network around the chip

Lightmatter is building photonic computing and optical interconnect technology. Its core idea is that moving data between processors becomes a larger constraint as AI clusters grow. Photons can move signals through optical paths with less electrical loss and higher bandwidth density.

This is not the same product category as Cerebras or d-Matrix. Lightmatter can become valuable even when the compute remains NVIDIA-based. The company has also joined NVIDIA's NVLink Fusion ecosystem.

Reuters reported that Lightmatter raised $400 million at a $4.4 billion valuation. Its position is closer to making every accelerator cluster scale better than replacing the incumbent accelerator.

The comparison table

Company	Primary target	Core architectural idea	Relationship to GPUs	Latest public funding or valuation signal	Maturity signal
Cerebras	Training and inference	Wafer-scale processor and large local memory	Alternative system; can be paired with other infrastructure	750 MW OpenAI deployment agreement	Public listing and deployed systems
SambaNova	Enterprise inference	Full-stack accelerator systems	Alternative system	$1B round at $11B valuation	Enterprise and financial-services deployments
d-Matrix	Data-center inference	Digital in-memory compute	Works beside GPUs in heterogeneous systems	$275M round at $2B valuation	Corsair announced in full production
Tenstorrent	Training and inference	RISC-V plus Tensix AI cores	Alternative and accelerator-card path	Product pricing is public; valuation not used here	Galaxy systems in general availability
Etched	Transformer inference	Transformer-specific ASIC	Narrow replacement for selected workloads	$300M round at $10.3B valuation	First-silicon and contract claims require continued validation
Fractile	Inference	In-memory computing	Alternative accelerator	$220M Series B	First chip target reported for late 2026; confirm with company
Positron	Inference	Memory and accelerator system	Alternative accelerator	$230M Series B at $1B+ valuation	Atlas shipping; Asimov production targeted for 2027
FuriosaAI	Inference	Power-efficient accelerator	Alternative accelerator	$125M funding round	RNGD in mass production; Broadcom platform in development
Lightmatter	Interconnect	Photonic data movement	Complements GPU clusters	$400M round at $4.4B valuation	Partner ecosystem and optical products

There is no single “NVIDIA alternative” architecture. The specialized bets target the parts of the system where general-purpose GPUs carry too much cost or complexity.

What's missing

The market still lacks a universal comparison for production AI systems.

Tokens per second is not enough. A system can produce tokens quickly while using a large amount of power, supporting a narrow model family, requiring a custom compiler, or failing under real concurrency. A low-latency demo can also hide the cost of loading weights, moving data between stages, and keeping the system utilized.

The comparison needs six measurements:

time to first token
sustained output tokens per second
cost per million output tokens
energy per million output tokens
model and operator coverage
software migration effort

Those measurements need to be collected on the same model, prompt mix, quality target, concurrency level, and end-to-end system boundary. Until then, the market is full of company claims that answer different questions.

So what

The next AI hardware market will not be won by one chip that replaces every GPU.

It will be won by a portfolio of architectures. General-purpose GPUs will remain the broad platform for workloads that change quickly. Specialized processors will take slices where latency, memory movement, power, or cost matter more than flexibility. Optical interconnects will make larger clusters possible. Open architectures will give some customers more control over the stack.

That is the practical way to read valuations. A high valuation says that investors believe a bottleneck is valuable. It does not say the architecture has won.

The better question for any AI chip startup is simple: what constraint does this company remove, and can it prove the result on a production workload without asking the customer to rebuild the entire software ecosystem?

That is where the competition is.

The Five-Layer AI Stack Is an Inverted Pyramid

Amit — Wed, 05 Aug 2026 00:24:16 +0000

TL;DR

Jensen Huang's five layers are energy, chips, infrastructure, models, and applications. The dependency chain is clear, but it is not a map of market power.
Participation expands upward: a few physical platforms support many models and far more applications.
Power concentrates at bottlenecks below and distribution gateways above. TSMC, ASML, three global clouds, and a short list of digital gatekeepers make that concentration visible.
Applications create the customer value. That does not mean application vendors keep most of the profit.

AI looks like software at the top and heavy industry at the bottom.

Jensen Huang describes AI as a five-layer cake: energy, chips, infrastructure, models, and applications. Each layer depends on the one below it. An application calls a model. The model runs on computing infrastructure. The infrastructure contains chips. The chips consume electricity.

That is the right dependency map. It is the wrong shape.

My read is that the stack behaves like an inverted production pyramid. Participation expands as you move upward. A small number of physical platforms support more models, which support far more applications and industry workflows.

Then the pyramid meets a second bottleneck. Thousands of applications still have to pass through a small number of distribution systems to reach customers.

flowchart BT
    E["Energy<br/>local power and grid access"] --> C["Chips<br/>qualified manufacturing"]
    C --> I["Infrastructure<br/>facilities, systems, and cloud"]
    I --> M["Models<br/>frontier, open, and specialized"]
    M --> A["Applications<br/>many products and workflows"]
    A --> D["Distribution gateways<br/>identity, discovery, billing"]
    D --> U["Customers"]

The stack widens in solution variety. It narrows around production capacity. It narrows again around customer access.

Width Is Not Power

An inverted pyramid only works after defining what its width represents.

Measure	The question it answers
Participation	How many companies or products can enter this layer?
Concentration	How much supply or spending belongs to the largest providers?
Value creation	Where does AI produce a useful customer outcome?
Value capture	Who retains the revenue and profit?
Distribution power	Who controls customer access, identity, and billing?

These measures do not move together.

A layer can contain thousands of products while a few suppliers control its inputs. An application can create the customer outcome while model, cloud, chip, data, and distribution providers collect part of the profit. More companies at the top does not automatically mean more competition or more economic power.

This distinction is the key to reading the five layers.

Energy Is Broad Globally and Narrow Locally

Energy looks like the broadest physical market. Countries have thousands of utilities, generators, fuel suppliers, and grid operators. It is not a global oligopoly.

But AI infrastructure does not need electricity in the abstract. It needs deliverable power at a specific site, on a specific schedule, with transmission, transformers, cooling, permits, and financing already aligned.

The International Energy Agency estimates that data centers consumed about 415 terawatt-hours of electricity in 2024, or 1.5% of global consumption. Its base case reaches roughly 945 TWh in 2030. That global share can look manageable while individual data-center regions face severe local constraints.

The hidden bottleneck can be an ordinary grid component. The US Department of Energy reports that distribution-transformer lead times expanded from three to six months in 2019 to 12 to 30 months in 2023.

So the bottom of the pyramid is not "a few energy companies." The narrow point is the combination of powered land, grid access, equipment, and permission to build.

Large companies such as NextEra Energy, Constellation, and GE Vernova supply generation and grid technology. Startups including Fervo Energy and Oklo are pursuing new generation paths. But a power announcement is not operating capacity. Licensing, construction, fuel, transmission, and interconnection still determine when a project becomes real.

Chips Contain the Narrowest Global Chokepoints

The chip layer contains many products but only a few qualified suppliers at critical stages.

NVIDIA and AMD sell merchant accelerators. Google, Amazon, and Microsoft design custom chips for their own platforms. Startups such as Cerebras, Groq, and Lightmatter attack different parts of compute and interconnect performance.

That design activity still funnels into a concentrated manufacturing chain.

The US Department of Commerce stated in April 2024 that
TSMC manufactures more than 90% of the world's leading-edge logic chips.
ASML describes extreme-ultraviolet lithography as technology unique to ASML. High-bandwidth memory comes from a small supplier group led by SK hynix, Samsung, and Micron.

This is the inverted pyramid at its clearest. A large application market can rest on one to three scaled suppliers at several manufacturing steps.

Custom silicon can weaken one accelerator vendor's position. It does not remove the dependence on foundries, lithography, memory, packaging, and materials.

Infrastructure Is Several Markets Compressed Into One

Huang's infrastructure layer includes land, power delivery, cooling, construction, networking, and orchestration. Economically, that combines at least three different markets:

Physical facilities: powered land, buildings, substations, cooling, and colocation.
Compute systems: servers, racks, storage, networking, schedulers, and inference software.
Cloud services: capacity sold through APIs, managed platforms, and long-term contracts.

The first two contain many specialists. Vertiv and Schneider Electric connect power and cooling. Equinix and Digital Realty operate data centers. CoreWeave, Crusoe, and Nebius sell specialized AI capacity.

Global cloud is much more concentrated. Synergy Research Group reported that Amazon, Microsoft, and Google held 63% of worldwide cloud infrastructure spending in the first quarter of 2026.

The competitive unit is no longer a GPU or a data-center building. It is the ability to assemble powered land, cooling, networking, accelerators, software, financing, and paying workloads into an operating system at high utilization.

That favors companies able to coordinate several sublayers at once.

Models Form Their Own Hourglass

The model layer looks wide when counting model artifacts. A
Data Provenance Initiative study examined 851,000 models on the Hugging Face Model Hub, representing 97.6% of downloads in its dataset.

Frontier model development is far narrower. OpenAI, Anthropic, Google DeepMind, and a small set of other laboratories train leading general-purpose systems. Around them sits a much wider ecosystem of open-weight models, fine-tunes, quantizations, routers, and specialized models.

That produces an hourglass inside the pyramid:

Many research projects and model artifacts
          Few frontier laboratories
Many hosted, adapted, and specialized variants

Model commoditization and model concentration can happen at the same time. Routine capabilities become cheaper and easier to substitute. Producing the strongest new general-purpose model can still require more capital, compute, data, and research talent.

Applications Expand, Then Distribution Narrows

Applications create the widest solution surface because industries do not share one universal workflow.

Cursor and GitHub Copilot target software development. Abridge works on clinical documentation. Harvey focuses on professional services. Waymo applies AI to autonomous driving. Each category brings different data, regulation, trust, integrations, and measures of success.

That diversity lowers the value of a generic market map. It also explains why application companies can keep entering even when the lower stack concentrates.

But application creation and application distribution are different markets.

The European Commission currently lists seven Digital Markets Act gatekeepers: Alphabet, Amazon, Apple, Booking, ByteDance, Meta, and Microsoft. The designation covers 23 core platform services, including app stores, search, operating systems, social networks, marketplaces, browsers, advertising, and messaging.

The list is not a measure of AI application share. It shows that the routes through which digital products reach customers already concentrate around a short set of platforms.

An application can call any model and still depend on one operating system, productivity suite, app store, cloud marketplace, identity provider, or social network for distribution. The top of the stack therefore has a gateway, not unlimited width.

The Largest Companies Cross the Layers

The major platforms do not stay inside one layer.

Company	Cross-layer position
NVIDIA	Chips, networking, systems, infrastructure software, models, and industry platforms
Amazon	Power procurement, custom chips, cloud infrastructure, models, marketplaces, and customer distribution
Microsoft	Custom chips, Azure infrastructure, models, developer tools, productivity software, identity, and distribution
Google	Custom chips, global infrastructure, models, Search, Workspace, Android, and YouTube
Meta	Custom chips, internal infrastructure, open models, social applications, and advertising distribution

Vertical integration can reduce coordination costs and improve complete-system performance. It also lets one company move value between layers. A platform can subsidize a model to sell cloud capacity, build a chip to lower inference cost, or bundle an application to protect distribution.

The five-layer cake describes the production system. The integrated platforms compete across the whole meal.

Applications Create Value, but Who Captures It?

Huang argues that applications are where the largest economic benefit appears. That claim holds: applications turn computed intelligence into completed code, clinical documentation, legal work, customer service, scientific discovery, or physical action.

It does not follow that application vendors keep most of the profit.

The application may create the customer outcome while cloud, model, data, chip, and distribution providers collect part of the revenue. Infrastructure suppliers can earn exceptional returns during scarcity. Distribution owners can bundle competing features. Model providers can move upward into applications, while application vendors can switch models or build their own.

The durable position is not necessarily an entire layer. It is a control point that other companies cannot easily route around:

a physical bottleneck that takes years to reproduce;
a software ecosystem that raises switching costs;
a workflow with proprietary context and customer trust;
a distribution surface that controls access, identity, or billing.

What the Five-Layer Model Misses

Jensen's model leaves two important forces implicit.

Data moves through several layers. It trains models, grounds applications, creates regulatory obligations, and becomes proprietary context inside a workflow.

Distribution sits above applications. It determines whether a product can reach customers, earn attention, establish identity, and collect payment.

Adding those forces does not invalidate the five-layer cake. It explains why the number of participants and the location of market power tell different stories.

So What

The inverted pyramid is a strong map of dependency and participation. It is a weak map of profit until the bottlenecks and gateways are drawn onto it.

For builders and investors, the question is not only, "Which layer are you in?" The sharper question is, "What do you control that the layers above or below cannot easily replace?"

The open question is whether applications eventually retain more of the value they create, or whether model, cloud, chip, energy, and distribution providers continue to collect most of it.

The answer may differ by industry. Coding tools can reach users directly. Healthcare and legal products depend on regulated systems of record. Consumer applications can scale quickly while remaining exposed to mobile and web distribution gates. Physical AI adds manufacturing and safety constraints back into the top layer.

One inverted pyramid may eventually become a different market map for every industry.

MCP Became Stateless: Migrating a Real AgentCore Gateway

Amit — Tue, 04 Aug 2026 05:21:24 +0000

MCP no longer needs a protocol session for remote tool calls. The 2026-07-28 specification removes the initialization handshake and Mcp-Session-Id, then makes every request carry enough information to stand alone.

I migrated a working Amazon Bedrock AgentCore Gateway that exposes the managed WebSearch connector. The change was smaller than the specification makes it sound: one gateway configuration update and one client-side request change. The WebSearch target, IAM authorization, tool schema, and local Codex connection stayed intact.

The result is the useful part. The gateway now serves old and new MCP clients at the same endpoint, and the new path reaches WebSearch without a handshake or protocol session.

The state before the migration

My existing setup already used the pattern from Web Search as a Managed Connector:

Codex
  -> local stdio MCP shim
    -> SigV4-signed HTTPS request
      -> AgentCore Gateway
        -> managed WebSearch connector

The local shim presented one tool to Codex, signed the outbound request with an AWS identity, and called WebSearch through the gateway. It worked, but both the shim and gateway were pinned to MCP 2025-06-18.

The request carried MCP-Protocol-Version: 2025-06-18 in an HTTP header. Its JSON-RPC body contained the tool name and arguments, but no per-request client information or capabilities.

What stateless MCP changes

Earlier remote MCP versions established protocol context through initialize, followed by notifications/initialized. Streamable HTTP servers could then issue an Mcp-Session-Id that clients returned on later requests. That couples subsequent traffic to state established earlier.

SEP-2575 removes the initialization handshake, while SEP-2567 removes protocol-level sessions from Streamable HTTP. Under 2026-07-28, each request declares its version, method, tool name, client identity, and capabilities.

The new WebSearch call adds three HTTP headers:

MCP-Protocol-Version: 2026-07-28
Mcp-Method: tools/call
Mcp-Name: web-search-tool___WebSearch

It also adds request metadata inside params:

{
  "jsonrpc": "2.0",
  "id": 1,
  "method": "tools/call",
  "params": {
    "name": "web-search-tool___WebSearch",
    "arguments": {
      "query": "MCP 2026-07-28 specification",
      "maxResults": 2
    },
    "_meta": {
      "io.modelcontextprotocol/protocolVersion": "2026-07-28",
      "io.modelcontextprotocol/clientInfo": {
        "name": "agentcore-websearch-shim",
        "version": "0.2.0"
      },
      "io.modelcontextprotocol/clientCapabilities": {}
    }
  }
}

The header and body versions must agree. AgentCore Gateway rejects mismatches, which gives gateways and other HTTP infrastructure a trustworthy routing signal without parsing the entire body.

Normal results also carry resultType: "complete". Interactive operations can instead return resultType: "input_required" and continue through a later request using the specification's Multi Round-Trip Request model. State becomes explicit data, not hidden connection history.

Quantifying the change

For a conventional session-aware client, the first tool call drops from an initialization exchange followed by the tool exchange to one self-contained tool exchange. That removes one network round trip from the cold path. The savings equal the client-to-server round-trip time, but only before the first call.

AgentCore Gateway already accepted my shim's direct tools/call without a preceding handshake, so its remote request count stayed 1 -> 1. I measured the compact JSON body for the same WebSearch call:

Measure	MCP 2025-06-18	MCP 2026-07-28
Remote requests per search	1	1
Compact JSON request body	162 bytes	372 bytes

The new request adds 210 body bytes plus two routing headers. There is no credible latency win to claim for this client because WebSearch execution dominates the call and the old shim had already skipped the handshake. The immediate client benefit is conformance with the new protocol and access to its result, discovery, caching, tracing, and multi-round-trip contracts.

The server-side arithmetic is stronger:

Protocol session state: N active clients previously meant up to N protocol sessions or a shared session store. The new core holds zero protocol-session records. Application state remains separate.
Load balancing: any request can reach any healthy instance. The stateless MCP proposal describes the previous choice between connection affinity and shared state; the new protocol removes that requirement.
Cold-call traffic: a conforming client removes one response-bearing exchange before its first tool result, a 50% reduction from two exchanges to one.
Failure recovery: losing one server instance no longer invalidates protocol sessions pinned to it. In-flight work can still fail, but recovery becomes one ordinary request retry rather than session recovery plus retry.

The gain is not smaller JSON or faster search. It is removing coordination: fewer cold-start exchanges, no protocol session store, no sticky routing, and less custom recovery logic.

What this means for builders

Client and host builders replace session management with request construction. They need a 2026-07-28-capable SDK or equivalent support for the new headers, _meta, resultType, discovery, HTTP errors, cache metadata, and multi-round-trip responses. Supporting both versions means selecting the protocol explicitly for each request.

Remote server builders need handlers that understand each request in isolation. Application state still exists: a coding workspace, shopping basket, or long-running job needs an explicit handle passed as a tool argument and authorized on every call. Any healthy server instance can then process the next request.

Tool builders behind a gateway often change nothing. My WebSearch target, IAM policy, tool name, and arguments stayed the same because AgentCore Gateway owns the MCP boundary. The backend does not need to know which MCP version the caller selected.

Interactive-tool builders have the largest conceptual change. Instead of relying on a persistent connection, the tool returns input_required, the client gathers the missing input, and a later request resumes the work with explicit state.

Platform operators gain standard HTTP control points. Mcp-Method and Mcp-Name can drive routing, rate limits, metrics, and policy without opening the JSON body. Cache metadata and trace context bring catalog traffic and distributed tracing into the same request model.

Local stdio MCP servers do not need an immediate rewrite. The migration boundary is remote Streamable HTTP; a shim or gateway can adopt the new contract while the local host keeps its existing protocol version.

The migration boundary matters

The shim is the protocol boundary, so the change stayed narrow:

Component	Change required
Local Codex MCP configuration	No
Local stdio initialization	No
Shim's outbound Gateway request	Yes
Gateway `supportedVersions`	Yes
IAM/SigV4 authorization	No
Managed WebSearch target	No
WebSearch tool arguments	No

Dual-version rollout is the safe path

AgentCore Gateway makes the transition opt-in. Its supportedVersions configuration is a complete list, not an appended value. Replacing an older version with the new one would strand every client that has not migrated.

I kept the old version while adding the new one:

["2025-06-18", "2026-07-28"]

Both paths completed a WebSearch request, and the new path returned resultType: "complete". Six local tests also passed for argument validation, metadata, headers, SigV4 signing, and response parsing. Version selection at the request boundary lets clients migrate independently.

What else arrived with this release

Statelessness is the architectural center, but the revision goes further:

HTTP operations: Mcp-Method and Mcp-Name expose request intent to routing, metering, and policy layers.
Caching and tracing: list responses can declare cache lifetime and scope, while W3C trace context can cross client, gateway, and tool boundaries.
Schemas and extensions: tools support JSON Schema 2020-12, and extensions have a governed lifecycle. Tasks moved from core into an official extension.
Authorization and lifecycle: authorization aligns more closely with OAuth 2.0 and OpenID Connect practice. Roots, Sampling, and Logging are deprecated under a formal lifecycle policy, though they remain functional during the compatibility window.

These changes move remote MCP closer to ordinary HTTP infrastructure: independently routable requests, explicit caching, standard tracing, and transport errors represented by real HTTP status codes.

What's missing

I have not removed 2025-06-18 from the gateway. Both versions work, but successful tests do not prove that every occasional client has migrated. Removing the old version needs caller inventory and protocol-version telemetry, not confidence based on one active shim.

The other open question is how quickly MCP hosts will expose the new protocol natively. A translation shim makes the migration manageable, but the cleaner end state is a host that can negotiate 2026-07-28 directly and use server/discover, cache metadata, trace context, and multi-round-trip interactions without custom code.

So what

Remote MCP is becoming normal web infrastructure. Gateways can route and meter calls from headers, servers can scale without sticky sessions, and clients can retry self-contained operations against any healthy instance. Application state still exists, but it has to be named and carried deliberately.

For my client, the trade was 210 extra request bytes in exchange for removing an entire category of connection state. The migration follows the same logic: advertise both versions, update one client at a time, and retire the old path only when usage proves it is safe.

Building an Autonomous AI News Agent on Step Functions and AgentCore

Amit — Mon, 03 Aug 2026 07:57:53 +0000

TL;DR

A daily pipeline fetches 24 curated RSS feeds, clusters items describing the same real-world story across sources, categorizes survivors into nine fixed buckets with one LLM call, and writes a ranked digest — no watchlist, no web search, no human review.
The first design tracked 77 named companies and investors and searched the web for each one. Measurement killed it: only 8 of 77 ever had a working official RSS feed, and the curated feed list surfaced the same funding rounds and incidents anyway, with no targeting at all.
One "synthesizer" invocation that deduped, categorized, extracted funding data, and wrote prose all in a single call hit AgentCore Runtime's real synchronous ceiling — not the 8-hour session limit the container's own error message implied. Splitting into analyze → write → digest fixed it.
A Step Functions Map fanning out per-category writers hit a second real bug: $$.Map.Item.Value only resolves inside the Map's own ItemSelector, not inside a nested Task's Payload. Neither bug showed up in 79 passing unit, contract, and integration tests — only a real deployed execution surfaced them.
The whole system holds state in flat JSON and Markdown files in one S3 bucket. No database. A story-thread index is the only thing that persists across days, and it's checked and pruned on every run, not on a schedule.

The wrong unit of work

The first version of this pipeline tracked entities. A watchlist of AI labs, chip makers, and investors — 77 of them — each with its own scan session, each hitting AgentCore Web Search to ask "what happened with this company today." The architecture made sense on paper: bounded, per-entity context; a clean map-reduce shape; isolation so one company's noisy news couldn't bleed into another's summary.

Then the watchlist got measured. Of 77 tracked entities, 8 had a working official RSS feed. The other 69 depended entirely on web search returning something relevant on a given day, for a company that might not have shipped anything that week. Meanwhile, a small set of broad industry-news aggregators — the outlets that already do the work of deciding what's newsworthy — covered the same funding rounds and product launches, correctly attributed, with no targeting at all.

The fix wasn't a better watchlist. It was dropping the watchlist entirely. Twenty-four curated feeds — eight official company blogs, sixteen aggregators, replace a per-entity scan with one fetch that reasons across everything at once. AgentCore Web Search came out with it; there was no longer an entity-scoped gap for it to fill. Coverage went up because attribution was already being done for me, by outlets whose job is to do it.

This is the general shape of the mistake: modeling the problem after the tracked entity, when the actual unit of coverage was the story, and the story doesn't care which company you're watching.

What the pipeline does, once a day

flowchart TD
    A[EventBridge Scheduler<br/>00:30 daily] --> B[Scanner<br/>fetch 24 RSS feeds]
    B --> C[Analyze<br/>dedup, categorize, rank]
    C --> D{Map over categories<br/>with surviving items}
    D --> E1[Writer: category 1]
    D --> E2[Writer: category 2]
    D --> E3[Writer: category N]
    E1 --> F[Digest<br/>top stories + assembly]
    E2 --> F
    E3 --> F
    F --> G[Publish<br/>kill-switch check]
    G --> H[(published/ in S3)]

Scanner fetches every feed unconditionally — no per-source filtering — normalizes each item's date into a comparable ISO-8601 UTC string, drops anything older than 30 days as a cheap staleness guard, and writes the raw batch to S3. No LLM call happens here at all.

Analyze does the actual reasoning, in five steps, none of which write a sentence of prose:

Cross-feed dedup. Canonicalize URLs, then cluster items across every feed describing the same real-world event — URL match first, embedding similarity as a fallback. The number of independent sources covering a story becomes a corroboration signal, carried forward rather than discarded once dedup is done.
Funding extraction. Clusters that look like funding rounds get a structured extraction call — company, investors, amount, stage — and get force-categorized into a Funding & Capital bucket regardless of what general categorization would pick.
Categorization. One LLM call reads every surviving cluster and assigns it to exactly one of nine fixed categories. Closed vocabulary, not open-ended — a genuinely irrelevant cluster gets dropped, not stretched into a category it doesn't belong in.
Story-thread classification. Each item gets checked against a global, 14-day trailing index of everything already published. Below a similarity threshold, it's classified directly as new, duplicate, or update; the ambiguous middle band gets one LLM adjudication call. Duplicates are dropped; updates get a reference appended, not a rewritten paragraph.
Rank, cap, and select top stories. Per category, take the top N ranked items into a primary section and the next few into an overflow section. Separately, one call reads across every surviving category and picks the handful of stories that matter regardless of which bucket they landed in.

Writer and Digest are where prose gets written, and they're deliberately split from Analyze and from each other — the reason why is the whole second half of this post.

The ceiling that wasn't the one I expected

The first working version of this pipeline had one "synthesizer" stage: dedup, funding extraction, categorization, story-threading, ranking, and prose-writing for every category, all inside a single AgentCore Runtime invocation. It ran fine in testing. It failed on a real deployed run with:

BedrockAgentCore.RuntimeClientErrorException: Runtime initialization time
exceeded. Please make sure that initialization completes in 120s.

That error message is about cold starts. It was not a cold start. The actual cause: AgentCore Runtime runs a /ping health-check thread alongside the entrypoint handler, and a long synchronous sequence of LLM calls — dedup, then five or six structured extractions, then a prose loop over every category — blocks that thread long enough for the platform to conclude the container is unhealthy and kill it.

The number that actually matters isn't a session lifetime measured in hours. AgentCore Runtime's own service quotas list two separate limits: a 15-minute request timeout for synchronous requests, and an 8-hour maximum for asynchronous jobs — neither adjustable. A session can live for hours. One synchronous invocation inside it cannot run past 15 minutes and still return a response.

The fix was architectural, not a bigger timeout: split the one long invocation into three chained stages, each well clear of the ceiling.

flowchart LR
    A[Analyze<br/>dedup + categorize + rank<br/>no prose] --> B[Writer x N<br/>one category per call<br/>≤5 prose calls each]
    B --> C[Digest<br/>top stories + assemble<br/>reads back every draft]

Analyze does everything that doesn't require writing sentences — the fast half of what used to be one call. Writer renders exactly one category's Markdown per invocation, fanned out by a small Step Functions Map over however many categories actually survived that day (at most nine, discovered at run time, not a static config list). Digest runs once after every writer invocation finishes, writes the cross-category "top stories" prose, and reassembles the full digest by reading back each category's already-written draft from S3 — because nothing survives in memory across separate invocations; if the next stage needs it, it has to be a file.

The general pattern: a monolithic long-running call scales badly against a synchronous platform ceiling. Splitting into bounded, single-purpose stages that each stay comfortably under the ceiling — and pushing state through a shared object store instead of holding it in memory — is the same shape AWS's own reference samples use for multi-agent Step Functions pipelines. It's a more boring architecture. It's also the one that doesn't get killed mid-run.

The bug that 79 passing tests didn't catch

Splitting the pipeline into three stages meant fanning out per-category writers with a Step Functions Map. The obvious way to pass the current category into each Map iteration's nested Task is to reference the Map's own context variable directly:

{
  "Payload": {
    "category.$": "$$.Map.Item.Value"
  }
}

This synthesizes cleanly. It passes cdk synth. It passes every unit test, because unit tests exercise the Python handler, not the state machine's actual JSON. It fails at runtime with States.Runtime: The JSONPath '$$.Map.Item.Value' specified for the field 'category' could not be found in the input — because $$.Map.Item.Value is only resolvable inside the enclosing Map state's own ItemSelector, not inside a Task nested in ItemProcessor. The fix is to resolve it one level up and forward it as a plain reference:

// In the Map's own ItemSelector:
{ "category.$": "$$.Map.Item.Value" }

// In the nested Task's Payload, one level down:
{ "category.$": "$.category" }

Neither this bug nor the timeout ceiling above showed up in 79 passing unit, contract, and integration tests before deployment. Both required a real Step Functions execution against a real AgentCore Runtime container to surface. Tests verify that code does what the code says. They don't verify that a platform's actual behavior matches its documentation, or that a JSONPath scoping rule holds the way the mental model assumes. For infrastructure this deep in a managed platform's own execution semantics, a green test suite is a necessary check, not a sufficient one — the deployed system still has to run for real, and the output still has to get read, not merely checked for non-emptiness.

What actually persists

There's no database anywhere in this pipeline. Every handoff between stages is a flat file in one S3 bucket:

flowchart TD
    RSS["config/rss-feeds.yaml<br/>24 feeds"] --> Scanner
    Scanner -->|writes| Raw["signals/{date}/raw-items.json<br/>ephemeral, one run"]
    Raw --> Analyze
    Index[("state/stories-index.json<br/>global, 14-day lookback, forever")] <-->|read + write| Analyze
    Analyze -->|writes| Ranked["state/{date}/ranked-sections.json<br/>no prose, one run"]
    Ranked --> Writer
    Writer -->|writes, one file per category| Drafts["drafts/{category}/{date}.md"]
    Drafts --> Digest
    Digest -->|writes| DailyDigest["drafts/daily-digest/{date}.md"]
    DailyDigest --> Publish
    Publish -->|copies| Published[("published/<br/>terminal output, forever")]

The one piece of state that outlives a single run is state/stories-index.json — a global, append-only record of every story published in the last 14 days, checked on every run so the same event doesn't get written up twice as it develops across days. Everything else — raw feed items, the day's ranked sections, individual drafts — is scoped to one run and never read again once the digest is assembled.

The reasoning for staying file-based instead of introducing a database: at this pipeline's actual volume — 150 to 300 raw items a day, collapsing to 40 to 80 clusters after dedup — full-file reads aren't a bottleneck, and flat files in S3 are free to version, cheap to store, and trivial to inspect with a plain aws s3 cp. A database adds schemas, migrations, and connection handling to solve a query pattern that "read the whole file" already serves at this scale. The rule isn't "never use a database" — it's don't introduce one preemptively, and revisit only when a specific, measured limitation of the flat-file pattern actually gets hit.

Model tiering, and the model that didn't get replaced

Every LLM-calling stage uses one frontier-tier model for both structured extraction and prose — categorization, funding extraction, story-thread adjudication, and every "write this section" call. Nothing in this pipeline runs on a cheap tier today; the scan stage that would have used one makes no LLM call at all post-pivot, since fetching RSS feeds is pure mechanism, not judgment.

The obvious move, once this was running inside AWS infrastructure, was assuming a Bedrock-native model would be cheaper. It wasn't. Checked directly against OpenRouter's own management API and AWS's Bedrock pricing API: the frontier model already in use runs at roughly $0.44 per million input tokens and $0.87 per million output, against $1.00 and $5.00 for Bedrock's equivalent-tier Claude model on the same account. Same story for embeddings — cost parity between the OpenAI embedding model in use and Bedrock's Titan embedding model, at a scale of dozens to low-hundreds of vectors per run with a brute-force linear scan and no vector database — meaning a swap wouldn't change anything measurable either way.

The lesson isn't "OpenRouter beats Bedrock." It's that "we're already on AWS, so the AWS-native option must be cheaper" is a real assumption worth checking against actual current pricing before acting on it, not a fact.

So what

The design that survived contact with real deployment isn't the one that looked cleanest on a whiteboard. It's the one that got measured — a watchlist that turned out to be 8-of-77 useful, a synchronous ceiling that turned out to be 15 minutes rather than 8 hours, a JSONPath scoping rule that only a live execution would surface, a pricing assumption that didn't hold once checked. None of those were visible from the architecture diagram alone.

The pattern underneath: build the pipeline to be cheap to run end-to-end for real, then let real executions — not test suites, not synthesized templates — tell you where the design's assumptions were wrong. A green build says the code does what it claims to do. Only a deployed run with real output, read and judged rather than merely checked for non-emptiness, tells you the architecture was actually right.

Open Data Wasn't Missing. The Interface Was.

Amit — Mon, 03 Aug 2026 06:28:24 +0000

Open data has an interface problem.

The Registry of Open Data on AWS already
contained 1,153 active datasets when I tested it. The catalog spans satellite
imagery, weather, life sciences, genomics, LiDAR, census data, and web archives.
The hard part was not finding a file with public access. It was turning a
research question into the right collection, license, access path, and first
executable query.

The Registry of Open Data MCP server
puts those qualification steps inside an agent conversation. The
launch post
describes discovery as the entry point. My test found a more precise boundary:
the server is a strong qualification interface behind a weak natural-language
search interface.

What the server can do

Version 3.4.5 exposed 10 tools in my live session. They cover discovery,
qualification, bounded inspection, and handoff:

Capability	Tools	What they return	Boundary
Search and browse	`search_datasets`, `list_datasets`	Keyword results plus tag, organization, and license filters	Search uses lexical matching, not semantic ranking
Focus discovery	`discover_by_organization`, `discover_by_license`	Datasets scoped to a provider or supported license family	Results depend on Registry metadata consistency
Inspect one dataset	`get_dataset_details`	Description, license, documentation, update frequency, and resources	Metadata describes access; it does not perform analysis
Follow catalog relationships	`find_related_datasets`	Other datasets with shared tags	Shared tags do not prove scientific relevance
Inspect the catalog	`get_knowledge_base_stats`	Dataset, tag, organization, resource, and license counts	Counts are a live catalog snapshot
Inspect public storage	`preview_dataset`	Up to 10 S3 object keys without downloading their contents	Multiple buckets require selection; private objects need credentials
Read a bounded sample	`sample_dataset`	Up to 100 KB from a selected public object; text display is capped at 2,000 characters	Sampling is inspection, not bulk data processing
Find domain handoffs	`search_stac_endpoints`	STAC APIs and catalogs referenced by Registry entries	The returned STAC service remains a separate interface

The server runs locally over standard input and output. This is the MCP client
configuration shape I tested:

{
  "mcpServers": {
    "roda": {
      "command": "uvx",
      "args": ["awslabs.roda-mcp-server@latest"],
      "env": {
        "FASTMCP_LOG_LEVEL": "ERROR"
      }
    }
  }
}

The equivalent direct launch command is:

uvx awslabs.roda-mcp-server@latest

Catalog search and metadata inspection did not require AWS credentials. Live
storage access still reflected each dataset's actual permissions.

The 12-question test

I ran 12 target-retrieval tasks across wildfire, oceans, weather, agriculture,
genomics, single-cell biology, autonomous driving, terrain, web history,
biodiversity, flooding, and population data. Each task had a known-fit Registry
entry selected before the search.

Every task used the same sequence:

Search with the plain-language research question.
Search again with distinctive terms and a tag or organization filter.
Retrieve the target's metadata.
Check its license, documentation, and resources.
Run a live access preview.

This measured target retrieval, not universal relevance. Another result could
be useful and still count as a miss if the preselected target did not appear.

Measure	Plain question	Constrained query
Known-fit target in top 3	2/12	11/12
Known-fit target in top 20	3/12	12/12

All 12 targets returned license, resource, and documentation metadata. Ten
returned a live anonymous bucket preview. One required a bucket selection.
Common Crawl returned an access-denied result with credential instructions.

Complete benchmark results | Domain | Plain rank | Constrained rank | Access result | |---|---:|---:|---| | Wildfire | Miss | 11 | Public preview | | Oceans | Miss | 2 | Public preview | | Weather | 11 | 1 | Public preview | | Agriculture | 2 | 1 | Bucket selection required | | Genomics | 1 | 1 | Public preview | | Single-cell biology | Miss | 1 | Public preview | | Autonomous driving | Miss | 1 | Public preview | | Terrain | Miss | 2 | Public preview | | Web history | Miss | 1 | Credentials required | | Biodiversity | Miss | 2 | Public preview | | Flooding | Miss | 2 | Public preview | | Population | Miss | 1 | Public preview |

Across 48 per-case MCP calls, median latency was 50.2 milliseconds, average
latency was 99.5 milliseconds, and the slowest call took 854.0 milliseconds.
This was one run on July 29, 2026, not a reliability study.

The retrieval gap was not subtle. Plain questions matched between 421 and 1,153
datasets, and nine known-fit targets disappeared beyond the first 20 results.
The server removes only a few generic terms, accepts a record when any remaining
term appears as a substring, and then diversifies the result set by provider. It
does not calculate a relevance score.

Constrained queries changed the result. Distinctive dataset terms plus a tag or
organization filter recovered all 12 targets. The agent host currently has to
do the query planning that natural-language search implies.

The handoff test

Finding metadata is not the same as using data. I ran a second experiment to
test the handoff from MCP qualification to an actual analysis:

research question
  -> RODA MCP discovery and access qualification
  -> Earth Search STAC scene selection
  -> bounded Cloud-Optimized GeoTIFF reads
  -> NDVI change
  -> numerical and visual evidence

The target was Sentinel-2 Cloud-Optimized GeoTIFFs.
A broad MCP query for that name matched 304 datasets and missed the target in
the first 20. The constrained query sentinel s2 l2a cogs, filtered to
Element 84, returned three records and ranked the target second.

The detail call then supplied the Earth Search documentation,
Sentinel access terms, public S3 and notification resource types, and the STAC
collection links. The preview correctly stopped at another boundary: the entry
has four public buckets, so the server asked for a bucket selection rather than
guessing.

That metadata was enough to leave MCP and use the domain interface.

A bounded wildfire screen

The CAL FIRE incident record
places the 2021 Lava Fire near Mount Shasta at 41.459, -122.329, reports a
June 24 start, and records 26,409 acres burned. I defined a fixed rectangle
around that published location. It is an area of interest, not a reconstructed
fire perimeter.

The script queried the current Earth Search sentinel-2-c1-l2a collection for
two windows:

Pre-fire: June 1 through June 23, 2021
Post-fire: July 15 through August 15, 2021

For each window, it selected the lowest-cloud scene that fully covered the same
area. It read red, near-infrared, scene-classification, and true-color assets
directly from their Cloud-Optimized GeoTIFFs. The scene-classification layer
removed cloud, shadow, snow, water, and invalid pixels. No complete source scene
was written to disk.

Each 10-meter source raster was 10,980 by 10,980 pixels. The geographic window
was 2,125 by 1,938 source pixels, or 3.42% of the tile's pixel footprint. The
analysis resampled that window to 20 meters and computed:

NDVI = (near infrared - red) / (near infrared + red)

The selected dates were June 18 and July 18. Across 948,704 valid overlapping
pixels, mean NDVI moved from 0.4266 to 0.3549. Mean change was -0.0716;
median change was -0.0107. The lower tenth percentile reached -0.3377.

The screening threshold matters more than the average. Pixels with a drop of at
least 0.2 covered 17.74% of the valid area, or 67.3152 square kilometers at
the analysis resolution.

In the change panel, red marks lower post-fire NDVI and blue marks higher
post-fire NDVI. The strongest red region aligns visually with the scar in the
post-fire image. That is evidence that the handoff works. It is not a
burn-severity result. The rectangle includes land outside the final fire
boundary, and NDVI also responds to seasonality, smoke, terrain shadows, and
land management.

The experiment is reproducible from the repository:

uv run scripts/roda-sentinel-ndvi-experiment.py

The script reruns MCP discovery, queries live STAC metadata, selects scenes,
performs bounded raster reads, and regenerates the JSON and image artifacts.

What the server unlocks

The server does not make open data public. It turns catalog work into agent
actions: search, filter, inspect a license, enumerate resources, and test an
access boundary. That removes the manual transition between a research question
and a known technical entry point.

The Sentinel experiment shows why that matters. MCP did not calculate NDVI.
It identified the collection, exposed its terms and resources, and pointed to
the interface that could. Earth Search selected scenes. Rasterio read bounded
COG windows. NumPy performed the calculation. Each layer stayed inside its
actual capability.

This pattern generalizes beyond satellite data:

Registry discovery can hand ocean observations to a NetCDF or Zarr workflow.
It can hand web archives to Common Crawl's index.
It can hand single-cell data to a domain API.
It can hand point clouds to a LiDAR processing engine.

The unlock is not one universal data tool. It is a reliable routing layer from
intent to the correct specialized tool.

What's missing

Search ranking is the immediate gap. Broad natural-language prompts can match
nearly the whole catalog because common words survive tokenization and any-term
substring matching. Structured filters repair retrieval, but only after the
host agent extracts the right terms.

Access preview also stops at storage. It can show that a bucket is public,
requires credentials, or needs a selection. It does not inspect scientific
fitness, choose an analysis method, or validate a result.

The open question is where semantic selection belongs. It could move into the
MCP server, remain in the host agent over a larger candidate set, or become a
separate domain-specific discovery layer. The current server proves the value
of the qualification boundary. The benchmark also shows that the host still
does the most important retrieval work.

Every Way to Attribute Cost on Amazon Bedrock, Live-Tested

Amit — Thu, 30 Jul 2026 03:31:56 +0000

TL;DR

Amazon Bedrock has five distinct mechanisms for attributing inference cost, and I tested all five live in an account: application inference profiles, Projects, Workspaces, IAM identity tags, and request metadata joined to invocation logs.
Four of the five reach real AWS billing data (Cost Explorer / Cost and Usage Report). All four are bound to a resource or an identity you provision ahead of time — none vary per individual API call.
The fifth, request metadata, is the only one that's per-call, but it lands in logs, not the bill. You compute the dollar figure yourself, from token counts — a real number, but not necessarily the invoiced one, since it can't see discounts or commitments applied at the account level.
For a multi-tenant SaaS product, per-customer billing isn't a cloud-tagging problem at all — it's an application-layer metering problem, and AWS's own Well-Architected guidance says so directly.

The five mechanisms

Every one of these was tested with real API calls against a live Bedrock account, not read off a docs page. Each does something genuinely different, and picking the wrong one for your use case is where most cost-attribution confusion starts.

1. Application inference profiles — tag a model, not a call

An inference profile is a named resource that wraps a specific model. Tag the profile, route calls through it, and the tag shows up on the resulting billing line items.

aws bedrock create-inference-profile \
  --inference-profile-name "team-search-claude" \
  --model-source copyFrom="arn:aws:bedrock:us-east-1::foundation-model/anthropic.claude-haiku-4-5" \
  --tags Key=team,Value=search Key=cost-center,Value=eng-42

Every call that specifies this profile's ARN as the model ID gets billed under it, and the team / cost-center tags become groupable dimensions in Cost Explorer once activated. The catch: one profile per model per cost dimension. Ten teams sharing five models means fifty profiles to create and keep in sync.

2. Projects and Workspaces — one resource, two entry points

A Project is a billing-scoped container that can span multiple models, which fixes the per-model constraint of inference profiles. Workspaces are the same underlying resource, referenced from a different API surface.

curl -X POST https://bedrock-mantle.us-east-1.api.aws/v1/organization/projects \
  -H "Authorization: Bearer $TOKEN" \
  -d '{"name": "search-team-project", "tags": {"team": "search", "cost-center": "eng-42"}}'

Scope a call to it via a header — OpenAI-Project: proj_xxxx on the Chat Completions surface, or anthropic-workspace: proj_xxxx on the Anthropic Messages surface. I created one Project, then called both surfaces against the same project ID and confirmed both routed to the identical billing bucket. One resource, callable from either API shape.

3. IAM identity tags — tag the caller, not the call

Tag the IAM role or user making the calls, and the tag rides along on every request that role makes.

aws iam tag-role --role-name bedrock-search-service \
  --tags Key=team,Value=search Key=cost-center,Value=eng-42

Once the tag key is activated as a cost allocation tag, it shows up prefixed iamPrincipal/team in the billing export's tag column, alongside the caller's ARN in a dedicated line_item_iam_principal field. This is the mechanism that scales best across many models without creating a resource per dimension — one tagged role covers every model that role touches.

The limit is architectural, not a missing feature: the tag lives on the identity, not the request. If one shared service role serves ten thousand different end-customers, the bill sees one tagged identity, not ten thousand.

4. Session tags — a different tag per login, not per role

STS lets you attach tags at the moment a role is assumed, distinct from the role's own static tags. This is the mechanism that gets you closer to per-tenant, because a federated identity provider can mint a differently-tagged session for every login.

sequenceDiagram
    participant User as Tenant User
    participant IdP as Identity Provider
    participant STS as AWS STS
    participant Bedrock

    User->>IdP: Login
    IdP->>IdP: Read tenant_id attribute
    IdP->>STS: AssumeRoleWithWebIdentity (principalTags: tenant_id)
    STS->>STS: Attach session tag to temporary credentials
    STS-->>User: Temporary credentials (1hr TTL)
    User->>Bedrock: Converse(...) using tagged session
    Bedrock-->>User: Response
    Note over STS,Bedrock: Session tag rides on every call<br/>until credentials expire

Wiring this up: a Cognito Identity Pool (or any OIDC/SAML identity provider — Okta, Auth0, Entra ID all support the identical pattern) maps a custom user attribute to a principal tag.

aws cognito-identity set-principal-tag-attribute-map \
  --identity-pool-id "us-east-1:xxxx" \
  --identity-provider-name "cognito-idp.us-east-1.amazonaws.com/us-east-1_xxxx" \
  --principal-tags tenant_id=custom:tenant_id

The IAM role being assumed has to explicitly trust sts:TagSession, not only sts:AssumeRoleWithWebIdentity — leaving it off doesn't produce an obvious tagging error, it produces an unrelated-looking InvalidIdentityPoolConfigurationException. I hit this omission once and it cost me twenty minutes tracing it back to the trust policy.

Getting the tag to actually show up in the billing export requires two separate switches, both easy to miss: INCLUDE_IAM_PRINCIPAL_DATA has to be turned on in the export's table configuration, and the export's own query has to explicitly select the tags column — a separate column from resource_tags, which is where I looked first and found nothing. Missing either switch produces a report with no error and no tenant breakdown, which reads exactly like the tag isn't working when it actually is.

This mechanism scales to per-tenant, but the cost is real: minting a session per tenant means caching credentials properly. A naive implementation that re-authenticates on every inbound request will hit identity-provider rate limits well before it hits any meaningful production traffic — Cognito's GetCredentialsForIdentity, for instance, defaults to 200 requests per second, account-wide. The fix is the same one every AWS SDK's credential provider already implements: cache the session for its lifetime, refresh a few minutes before expiry, never mint one per request.

5. Request metadata — the only per-call mechanism, and it skips the bill

Every other mechanism attributes cost by who's calling. This one attributes by what's in the call, and it's the only one that's genuinely per-request.

aws bedrock-runtime converse \
  --model-id us.anthropic.claude-haiku-4-5-20251001-v1:0 \
  --messages '[{"role":"user","content":[{"text":"hi"}]}]' \
  --request-metadata '{"customer_id":"cust-042","feature":"chat-widget"}'

That metadata lands in CloudWatch invocation logs — automatically, alongside input and output token counts — the moment you turn on model invocation logging. It does not appear anywhere in Cost Explorer or the Cost and Usage Report. To get a dollar figure, you query the logs and multiply.

filter ispresent(requestMetadata.customer_id)
| fields requestMetadata.customer_id as customer_id,
         input.inputTokenCount as inputTokens,
         output.outputTokenCount as outputTokens,
         (input.inputTokenCount * 0.000001) + (output.outputTokenCount * 0.000005) as estCostUSD
| stats sum(inputTokens), sum(outputTokens), sum(estCostUSD) by customer_id

I ran this against two simulated customers and got a clean per-customer split — real token counts, real cost math, computed in one CloudWatch Logs Insights query. I haven't compared this number against an actual invoice line item, but AWS's own cost-management guidance is explicit that a token-times-published-rate estimate doesn't account for volume discounts, committed spend, or whatever pricing tier the account is on — it's a real, defensible number for a dashboard, and it's built from a different formula than whatever your invoice actually applies.

The map, end to end

flowchart TD
    A[API call to Bedrock] --> B{What are you tagging?}
    B -->|A model resource| C[Inference Profile]
    B -->|A billing container| D[Project / Workspace]
    B -->|The calling role| E[IAM identity tag]
    B -->|The calling session| F[STS session tag]
    B -->|The request itself| G[Request metadata]

    C --> H[Cost Explorer / CUR]
    D --> H
    E --> H
    F --> H
    G --> I[CloudWatch Logs]
    I --> J[Self-computed estimate]

    H --> K[Real invoice dollars,<br/>bound to a resource or identity]
    J --> L[Real per-request granularity,<br/>estimated dollars]

Four paths converge on the same billing platform and inherit the same shape: attribution follows a resource or an identity, aggregated by day and usage type, never by individual request. One path breaks out to the request layer and trades the aggregation ceiling for an estimate. There's no version of this that gives you both a per-request breakdown and an invoice-accurate number, on Bedrock, today.

Where this stops being an AWS problem

If you're running a multi-tenant SaaS product on Bedrock, the natural next question is whether any of the five mechanisms above solve per-customer cost — team A pays for tenant X's usage, and you need to know the exact number.

They don't, and the reason isn't a gap in Bedrock specifically. AWS's own Well-Architected SaaS Lens states the design point directly: measuring per-tenant consumption in a shared-resource architecture requires the application itself to instrument tenant activity and correlate it with billing data afterward — the billing report alone was never going to enumerate an unbounded, growing customer base. The reference pattern in the same guidance is: capture tenant activity at the request layer (request counts, token counts, whatever correlates with cost in your architecture), store it, then apply that consumption ratio against the aggregate AWS bill for the period. That's mechanism five above, generalized — the SaaS billing layer sits next to CUR, reading from it, not inside it.

I went and checked whether another provider had actually solved this differently rather than only packaged it better. Google's Vertex AI lets a single service account attach a label to every individual request and have that label reach the actual Cloud Billing export — no per-tenant credential required, which is a real architectural difference from minting a tagged STS session per tenant. But the mechanism has a limit stated plainly in Google's own documentation: each label key holds at most 1,000 unique values, for the lifetime of the billing account, silently dropping anything past that with no error surfaced anywhere. A product with a few hundred tenants gets real per-tenant billing visibility with none of the identity plumbing Bedrock's session-tag pattern needs. A product that expects to grow past a thousand tenants hits the same wall — later, and more quietly than a missing feature would announce itself.

So what

Pick the mechanism by what you're actually trying to attribute, not by which one sounds most granular. Team, department, cost center — tag the role or the resource, and it's done; that's what all four billing-linked mechanisms are actually built for. Per-customer, in a product with a growing and unbounded tenant base — nothing in Bedrock's billing layer, or in Vertex's once you check the fine print, gives you that natively past a bounded scale. Build it where AWS's own architecture guidance says to build it: instrument the request layer, correlate against the bill yourself, and treat the result as a well-reasoned estimate rather than a substitute for the invoice.

I haven't found a platform that closes this gap natively — a request-level tag that lands in real, invoice-accurate billing data with no cardinality ceiling. I don't know if that's a hard problem or an unbuilt one.

Web Search as a Managed Connector: Wiring Amazon Bedrock AgentCore WebSearch Into Any Agent

Amit — Sat, 25 Jul 2026 06:17:21 +0000

Most "add web search to your agent" integrations are a wrapper around someone else's search API: an API key in an env var, a rate limiter, a result parser, and a bill from a third party. Amazon Bedrock AgentCore WebSearch removes all four. It ships as a managed MCP connector you attach to a Gateway — the search index, auth, and result shaping happen inside AWS, and your agent discovers the tool with a standard tools/list call.

I wired it into my local agents and have run 331 real searches through it. This post covers what makes it worth choosing over a search-API wrapper, how the wiring actually goes together (including the auth), and what the numbers look like after about 10 days of real use.

What the managed connector actually is

WebSearch on AgentCore went GA on June 16, 2026, currently in us-east-1 only. It is a built-in connector target on an AgentCore Gateway, spoken over MCP. You attach it with connectorId: "web-search" and the Gateway handles schema management, endpoint resolution, and service authentication.

Two things make it more than a search proxy. First, it is backed by a web index Amazon operates directly, spanning tens of billions of documents, rather than reselling a third-party engine. Second, it returns semantic snippets tuned for a model's context window — ranked excerpts with source URLs, titles, and publication dates — not raw HTML you then have to strip. The design goal is intelligence per token, and it combines the web index with knowledge-graph facts so entities resolve to verified data rather than inferred page text.

The practical consequence: queries never leave your AWS environment. There is no outbound call to an external search vendor, so there is no data egress and no third-party credential to rotate.

Why choose it over a search-API wrapper

The reason an agent needs web search at all is grounding. A model's knowledge is frozen at training time, so anything current — a release from last week, a price that changed yesterday, a filing from this morning — is either unknown or hallucinated. WebSearch closes that gap by grounding responses in current, cited web knowledge, which is the difference between an agent that guesses and one that answers.

The rest of the case is what you don't build. Rolling your own web search for an agent is several projects stacked together: procure a third-party search API, manage keys and quotas and rate limits, parse inconsistent result formats, write snippet-extraction logic so the model gets passages instead of raw HTML, reason about where queries travel and how they're retained, and keep coverage fresh over time. The managed connector collapses all of it into one connectorId.

Concretely, the benefits that matter:

Grounding over guessing — real-time facts and cited sources instead of stale training data, which directly cuts hallucination on timely questions.
Zero infrastructure — no search API to provision, no quotas to tune, no result parser to maintain. The Gateway exposes it as a standard MCP tool.
Framework-agnostic — any MCP-compatible client works: Strands, LangChain, LangGraph, CrewAI, or a hand-rolled shim. Nothing is tied to one agent framework.
A purpose-built index, not a reseller — backed by an index Amazon operates directly (tens of billions of documents, refreshed within minutes), the same search infrastructure behind Alexa+, Amazon Q Business, and Kiro. Broad coverage answers long-tail questions a popularity-ranked engine misses, and a knowledge graph resolves entities to verified facts.
Zero data egress — queries stay inside AWS, which removes an entire category of security and compliance review for teams that care where data travels.
Usage-based pricing — $7 per 1,000 queries with no upfront commitment, plus Free Tier credits for new accounts. You pay per search, not for standing infrastructure.

The through-line: improvements to freshness, coverage, and relevance flow to your agent automatically through the same connector, with no version upgrade or migration on your side.

The shape of the wiring

The connector lives in AWS. The caller can be a local agent on your laptop speaking MCP over stdio, or — more commonly — an autonomous agent deployed on AgentCore Runtime or your own compute, calling the Gateway directly. Either way the caller discovers the tool with tools/list and invokes it with tools/call; the only difference is how it signs. A deployed agent signs with its execution role; a local agent uses a small shim. The path is the same shape:

Caller
  · deployed agent on AgentCore Runtime / Lambda / ECS  → execution role signs (SigV4)
  · local agent on your laptop (Codex / Cursor / Zed / Claude Code / Kiro / GitHub Copilot) → MCP shim signs (SigV4)
      → POST https://<gateway-id>.gateway.bedrock-agentcore.<region>.amazonaws.com/mcp
        → target (connectorId: web-search)
          → Amazon web index + knowledge graph
            → ranked snippets + source URLs + titles + dates

You create the Gateway once with authorizerType: AWS_IAM and protocolType: MCP, then add a connector target pointing at web-search. The Gateway assumes an execution role to reach the backend; that role needs no special search permissions because AWS operates the search stack.

If you want to bound what the tool can reach, the connector supports a domain denylist — a list of domains the Web Search Tool is not allowed to query, configured on the target at creation time through the SDK or CLI (create-gateway-target, the same call that sets connectorId: web-search).

One thing worth calling out, because it trips people up: configuring the connector needs current AWS tooling (or the console), but calling it does not. The invocation path is just a SigV4-signed JSON-RPC POST to the gateway URL, so an agent or shim runs on whatever SDK version it already has.

Wiring in the auth

The Gateway is IAM-authorized, so the caller has to present a SigV4-signed request from an AWS principal. That is the entire auth model: does the caller hold valid AWS credentials for the account that owns the gateway? There is no token endpoint, no client secret, no refresh loop.

The signing is identical whether the caller is a deployed agent or a local one — it resolves credentials from the standard AWS chain (environment variables, then a named profile, then an instance or container role) and signs each MCP request before sending it. A deployed agent on Runtime or Lambda gets its execution role from that chain; my local shim gets a profile from the same chain. Same code, different credential source:

import botocore.session, botocore.auth, botocore.awsrequest

def sign(url, body, region):
    creds = botocore.session.Session().get_credentials().get_frozen_credentials()
    req = botocore.awsrequest.AWSRequest(method="POST", url=url, data=body, headers={
        "Content-Type": "application/json",
        "Accept": "application/json, text/event-stream",
        "MCP-Protocol-Version": "2025-06-18",
    })
    botocore.auth.SigV4Auth(creds, "bedrock-agentcore", region).add_auth(req)
    return dict(req.headers.items())

Notice what get_credentials() does not take: no API key, no secret, no token. A deployed agent presents its execution role automatically; on my laptop the tool config sets a profile name and a region and botocore does the rest. In neither case is there a search credential in the config to leak, rotate, or expire — the AWS identity the caller already has is the credential.

Because resolution is per-process, the same signed path works unchanged across every local agent harness that speaks stdio MCP. Point Codex, Cursor, Zed, Claude Code, Kiro, or GitHub Copilot at the same shim and they all inherit web search, each using whatever profile its environment provides — while a fleet of deployed agents inherits it through their execution roles with no shim at all.

What 331 real searches look like

Numbers from my own gateway over about 10 active days since I started using it — not a benchmark, just the real traffic from running agents.

Metric	Value
WebSearch invocations	331 across 10 active days
Busiest day	91 calls
End-to-end latency p50	635 ms
End-to-end latency p90	784 ms
End-to-end latency p99	1,013 ms
Connector backend time (p50)	547 ms
SystemErrors	0
Throttles	0
UserErrors	1

Two things stand out. Sub-second p50 end to end, with zero system errors and zero throttles across every call — the managed path held up under bursty agent traffic without any tuning on my side. And the latency is dominated by the search itself: ~547 ms of the ~635 ms p50 is connector backend time, meaning the Gateway and signing overhead adds well under 100 ms. You are paying for the index doing real work, not for glue.

Where these numbers come from

Every figure above is emitted automatically — there is nothing to instrument. The Gateway publishes to the CloudWatch namespace AWS/Bedrock-AgentCore, dimensioned per tool, so you can slice latency and error rate for the WebSearch target specifically. TargetExecutionTime is the connector's own backend time, which is what let me separate search time from gateway overhead above.

What it costs, from the actual bill

The pricing claim is easy to verify because it shows up as its own line. In Cost Explorer and the Cost and Usage Report, WebSearch appears under the service Amazon Bedrock AgentCore with the usage type USE1-WebSearchTool:Consumption-based:Queries. On my bill, 144 queries in one month cost $1.008 and 152 the next cost $1.064 — exactly $0.007 per query, or the advertised $7 per 1,000, with no rounding surprises. The Gateway itself bills separately and trivially, as USE1-Gateway:Consumption-based:API-Invocations (fractions of a cent for the same traffic) plus an even smaller Gateway:Consumption-based:Tool-Indexing line — a fraction of a tenth of a cent, and it scales with the number of tools registered on the gateway rather than with search volume, which fits a per-schema indexing cost for the Gateway's semantic tool discovery. So the cost model is legible: one metered line for searches, one negligible line for gateway invocations, and nothing for standing infrastructure.

What's missing

Three honest constraints. It is us-east-1 only right now, so if your agents or data residency live elsewhere you are making a cross-region hop or waiting for the regional rollout. The query is capped at 200 characters, which is fine for natural-language questions but forces you to decompose anything long. And the connector is semantic web search for grounding, not a data feed: the acceptable-use terms require you to retain and display source citations on any result you surface, and prohibit bulk-extracting results or using them to build a competing index.

The open thread I have not closed is per-user authorization. The external path itself is not in doubt: alongside my own IAM-authorized agents, I run a separate JWT-authorized gateway that fronts several tools, where callers present an OAuth token and carry no AWS credentials at all. That proves an external caller can reach a connector without a shared secret. But my gateway authorizes at the coarsest grain it can — it validates that the token comes from my issuer and carries my audience, and nothing more. Every caller behind that token is identical to the gateway.

The capability to go finer is already there; I just have not wired it. A JWT-authorized Gateway evaluates Cedar policies on every call, constructing an authorization request whose principal is the token's sub claim, whose action is the tool name, and whose resource is the gateway — so you can write a rule that says this user may call this tool and have it enforced before the connector ever runs, with conditions that read claims like scope or role or even the tool's own arguments. Per-user scoping is a policy I have not authored, not a feature the platform is missing. And because a Gateway's authorizer is IAM or JWT, fixed at creation, mixing my own agents and external users still means two gateways in front of the same connector — which is exactly why I run them separately today. Authenticated external access works; per-user authorization behind it is the Cedar layer I would wire before I called the multi-user story done.

So what

If you are giving an agent web search, the question is not which search API to wrap. It is whether the search itself is good enough to ground the agent, and whether the data path is one you can defend. WebSearch answers both: an Amazon-operated index of tens of billions of documents, refreshed within minutes and paired with a knowledge graph, so the agent reasons over current, cited facts instead of stale training data — and every query stays inside your AWS boundary, with no egress to a third-party engine and nothing retained outside AWS. That last part is what turns it from a convenience into something you can put in front of a security review.

The auth model is the tax you don't pay on top of that. Because the connector is IAM-authorized, the AWS credential you already manage becomes the credential for search — no new secret, no new parser, no new vendor bill, nothing to rotate. Good search results, a data path that stays where it belongs, and an integration with nothing left to leak: that is the combination worth choosing, and it is why running web search on AWS makes sense rather than bolting a search API wrapper onto the side of your agent.

Does Graphify Actually Help an AI Coding Agent?

Amit — Sat, 25 Jul 2026 05:55:55 +0000

The thesis: a codebase map earns its place only when it helps an agent reach a correct answer with lower measured cost.

Graphify promises a local knowledge graph for code, configuration, and documentation. It turns files into nodes and relationships, then lets an agent query the graph instead of rediscovering the repository through search. That is a plausible mechanism. It is not evidence of a productivity gain.

The first result is a measurement rule: do not accept the token-reduction claim, the graph visualization, or the feeling that exploration was easier as proof. Test the tool on the work it is meant to improve.

The Result So Far

Against opening four relevant files in full, Graphify reduced retrieval context from 15,186 to 1,618 tokens: an 83% reduction.
Against disciplined search plus narrow source snippets, it used 1,618 versus 1,488 tokens: 9% more.
In a three-question freshness-aware run, the wrapper arm had a 20-second median versus 17.4 seconds for disciplined search. It fully covered two of three answer keys; the baseline covered all three.

The wrapper removes command-syntax errors and can narrow exploration. It has
not yet shown a speed, token, or correctness advantage over careful search and
source verification.

Why the Test Has to Be Local

The published evidence on AI coding productivity points in opposite directions. In a randomized trial, METR found that experienced open-source contributors using early-2025 AI tools took 19% longer on their own tasks, even though they believed the tools made them faster. METR now labels that result historical and says it no longer reflects current tools.

The durable lesson is not that AI slows people down. The durable lesson is that reported confidence and measured completion are different things.

Databricks reached the same practical conclusion from the other direction: evaluate against real tasks from the codebase, review the task and its answer by hand, and use held-out tests rather than an LLM judge that rewards an answer for sounding correct.

Graphify's own documentation makes a narrower claim. Its local parser extracts structure and relationships; its graph can answer scoped queries without reopening every source file. The right experiment tests that exact claim: repository understanding, impact analysis, and architectural navigation.

The Pilot

Use twelve read-only tasks drawn from one unfamiliar but representative repository. Six run without the graph. Six run with a fresh code-only graph. A second set of matched tasks reverses the order. This counterbalanced design prevents the first condition from winning because the operator learned the repository during the trial.

The task set needs four categories:

Category	Example question	Required evidence
Ownership	Which service enforces a budget or access rule?	Handler, shared policy code, and test
Change impact	What changes when a new endpoint or tool backend is added?	Application code, configuration, infrastructure, and docs
Architecture	Why are two components separate instead of merged?	Current code plus the decision record
Operations	What is the approved deployment or recovery path?	CLI contract, infrastructure definition, and operator guide

Before the first run, a reviewer writes an answer key for every task. The key names the facts that must appear and the files that establish them. It also lists dependencies that are easy to miss. No tool sees the answer key.

Each task has a fixed stopping rule: the participant submits a short answer with citations to local files, then states the change or decision they would make. The reviewer scores the answer before seeing whether Graphify was available.

What to Measure

Time matters, but it is not enough. A fast answer that misses the infrastructure change is worse than a slow answer that finds it.

Record five fields for every task:

Correctness: Did the answer match the answer key?
Evidence coverage: Did it cite the source files and decision records that establish the answer?
Dependency coverage: Did it identify every required cross-layer effect?
Elapsed time: From task start to submitted answer.
Exploration cost: Raw files opened, search calls, and model usage when the harness exposes it.

The reviewer also records unsupported assertions. This is the failure mode a map can create: a clean-looking path through the graph can make an inferred connection feel established. Graphify distinguishes extracted and inferred edges. The evaluation must preserve that distinction.

The Decision Rule

This is a pilot, not a claim about the category. Twelve tasks establish whether the next test is worth running.

Keep Graphify optional for unfamiliar, cross-layer questions. Admit it to the
default workflow only when it is at least as correct and has either a 20% lower
median completion time or a 20% lower measured retrieval-token cost. If it
saves tokens but produces more unsupported claims, it failed.

Keep it out of the default workflow when the task already identifies the files to change. A graph has its highest chance of helping during onboarding, architecture review, incident investigation, and change-impact analysis. It has little reason to sit between a builder and a two-file fix.

The Safety Boundary

The first run is code-only and local. Exclude secrets, state, local environment files, generated bundles, dependency directories, and deployment output. Do not enable document or media enrichment, strict hooks, shared graph hosting, or a project-scoped agent instruction during the pilot.

That leaves one variable under test: whether a local structural map improves the agent's understanding of the repository.

Results

The first run is complete. It is an exploratory four-task slice, not the
twelve-task counterbalanced study described above.

Two fresh agents each completed one baseline task and one Graphify-assisted
task. The Graphify condition required graph-first discovery, then limited raw
source verification. The baseline condition used normal repository search and
raw source reads. No task changed code, contacted AWS, or enabled a persistent
agent hook.

Condition	Tasks	Median time	Raw source files opened	Answer quality	Unsupported assertions
Baseline	Budget enforcement; conditional routing	17.5s	6.5 average	Correct, source-cited, and bounded	0
Graphify	Cost attribution; load-balanced routing	16.5s	4 average	Correct after raw-source verification	0

The graph itself took 7.1 seconds to build from 299 code files. It produced
4,005 nodes and 10,229 edges in a 12 MB local artifact. Focused explain
lookups took 0.32 to 0.36 seconds and returned source locations plus
extracted-versus-inferred relationship labels.

The read: the graph condition opened fewer raw source files. It did not clear
the adoption threshold. The observed one-second median difference is too
small, and the tasks were not identical in difficulty. This unpaired slice
cannot establish a speed or token benefit. The tool is an optional discovery
layer for unfamiliar code paths, not a replacement for source verification.

Two implementation limits surfaced:

Broad natural-language graph queries returned hundreds of nodes and were truncated. The useful pattern was a narrow query followed by explain on a discovered symbol.
The code-only safety mode omitted Markdown decision records and documentation. That makes it useful for code and infrastructure tracing, but insufficient for an architecture question whose authoritative answer lives in a decision record.

Token Result

The graph also needs a token test. Its value proposition is not that source
code disappears. The value proposition is that an agent reads a structural map
first, then opens only the evidence required to verify the answer.

The same load-balancing question was measured three ways. The count uses the
cl100k_base tokenizer over the captured
tool output and source text. It excludes the agent's system prompt, tool-call
envelope, and generated answer, so it measures retrieval context rather than
total model cost.

Retrieval pattern	Input tokens	Result
Open four relevant files in full	15,186	High context cost
Search, then read narrow source snippets	1,488	Lowest cost
Broad graph query, graph explanation, then source verification	2,515	69% more than disciplined search
Graph explanation, then minimal source verification	1,618	9% more than disciplined search

The result is clear. Graphify reduced context by 83% against opening all four
files in full. It did not save tokens against a careful search-and-snippet
workflow on this focused question.

That is not a failure of the graph. It defines the right use case. Use it when
the agent does not know where a cross-layer path begins, when the repository is
unfamiliar, or when the alternative is broad file reading. Do not force it
into a two-file question where search already identifies the exact symbols.

The expanded study needs to record retrieval tokens for every task, with the
same boundary: graph output, raw verification text, and the final generated
answer counted separately. A graph-first workflow earns a default only if it
reduces total context without reducing evidence quality.

Release Context

Graphify is getting attention because it is new, visible, and moving quickly.
The public repository
was created on April 3, 2026, and the
PyPI package first shipped the next
day. By July 24, GitHub release history
recorded 172 public releases, including 27 releases in the 27 days from June
28 through July 24.

The adoption signals are real but need interpretation. PyPI Stats
recorded 3.60 million package-download events through July 24, while the
project site reported 93,125 GitHub stars on July 20.
Package downloads are not active users, and star counts do not establish
production use. They establish attention.

Independent technical guides appeared across April through July, including
Analytics Vidhya,
Foss Engineer, and
Better Stack.
The Hacker News response was smaller: the July Show HN post had two points and
no comments, according to the public HN search record.

The read: there is genuine launch momentum around a project that is less than
four months old. The release rate is also a maturity signal. Treat the current
CLI as fast-moving developer tooling, not stable shared infrastructure.

Freshness And Safety Test

An isolated temporary corpus tested local extraction, ignored-file handling,
and incremental rebuilds without a model backend.

Check	Result
Python and Terraform/HCL extraction	Passed: expected symbols and resources appeared
Ignored secret-like file	Passed: it did not appear in the graph
Unchanged re-extraction	Passed: zero files changed and the graph hash stayed identical
Add and delete source files	Passed: the new symbol appeared and deleted-file nodes disappeared
Code-only follow-up update	Mixed: `update --force` added Markdown nodes after the initial code-only extraction

The last row matters. A code-only initial extraction is the correct safety
boundary. Do not assume every follow-up command preserves that boundary without
an explicit test on the exact version in use. Keep graph output outside the
repository and retain GRAPHIFY_QUERY_LOG_DISABLE=1 until query logging is
independently verified.

Expanded Agent Run

A second agent run was intended to broaden the task sample across identity and
routing questions. It did not produce a publishable performance score.

The experimental instruction described the graph command too loosely. Multiple
agents first invoked an invalid command before reaching the correct
graphify query "<question>" --graph <path> form. That invalidates elapsed
time as a comparison. The answers still required raw source verification, and
the graph output remained broad enough to need narrowing.

This is useful operational evidence. Graphify needs a precise local skill or
wrapper that supplies valid query syntax and a narrow-query pattern. Without
that layer, agent setup errors erase any small retrieval-time benefit.

Freshness-Aware Wrapper Run

The wrapper was then tested on the same repository revision with a fresh
code-only graph, three matched questions, and an explicit first command:
graph-discover query "<question>" --repo <repository> --budget 800. It passed
the integration test: all three graph-first runs used valid syntax, checked the
freshness record, and verified their conclusions in raw source. The wrapper
also kept graph artifacts outside the repository.

The result does not show a speed win. The baseline used disciplined search and
narrow source reads; the wrapper run used graph discovery followed by source
verification. The timing below is agent-reported and one run per condition.
Five runs only had whole-second resolution because the local date command
did not support millisecond output.

Question	Baseline time	Wrapper time	Baseline files read	Wrapper files read	Answer coverage
Team-budget enforcement	17.4s	34s	5	3	Full in both arms
OAuth login and credential storage	18s	18s	8	3	Full in both arms
Dynamic-routing rendering	14s	20s	4	4	Full baseline; wrapper missed one deployment-boundary fact
Median	17.4s	20s	5	3	Baseline 3/3 full; wrapper 2/3 full

The read: the wrapper fixes the invalid-command problem and can reduce source
exploration. It did not beat disciplined search on elapsed time in this sample,
and the routing answer shows why correctness remains the first gate. The
wrapper found the routing renderer but missed that the active data-plane
configuration is separately rendered and a per-team dynamic reload is still a
follow-up.

No retrieval-token trace was captured for this run. Fewer files opened is not
a token-savings result. The earlier focused token test still stands: Graphify
beat opening four full files, but it did not beat careful search plus narrow
snippets. The next study needs captured graph output and source excerpts under
the same tokenizer, then a blinded answer-key score before comparing medians.

Adoption Decision

Keep Graphify available only behind an explicit, freshness-aware wrapper for
unfamiliar, cross-layer questions. Do not commit graph artifacts, enable a
repository hook, or use semantic document extraction yet.

The next run needs the full twelve-task counterbalanced design, matched task
difficulty, and an independent reviewer who does not know the condition. Until
then, the result is directional evidence about navigation cost, not a published
productivity claim.

The Open Question

A codebase graph can reduce rediscovery. It can also become stale, omit an indirect dependency, or encourage an agent to trust a summary over the code. The pilot measures the first benefit. It does not settle the freshness problem.

The question is not whether an agent can draw a map. It is whether the map improves a correct decision before the code changes underneath it.

Prompt Caching Is a Harness Design Problem, Not an API Toggle

Amit — Sat, 25 Jul 2026 04:13:34 +0000

TL;DR

A 186-request matrix produced 84.4% normalized input savings with one write and 20 reads, but changing the cache key every call cost 25% more than uncached input.
Reordering tools or changing a description or schema forced a full rewrite; changing tool results after a stable breakpoint preserved the cached prefix.
TTL controls how long reusable work survives. The harness controls whether two requests contain reusable work at all.
Treat rendered-prefix stability, cache reads, cache writes, and post-compaction recovery as production metrics, not provider defaults.

A cache flag can be enabled while an agent pays to rebuild the cache on every call.

That is the central mistake in how prompt caching gets discussed. The API exposes the capability. The harness determines whether the capability produces a hit, a miss, or a stream of expensive writes.

I tested that distinction across 186 successful requests in 40 staged agent sessions, then ran a separate 30-track retention suite across OpenAI and Claude models on Amazon Bedrock. The result was consistent: prompt caching is not primarily a TTL decision. It is a request-construction discipline.

The provider owns the cache. The harness owns the prefix.

The same API produced opposite economics

The test matrix covered implicit and explicit caching, append-only conversations, prompt sizes from below the cache threshold to 240,000 tokens, tool mutations, cache-key changes, concurrent calls, and delayed retention probes.

The strongest cost result came from the simplest pattern: write one stable prefix, then reuse it.

Request pattern	Normalized input savings
One cache write, one cache read	32.5%
One cache write, three cache reads	61.2%
One cache write, five concurrent reads	70.8%
One cache write, twenty concurrent reads	84.4%
New cache key on every request	25% more expensive

The calculation uses the current GPT-5.6 pricing shape documented by OpenAI and Amazon Bedrock: writes cost 1.25 times normal input, while reads receive a 90% discount.

The feature did not create the savings. Reuse did. With no reuse, the write premium made caching worse than ordinary input.

That distinction becomes more important inside an agent loop. A coding harness rebuilds a request from system instructions, repository guidance, tool definitions, conversation history, tool calls, tool results, environment state, and the newest user turn. Every one of those components can preserve or destroy the common prefix.

Tool catalogs are part of the cache key

The tool tests made the harness boundary visible.

An unchanged tool catalog produced an 8,556-token cache read. Reordering the same tools caused a new write. Changing one tool description caused a new write. Changing one JSON schema caused a new write.

The tools remained semantically similar. The rendered prefix did not.

This matches the provider contracts. OpenAI requires tools and images to remain identical for an exact prefix match. Amazon Bedrock processes cacheable content in tools → system → messages order; changing tools invalidates the system and message caches that follow. Anthropic documents the same hierarchy.

The practical consequence is larger than “sort your JSON.” The tool registry is now part of the cost architecture.

A harness that discovers tools dynamically, changes descriptions at runtime, emits schemas in nondeterministic order, or injects session-specific metadata before its stable instructions can have caching enabled and still miss continuously. Tool discovery, serialization, and prompt construction are no longer separate implementation details. They meet in the rendered prefix.

Dynamic work belongs after the boundary

Dynamic tool results behaved differently in the same matrix.

I kept the instructions and tool catalog fixed, placed an explicit breakpoint after them, and changed only the tool result that followed. Both requests retained the same 8,556-token cache hit with no new write.

That is the architecture:

[stable tools]
[stable system instructions]
[stable repository or task context]
[cache breakpoint]
[conversation turns]
[tool calls and results]
[latest user input]

The exact fields differ by provider. The invariant does not: stable content first, volatile content later.

OpenAI's Prompt Caching 201 guide describes the same pattern in Codex: system instructions, tool definitions, sandbox configuration, and environment context stay consistently ordered, while new messages append to the conversation. The guide also reports one coding workload improving its hit rate from 60% to 87% after adding a stable prompt_cache_key.

The cache key matters because reuse has two conditions: the prefix must match, and requests must reach compatible cache capacity. A key that changes every call fragments reuse. A key shared across too much traffic can overflow one routing bucket. The harness owns that partitioning decision.

TTL is retention, not reuse

The original version of my prompt-caching explainer overemphasized the five-minute TTL. The live tests showed why that is the wrong organizing principle.

Provider and model contracts differ:

Tested model family	Requested retention	Observed result
GPT-5.6 Luna and Terra	30-minute minimum	Hit at 10 minutes; rewrite at 31 minutes
Claude Sonnet 5 and Opus 4.8	5 minutes	Rewrite at 6 minutes
Claude Sonnet 5 and Opus 4.8	1 hour	Hit at 31 minutes; rewrite at 61 minutes

Each delayed probe used an independent cache entry, so an earlier hit could not refresh the entry tested later. For GPT-5.6, 30m is the only supported value and sets a minimum lifetime, not an exact expiration. OpenAI may retain an entry longer. These observations establish the cache state at each probe time, not the exact millisecond of eviction.

TTL answers, “How long can matching work remain reusable?” Prefix identity answers, “Is this still the same work?” The second question belongs to the harness.

The six harness responsibilities

Prompt-cache performance emerges from six parts of the agent harness.

Harness component	Cache responsibility
Instruction manager	Keep durable instructions byte-stable; move timestamps and request IDs out of the prefix
Tool registry	Use deterministic tool order, descriptions, and schemas
Context builder	Put shared repository context before session-specific state
Conversation manager	Append turns instead of rewriting earlier history
Compactor	Treat compaction as a planned cache reset, then stabilize the new summary
Router and telemetry	Use stable cache keys and measure reads, writes, misses, cost, and time to first token

Compaction exposes the hardest tradeoff. Removing or summarizing old turns reduces context size, but rewriting history also breaks the existing prefix. OpenAI's caching guide calls this tension out directly: context engineering rewards selective change, while prompt caching rewards stability.

The right response is not to avoid compaction. It is to make the reset observable. Record the prompt and tool bundle fingerprint, expect one cold write after compaction, then verify that later turns read the new prefix. A cache miss caused by an intentional state transition is healthy. A cache miss caused by randomized tool order is a regression.

The research reaches the same conclusion

The strongest external confirmation comes from a study of more than 500 long-horizon agent sessions across OpenAI, Anthropic, and Google models.

The study measured 45–80% API cost reductions and 13–31% time-to-first-token improvements. Its more important result was architectural: system-prompt-only caching and strategies that excluded dynamic tool results produced more consistent benefits than naive full-context caching. In one condition, full-context caching made latency worse because the system paid write overhead for dynamic content that was not reused.

That is the same failure mode the cost table exposes. A cache write is an investment. The harness needs to decide whether the content before a breakpoint has a realistic path to later reads.

Anthropic describes the production consequence even more directly in its Claude Code prompt-caching engineering notes: the team builds the harness around prompt caching, alerts on hit-rate regressions, preserves the main conversation prefix during compaction, and avoids changing tool definitions mid-session. That is not an API configuration checklist. It is an agent architecture.

Latency also needs separate treatment. In my matrix, cache behavior was functionally reliable, but short-prompt latency did not improve consistently. One 128,000-token Terra request improved from 3.935 seconds cold to 1.868 seconds warm; the comparable Luna calls were effectively unchanged. Cost and token accounting were stronger signals than a handful of end-to-end timing samples.

What's missing: cache observability at the harness layer

Provider metrics report cached tokens and cache writes. They do not explain which harness change fragmented the prefix.

A production harness needs four additional signals:

A fingerprint for the rendered stable prefix.
A separate fingerprint for the ordered tool bundle.
Cache-read and cache-write tokens by prompt version.
An event marking intentional resets such as compaction, tool-version changes, or model switches.

Without those signals, a declining hit rate appears as a larger bill. With them, it becomes a deploy regression tied to a specific prompt or tool bundle.

Anthropic has started exposing cache diagnostics that identify whether two requests diverged in tools, system instructions, or message history. The broader opportunity is provider-neutral diagnostics inside the harness, before requests leave the application.

So what

Enabling prompt caching is the first line of implementation, not the end of the work.

The durable pattern is simple: serialize stable instructions and tools deterministically, place dynamic state after a deliberate boundary, append conversation turns, partition cache keys around real reuse, and make writes as visible as reads.

The unresolved part is dynamic tool discovery. Loading every tool preserves one stable catalog but spends context and can reduce tool-selection quality. Loading tools on demand improves relevance but changes the prefix. Deferred tool search and request-level tool restrictions offer a middle path, but the right balance still needs workload-level measurement.

That is the next cache problem for coding agents: not whether the API supports caching, but whether a changing tool surface can remain both selective and reusable.

Flux LoRA fine-tuning on fal.ai is shockingly simple

Amit — Tue, 21 Jul 2026 02:44:45 +0000

I needed consistent visual style across 50+ blog post images. Traditional fine-tuning pipelines felt heavy. fal.ai changed that.

This isn't a toy example, either — the header image on this very post was generated by the LoRA trained through this exact pipeline. Same base model, same platform, no separate demo needed.

The two-call pattern

Inference is one line:

import { fal } from "@fal-ai/client";

const result = await fal.subscribe("fal-ai/flux/dev", {
  input: {
    prompt: "Fallingwater in the style of aclvisual, architectural photography",
    image_size: "portrait_4_3",
    num_inference_steps: 28,
    guidance_scale: 3.5,
    num_images: 1,
  },
});

Fine-tuning a LoRA is two calls:

Upload your reference images to fal storage.
Start the training job.

// Upload images
const url = await fal.storage.upload(zipBlob, { filename: "refs.zip" });

// Train
const training = await fal.subscribe("fal-ai/flux-lora-fast-training", {
  input: {
    images_data_url: url,
    trigger_word: "aclvisual",
    steps: 500,
    create_masks: false,
  },
  onQueueUpdate: (update) => {
    if (update.status === "IN_PROGRESS") console.log(update.logs);
  },
});

console.log("LoRA ready:", training.lora_url);

Then use the trained LoRA in inference exactly like any other model:

const result = await fal.subscribe("fal-ai/flux-lora", {
  input: {
    prompt: "...",
    lora_url: training.lora_url,
    trigger_word: "aclvisual",
  },
});

What actually surprised me

No custom training scripts. No GPU management. No dataset formatting hell.
Training jobs are async and resumable. Close your terminal; the job keeps running.
The resulting LoRA is a single .safetensors file hosted by fal. You can use it immediately or download it.
Cost is predictable: ~$2–5 to train on 8–12 images, then pennies per generation.

I used this exact flow to train the house style LoRA that produces every post header image on this site, then regenerated the entire catalog against it. The whole process (upload → train → dozens of images) took under 30 minutes of wall time and cost less than a coffee.

Where it still falls short

fal's strength is speed and simplicity. What it doesn't give you is deep control over the training process itself. If you need custom schedulers, very specific captioning strategies, or multi-concept training with heavy regularization, you will eventually outgrow the hosted endpoints and move to something like Diffusers on your own hardware or a more configurable platform.

For most practitioner use cases—style LoRAs, character consistency, product shots—the hosted path is not just "good enough." It is dramatically better than the alternatives I tried.

So what

If your mental model of fine-tuning still includes "spin up a GPU, clone repo, fight with dependencies," update it. fal has collapsed the distance between "I have some images" and "I have a working LoRA I can call from code" to roughly the same effort as calling an inference endpoint.

The open thread: how long until every major model host offers this same two-call training experience as a table-stakes feature? And what does that do to the current "fine-tuning as a service" market?

Building AI Voice Agents with Amazon Nova Sonic and LiveKit: Patterns, Architecture, and a RAG Latency Surprise

Amit — Tue, 21 Jul 2026 01:22:15 +0000

I spent a few hours building a series of voice AI agents on Amazon Nova 2 Sonic and LiveKit's Agents framework — starting from a basic voice round-trip and working up to a phone-answering restaurant concierge with real persistence, and a separate agent grounded in a real knowledge base. This is what actually held up: the patterns worth reusing, the architecture that survived a real phone call, and one latency finding that contradicts the standard assumption about voice RAG.

The stack

Amazon Nova 2 Sonic is a speech-to-speech model — it runs speech understanding, reasoning, tool use, and speech generation as one bidirectional stream, not a chained ASR → LLM → TTS pipeline. That matters for more than latency: acoustic context — tone, pace, hesitation — never gets flattened into a text transcript in between, so the model can respond to how something was said, not just what was said.

LiveKit's Agents framework handles the part that has nothing to do with conversational AI and everything to do with real-time media: audio capture and streaming, voice activity detection, turn-taking, WebRTC transport, session lifecycle. LiveKit's own plugin for Nova Sonic means you write an AgentSession and a handful of tools; you don't write a media pipeline.

Together, that's a small, honest surface area: the model owns the conversation, LiveKit owns the wire.

Patterns that held up

A few things worked consistently well enough, across enough different agents, that I'd call them defaults rather than one-off wins.

Interruption and multi-turn memory work out of the box. Barge-in — talking over the agent mid-response — and remembering earlier turns in the same conversation both worked cleanly on the first real test, with no special handling required on my end. This is the thing a cascaded STT/LLM/TTS pipeline has to engineer explicitly; a native speech-to-speech model gets it from the architecture.

Tool calls stay responsive during slow operations, if you let them. A simple tool call (a time lookup) completed in about a second. A deliberately slow tool (a 5-second simulated lookup) didn't block the conversation — the model replied to a second, unrelated request while the slow tool was still in flight, then picked back up within roughly 0.01 seconds of the tool actually completing. It's watching for completion, not blocking on it. That's a real design affordance: you can let a slow backend call run without the caller sitting in dead air the whole time, as long as your prompt gives the model permission to keep talking.

Small toolsets, with a confirmation gate before every write. The restaurant concierge agent runs on 9 tools — inside LiveKit's own guidance of 5 to 10 tools per agent, since tool-selection accuracy degrades past that. Every write (booking a reservation, confirming an order) takes a read_back parameter the model can only set to true after it has actually spoken the details back to the caller and gotten confirmation — call it with read_back=false and the tool hands back the exact wording the model should use to read the details back first. Writes also run with interruptions disabled, so a caller talking over the agent mid-confirmation can't leave a booking half-committed. That's the difference between a demo and something you'd trust with a real reservation.

Multilingual support is broad, and the one documented limitation is narrower than it reads. Nova 2 Sonic supports 16 voices across 8 languages, including mid-conversation language switching. AWS's own AI Service Card states plainly that the model "does not support real-time speech-to-speech translation," and that's true if you ask it to act as a continuous interpreter — across repeated testing, it occasionally self-limited on that framing, in-language, and then still attempted a partial translation anyway. For straightforward one-off sentence translation, though, it was accurate and consistent across dozens of test runs in two languages. Worth knowing the difference before you rule the capability out entirely.

Telephony deployment looks nothing like a call center you'd provision yourself. A phone number tied to a LiveKit SIP dispatch rule routes inbound calls to an explicitly named agent — different from the automatic dispatch used for browser/local testing, and LiveKit's documented recommendation for telephony specifically. The worker that actually answers calls runs as a container on LiveKit Cloud's managed compute, deployed the same way you'd deploy any containerized service — and it scales to zero replicas when nobody's calling, spinning one up per inbound call. There's no EC2 instance sitting idle, no process to keep running by hand. The AWS side of the bill is similarly usage-shaped: DynamoDB on on-demand billing, Bedrock metered per invocation. This is a serverless cost model end to end, not a traditional always-on stack.

Reference architecture: telephony + persistence

This is the shape that actually took a real inbound phone call, in production:

flowchart TD
    A["Caller (PSTN)"] --> B["LiveKit SIP number<br/>dispatch rule routes by agent_name"]
    B --> C["LiveKit Cloud managed agent<br/>scales to zero when idle,<br/>one replica per inbound call"]
    C --> D["Nova 2 Sonic — bidirectional stream<br/>speech understanding + reasoning +<br/>tool selection + speech generation<br/>(one model, one connection)"]
    D --> E["Read-only lookups<br/>menu, hours, availability<br/>served from a static fixture"]
    D --> F["Write attempt<br/>book reservation / confirm order"]
    F --> G{"read_back confirmed<br/>by caller?"}
    G -->|no| D
    G -->|yes| H["Commit write"]
    H --> I["Persistence — DynamoDB<br/>on-demand billing<br/>keyed by confirmation code"]

No knowledge base in this path — the menu and hours are a static fixture, deliberately kept simple. That's a separate pattern, tested as its own agent.

Grounding a voice agent in a knowledge base — and a real latency surprise

Adding Retrieval-Augmented Generation to a voice agent is straightforward: register a tool backed by Amazon Bedrock Knowledge Bases, let the model call it, let it speak the retrieved content back. I tested this as its own standalone agent, against a real technical knowledge base — not a toy dataset — and it worked correctly: asked about a specific technical topic, the reply cited real, specific details that were actually present in the source content, not fabricated.

But the response took noticeably longer than every other tool I'd tested — 46 to 100-plus seconds end to end, against roughly a second for a simple lookup. The instinct, and the standard advice, is to blame the vector database. A Salesforce AI Research paper on voice RAG frames the problem exactly that way: "A typical vector database query adds 50–300ms to the response pipeline, which... pushes total latency well beyond the 200ms budget required for natural conversational flow." That's a reasonable place to start, and for a lot of RAG pipelines it's exactly right.

I wanted real numbers instead of a guess, so I isolated each leg of the pipeline and timed it separately, across three identical runs:

Leg	Run 1	Run 2	Run 3
Direct retrieval, isolated (no model)	0.46s	0.40s	0.41s
Time to tool invocation (model decides to call it)	1.24s	1.08s	1.21s
Retrieval call itself, inside the pipeline	0.55s	0.48s	0.53s
Tool completion → final spoken reply	89.67s	55.39s	68.53s
Total, request to reply	91.47s	56.95s	70.28s

Retrieval and tool-decision latency were both fast and consistent — under 1.3 seconds combined, matching an isolated direct-call benchmark almost exactly. The summarization step — Nova Sonic turning roughly 1,500 characters of dense retrieved content into a spoken answer — accounted for over 99% of total latency, every single time, ranging 55 to 90 seconds by itself. It wasn't a bug or a fluke: I initially suspected a known class of Nova Sonic tool-calling issue where the model appears to hang after a tool response when multiple tools are registered, but isolated single-tool testing with a longer timeout ruled that out cleanly. The model wasn't hanging — it was genuinely taking that long to compose a spoken answer from that much dense source material.

That's the opposite of where the conventional wisdom points. If you're building a voice agent grounded in a knowledge base and you're worried about latency, a faster vector database or a semantic cache in front of it is an answer to a question you probably don't have. The lever that actually moves is what you hand the model to summarize: fewer chunks, shorter chunks, or a retrieval step that returns something closer to a direct answer than a stack of dense paragraphs for the model to condense on the fly.

It also changes what you tell the caller. A 60-to-90 second silence on a phone call reads as broken, even when it's working correctly. The fix that mattered most wasn't architectural — it was conversational: have the agent say "let me check that" before it goes quiet, the same thing a real support rep says when they're actually looking something up. That one line of UX design did more for the felt experience than any latency optimization would have.

Where this fits in the industry

Voice AI has moved past proof-of-concept into a real, fast-growing category — one market estimate puts voice AI agents at $2.4B in 2024, growing to $47.5B by 2034 at a 34.8% CAGR, with BFSI and large enterprises as the earliest heavy adopters. In the restaurant vertical specifically — the use case I built the telephony pattern around — the pattern is consistent across every case study I found: Slang.ai reports 10x ROI and 200+ reservations booked monthly per location on average across its customer base, with 25M+ calls processed across 2,000+ locations. The category is crowded now — Loman, Hostie, Revmo, and others are all building variations of the same core loop: answer the phone, understand intent, take the action.

What's notable is that the reference pattern for this use case is converging across vendors, independent of which model or framework you use. LiveKit's own official example for its Agents framework is, not coincidentally, a restaurant reservation and ordering system with a small set of well-scoped tools — the same shape I converged on independently.

The other clear industry signal: real telephony integration is now a first-class feature, not a bolt-on. Nova 2 Sonic launched with direct integration into Amazon Connect, Vonage, Twilio, Audiocodes, and media platforms like LiveKit and Pipecat — the industry has moved past "can this work on a phone call" to "which integration path fits your existing infrastructure."

What's still open

I haven't yet combined the two patterns — a phone agent that both takes orders and answers open-ended knowledge questions in the same session. If I did, I'd keep the grounded-lookup tool on its own latency budget and give the model explicit permission to acknowledge the wait for that specific tool, rather than treating every tool call as equally fast. I also don't yet know where the summarization-latency curve bends — at what chunk size does 60-second summarization become 15 seconds, and is that relationship linear or does it fall off a cliff past some token threshold? That's the next thing I want real numbers on, not a guess.

Which Tier Does Your Vector Workload Live On?

Amit — Thu, 09 Jul 2026 22:33:51 +0000

TL;DR

Since Bedrock Knowledge Bases launched in late 2023, the default vector store was OpenSearch Serverless with a ~$700/month floor you paid whether you queried it or not. Amazon S3 Vectors added a true pay-per-use tier, and the question changed to which tier of the cost/latency curve your workload belongs on.
The tiers sort cleanly: hot (OpenSearch, sub-50ms, always-on), warm (Aurora + pgvector, beside relational data), cold (S3 Vectors, ~100ms, pay-per-use). Picking a store is picking a tier, not picking a winner.
The cold tier is a category the market already validated. Turbopuffer built a ~$100M business storing vectors on object storage for Cursor, Notion, and Anthropic; Cursor cut its vector-database cost 95% moving to it. S3 Vectors is AWS making that same architecture a managed primitive.
Agent workloads belong on the cold tier, and the reason is structural: an agent's retrieval is one step inside a multi-second reasoning loop, so 100ms versus 10ms is noise. The latency that disqualifies a customer-facing search box is invisible to an agent.
S3 Vectors is not a database replacement. It is semantic-only, has no hybrid search, and trades single-digit-ms latency for cost. Know the three limits before you build on it.

When Amazon Bedrock Knowledge Bases launched at the end of 2023, the default vector store was OpenSearch Serverless, and it came with a bill before you stored anything. The floor was roughly $700 a month — four OpenSearch Compute Units (two for indexing, two for search with standby) at about $0.24 per unit-hour, running whether you queried once a day or a thousand times a second. For a production search system, that floor is rounding error. For a personal knowledge base or an internal document search, it is the reason the project never ships.

Amazon S3 Vectors, generally available since December 2025, prices the way the rest of S3 does: no minimum charge, pay for what you store and what you query, nothing while idle. One 30-project analysis puts the entry point at roughly $0.60 for a first month against OpenSearch Serverless's hundreds, and finds S3 Vectors 15 to 66 times cheaper across small and medium workloads. OpenSearch has since narrowed its own gap — fractional compute units and a scale-to-zero tier cut the idle cost — but the arrival of a true pay-per-use tier is what reset the default. The question stopped being "which vector database wins" and became "which tier of the cost and latency curve does this workload belong on."

The Tiers

Every AWS-native vector option now sorts onto one curve that trades latency for cost. Picking a store is picking a point on that curve.

Tier	Latency	Fits	AWS option
Hot	Sub-50ms, always-on	High QPS, customer-facing search	OpenSearch Service
Warm	50–200ms	Vectors beside relational data, ACID	Aurora PostgreSQL + pgvector
Cold	~100ms warm, sub-second cold	Large or long-lived, infrequent or bursty	S3 Vectors

Amazon Bedrock Knowledge Bases sits across the whole curve as the managed RAG layer, and it supports every store as a backend — OpenSearch Serverless and Managed Cluster, Aurora, Neptune Analytics, Pinecone, MongoDB Atlas, Redis, and S3 Vectors. You choose the tier; Bedrock runs the ingestion, chunking, embedding, and retrieval on top of it. The store is a decision about cost and latency, not a decision about capability.

The shift since S3 Vectors arrived is that the top and bottom of the curve now combine instead of compete. You keep hot vectors in OpenSearch for the queries that need speed and spill the long tail into S3 Vectors for cost, and AWS supports importing directly from one to the other. "Pick one database" was the wrong frame; "which tier for which slice of the data" is the right one.

The Cold Tier Is Already a Business

S3 Vectors can read like an AWS experiment if you have not been watching the vector-database market. It is AWS's managed entry into a pattern that a startup has already turned into a company.

The storage math behind the cold tier is not subtle. When you embed text, the vectors are much larger than the source — Turbopuffer measures 1KB of text expanding to roughly 16KB of vector data after chunking and embedding. Keeping all of that in memory or on SSD is where the bill comes from. Object storage runs about $0.02 per GB against roughly 16 times that for SSD, and most vectors in a real corpus are read rarely. Storing the cold majority on object storage and caching only the hot slice is the whole idea.

Turbopuffer built its company on exactly that idea. It stores vectors on S3, GCS, or Azure Blob and caches hot data on SSD and RAM by access pattern. Sacra estimates it reached about $100M in annualized revenue by March 2026, up 2,400% year over year, serving Cursor, Notion, Anthropic, Linear, and Superhuman. When Cursor moved its codebase index to this architecture, it reported cutting vector-database cost by 95% — because most codebases are queried infrequently, and paying in-memory rates to keep them all live was the expensive mistake.

Turbopuffer is one vendor of several. LanceDB is an open-source, file-based vector store that runs directly on any S3-compatible backend, and AWS itself published an architecture for 1B+ vectors on LanceDB, S3, and Lambda where the only fixed cost is the S3 storage footprint and each query costs fractions of a cent. Spice.ai integrated S3 Vectors into its open-source engine. The industry even converged on shared vocabulary for it — hot, warm, and cold vector tiers — because enough teams are building this way that the pattern needed names.

S3 Vectors is AWS making that architecture a managed primitive: the cold tier without running the caching layer yourself. The reason to trust the tier is that a $100M business, a marquee set of AI customers, and AWS's own reference architecture all point at the same economics.

Why Agents Belong on the Cold Tier

Here is the insight that makes S3 Vectors the default for agent work, and it is easy to miss if you only look at the latency number.

A customer-facing search box lives or dies on latency. A user typed, a box must fill, and 100 milliseconds versus 10 is the difference between snappy and sluggish. That workload belongs on the hot tier, and the OpenSearch bill is the cost of the experience.

An agent is a different shape of workload. When an agent retrieves, the retrieval is one step inside a reasoning loop that already takes seconds — the model is thinking, calling tools, reading results, thinking again. As one analysis puts it, adding a vector lookup that takes 100ms instead of 10ms is rarely the bottleneck when the model-side latency of a single tool call is already 500ms to several seconds. The 90 milliseconds you save on the hot tier vanish inside a loop that was never going to be fast. You paid the always-on floor to optimize a step that is not on the critical path.

This is why AWS positions S3 Vectors as purpose-built for agent memory: agent memory grows continuously, is queried in bursts, and tolerates hundreds of milliseconds because it lives inside a slow loop. The cold tier fits the shape of the work. The hot tier would be paying a premium for speed the agent cannot use.

The access pattern is the same one that made the cold tier a business. Cursor's codebases sit dormant most of the time and get queried in bursts; agent memory has the identical profile — mostly idle, occasionally read, never on a human's critical path. The workload that justified an object-storage-native database for code search is the workload most agents already have.

Where S3 Vectors Fits

The official use cases share one profile: large or growing corpora, moderate or bursty query rates, latency budgets measured in hundreds of milliseconds, and a preference for zero infrastructure. That profile shows up in five recurring patterns:

Semantic search over large collections — documents, media, medical images, video archives where cost per vector dominates.
RAG long tail — the bulk of a knowledge base that is queried occasionally, with a hot tier in front only if some subset gets constant traffic.
Agent memory — persistent, growing, bursty, latency-tolerant. The canonical fit.
Batch evaluation corpora — embeddings you query in batches, not in real time.
Cold tier of a tiered design — the durable, cheap floor under an OpenSearch hot tier.

My own case is the RAG long tail. I put my blog — 70-plus posts — into a Bedrock Knowledge Base on S3 Vectors. Small corpus, queried when I am writing and want to know what I already said, entirely latency-tolerant. A retrieval returns the relevant posts with strong semantic scores:

query: "do agent memories fade while skills persist"
  0.825  memories-fade-skills-persist.md
  0.753  skills-are-git-native-distribution.md
  0.725  agent-sprawl-is-a-skills-problem.md
  0.714  what-is-a-skill.md

I reach that knowledge base from my agents through one tool on a gateway — a pattern worth its own post, How AgentCore Gateway Turns Any API Into an Agent Tool. The point here is the storage decision underneath it: this workload is cold-tier, and the cold tier no longer costs $700 to enter.

The Three Limits

S3 Vectors buys its cost advantage with tradeoffs, and a design that ignores them will disappoint.

Semantic-only, no hybrid search. AWS documents that S3 Vectors does not support hybrid search — the combination of vector similarity with keyword matching. For content full of exact tokens like part numbers or version strings, pure semantic recall scores poorly, and OpenSearch with lexical scoring remains the right tier. My prose has no such tokens, so the limit does not bite, but a corpus of technical specs would feel it on the first query.

Latency is a floor, not a target. Warm queries land around 100 milliseconds, cold ones sub-second — fine inside an agent loop, disqualifying for an interactive search box. When part of the workload needs speed, the answer is not to abandon S3 Vectors but to tier: keep the bulk cold, promote the hot subset to OpenSearch, which AWS supports importing into directly.

Metadata is capped. S3 Vectors limits custom metadata per vector, which constrains how much filtering you can push into the store. For rich per-document filtering at scale, that cap is a real design input, not a footnote. I take apart exactly what that cap does to ingestion and filtering in What Actually Gets Stored When You Put a Vector in S3 Vectors.

So What

Stop asking which vector database is best. Start asking which tier your workload sits on, because the answer is usually obvious once you look at the shape of the queries. Customer-facing and latency-critical goes hot. Sitting next to relational data goes warm. Large, growing, bursty, and patient — which describes most internal RAG and nearly all agent memory — goes cold.

The reason this matters now is that the cold tier stopped being expensive. The $700 floor was quietly deciding architectures — pushing people toward one always-on database because standing up anything felt like the same fixed cost. Remove the floor and the honest answer for most agent and internal workloads is the cheap tier, running at pennies, tolerating a latency the agent never notices. The premium tier is still there when you need speed. Most of the time, you do not.