DEV Community: AWS Community Builders

Has Apache Iceberg Really Solved Vendor Lock-in? Some Thoughts

Aki — Sat, 26 Sep 2026 14:09:54 +0000

Original Japanese article: Icebergは本当にベンダーロックインを解消したのか考えてみた

Introduction

I'm Aki, an AWS Community Builder (@jitepengin).

With the spread of Apache Iceberg, I hear more and more that data platforms have become more open.
"Iceberg is an open format, so you can avoid vendor lock-in" is an explanation I see often, and one I've given myself.

Working with data platforms built on AWS, as well as Databricks, Snowflake, and Microsoft Fabric, I've found plenty of situations where that explanation holds true.
But I started wondering how far it really goes, so this time I want to sort out which kinds of lock-in Iceberg has solved, and which it hasn't.

To give the conclusion up front: what Iceberg achieved is data portability.
Governance, meaning permissions, auditing, lineage, and sharing, does not travel with the data in the same way.
My current view is that lock-in hasn't disappeared; it has simply moved up to a higher layer.

By the way, I'm a big fan of Unity Catalog!

Where Did Lock-in Used to Live?

Let's start by looking back at how things used to be.

In traditional data warehouses and data lakes, the file format, the way metadata was managed, transaction management, partition management, and the access APIs were all closed off inside the service.
In that structure, "the data living inside that service" was itself the lock-in.
Moving to a different environment meant starting by extracting all of the data and converting it.

Open table formats are what solved this.
Data files sit in cloud storage such as S3 as Parquet, and metadata sits right alongside them in an open specification.
Because both specifications are public, neither is locked into a vendor-specific format.

So what, then, is not open?

The Catalog Defined by the Iceberg Spec Is Actually Very Narrow

Let's go back to the spec and check what role the catalog plays in Iceberg.

The "Overview" section under "Catalog" in the official Apache Iceberg documentation explains that creating, dropping, and renaming tables are the catalog's responsibilities, and that its most important responsibility is tracking a table's current metadata and providing it when the table is loaded.
It also says that multiple kinds of compute engines can share a common data layer by sharing the same catalog.

https://iceberg.apache.org/terms/

So how does that "current metadata" get updated?
That's covered in the Table Spec.

https://iceberg.apache.org/spec/

Every change to table state creates a new metadata file, which replaces the old metadata through an atomic swap.
Writers optimistically create a metadata file on the assumption that the current version won't change before they commit, and then commit by swapping the table's metadata pointer from the base version to the new version.
This is what's known as compare-and-swap (CAS): even when multiple writers commit at the same time, updates based on stale state can be detected and treated as conflicts.

Here's what that looks like as a diagram.

   Catalog
┌──────────────────────────────────────┐
│  table_a → s3://.../v3.metadata.json │
└──────────────────────────────────────┘
              │
              │ Pointer to current metadata
              ▼
┌──────────────────────────┐
│  metadata.json           │  ← Open
│    schemas / snapshots   │
│    partition-specs       │
│  manifest list / files   │  ← Open
└──────────────────────────┘
              │
              ▼
┌──────────────────────────┐
│  Parquet / S3            │  ← Open
└──────────────────────────┘

Schemas, snapshots, and partition specs all actually live inside metadata.json.
Managing the list of snapshots used for time travel is not the catalog's responsibility either.

One thing worth pointing out here: the same Table Spec states that the atomic operation used to commit metadata depends on how tables are tracked, and that the spec does not standardize it.
In other words, what the Table Spec defines stops at "swap the pointer atomically"; where and how that pointer is kept is left to the catalog implementation.
That is the minimum set of catalog responsibilities visible from the table spec.

The REST Catalog Spec Broadened the Catalog's Role

At the implementation level, though, things don't stop there.

To support more languages and engines, the Iceberg project created a common API called the REST Catalog Protocol.
And with the REST Catalog, the commit mechanism itself changes.

https://iceberg.apache.org/rest-catalog-spec/

According to the "Commit protocol" section of the REST Catalog Protocol page, unlike client-side catalogs, a REST client neither writes metadata files nor swaps the pointer itself.

https://iceberg.apache.org/docs/nightly/rest-protocol/

Instead, the client sends the server the conditions that must still hold for the commit to be valid (Requirements), along with the metadata changes themselves (Updates).
The server checks every requirement against the current metadata and, if they all hold, applies the updates and writes the new metadata.

In other words, with the REST Catalog, responsibility for committing has moved from the client to the catalog.
The same page explains that because the server owns the commit logic, it enables server-side conflict resolution, multi-table commits, and secure table sharing through credential vending or remote signing.

Let's also look at credential vending.
The server can include short-lived, table-scoped storage credentials (storage-credentials) in its load-table response.
However, whether to vend credentials is up to the server, and a separate mechanism called remote signing, where the server only signs requests, is also defined.
So the accurate reading seems to be that the spec defines access delegation mechanisms as optional.

What the spec does not define, on the other hand, is the authorization model behind that delegation (how "who can access what" is expressed).
Even though how you call credential vending is standardized, the logic that decides "who gets which permissions," and the way that is expressed, are left to each catalog implementation.

This looks like the boundary with what real-world data catalogs actually cover.

What Real-World Data Catalogs Cover

When you look at catalogs as actual products, they take on even more.

They offer a wide range of capabilities: namespace management, serving table metadata, credential vending, access control including row- and column-level security, tag-based and attribute-based permissions, data discovery, lineage, audit logs, data sharing, and integration with external tables and other catalogs.

This is not at all a criticism of the products.
What we actually expect from a catalog in production is, if anything, these additional capabilities.
A pointer plus a credential vending endpoint doesn't settle the authorization logic of who can read which table, and it doesn't give you auditing either.

The problem is that because these additional capabilities aren't open, each implementation ends up with its own unique shape.

As a concrete example, let's look at Databricks Unity Catalog.
Unity Catalog's credential vending is a mechanism that issues short-lived credentials to external systems connecting via the Unity REST API and the Iceberg REST Catalog.

https://docs.databricks.com/aws/en/external-access/credential-vending

The "Credential Vending" section of Chapter 9, "Open Access," in the book Data Governance with Unity Catalog on Databricks explains that the principal accessing the data must first be granted the necessary read or write permissions, and that each vending API call is recorded in the audit system table as generateTemporaryTableCredential.

The credential vending entry point follows the REST Catalog spec, but both the permission model that decides "who gets which permissions" and the audit mechanism that records those calls are Unity Catalog's own implementation.
Here, too, you can see access control and auditing converging in the same place: the catalog.

Catalog-Specific Capabilities Create Lock-in

Organizing this structure from a migration perspective gives the following.

Layer	Standardization	Moving to another catalog
Data files	Iceberg Table Spec	◎ Usable as-is
Metadata	Iceberg Table Spec	◎ Usable as-is
How the metadata pointer is kept	Not standardized (Table Spec only defines the atomic swap)	○ Tables must be registered in the new catalog
Catalog operation APIs, commits, how credential vending is called	REST Catalog spec	○ The calls are common, but whether credentials are vended and how permissions are decided depend on the implementation
Access control and permission model (who is allowed what)	Out of spec	× Reconfigure or rebuild per implementation
Views	An Iceberg View Spec exists, but exposure and SQL dialects depend on the implementation	△ to × Definitions may need migrating or rewriting
Auditing	Usage metrics reporting is in the spec, but audit log format is not	△ Integration or conversion needed
Lineage	Outside the scope of the Iceberg spec	△ to × Integration, conversion, or rebuilding needed
Data sharing	Access delegation mechanisms are in the spec, but the sharing model is not	△ Redesign depending on the sharing approach
Table maintenance	Outside the scope of the Iceberg spec	△ to × Operations need to be revisited

On auditing, the REST Catalog Protocol includes a mechanism where, if the server exposes a metrics endpoint, clients report scan and commit metrics to it.
See the "Metrics reporting" section below.

https://iceberg.apache.org/docs/nightly/rest-protocol/

In the sense that catalog operators can see how tables are being used, you could say part of auditing is starting to make its way into the spec.
However, it does not define the format of audit logs that record "who accessed what, and when" for governance purposes.

On views, Unity Catalog's official documentation states that while the Iceberg REST Catalog API exposes tables, it does not expose views, so external Iceberg engines cannot read views defined in Unity Catalog.

https://docs.databricks.com/aws/en/iceberg/

So there are cases where you can take your tables with you, but not the views you've built on top of them.

The parts the Iceberg spec covers are genuinely open.
The reason a catalog migration can be "just re-register the tables in the new catalog, without copying data" is precisely because the data and metadata layers are open.

Of course, there are cases where you can export lineage information to another system or feed audit logs into an external platform.
However, nothing in the Iceberg spec guarantees that you can move between catalogs while keeping the same semantics, the same permission structure, and the same operability.

What gets locked in, I think, is not so much the catalog itself as the management and governance machinery built on the assumption of that catalog.
If your permission settings, authorization model, auditing, lineage, sharing, and operational jobs depend on a specific catalog's features or representations, changing the catalog means revisiting all of them too.
And much of what we expect from a catalog in production is concentrated right here.

The Preconditions Behind "The Data Is Still in S3"

So far, I've been writing on the premise that "the data remains in storage, and since it's Iceberg, other engines can read it."
That premise, however, comes with conditions.

The first is that the storage is under your own control.
Unity Catalog's official documentation explicitly states that credential vending is not supported on workspaces that use default storage.

https://docs.databricks.com/aws/en/iceberg/

In setups where data lives in vendor-managed storage, the premise that "storage lock-in has been solved" may not hold in the first place.

The second is that the table is actually managed as Iceberg.

Unity Catalog's Iceberg REST Catalog provides read, write, and create access for managed Iceberg tables, but only read-only access for Delta Lake tables that have Iceberg reads enabled.

https://docs.databricks.com/aws/en/tables/managed

A table that is really Delta underneath and merely readable as Iceberg cannot be written to from outside, so the "◎ for data files and metadata" in the table above doesn't apply as-is.

When evaluating data portability, it seems necessary to check not just the format, but also "where the data lives" and "which metadata is the source of truth."

Why Does Everything Converge on the Catalog?

Why do capabilities like permissions, auditing, sharing, and lineage all end up concentrated in the catalog?

This isn't speculation; the official Iceberg documentation itself explains it.
The "Credential vending" section of the REST Catalog Protocol page states that because vended credentials are scoped to a table's storage prefixes, the catalog becomes the single point of access control.

https://iceberg.apache.org/docs/nightly/rest-protocol/

As mentioned earlier, with the REST Catalog the catalog also takes on commits.
That means both the place that finalizes the table's state and the place that grants access to storage are the catalog.

From here on this is my own interpretation, but if state management and access authorization both converge in one place, it's natural to record audits there too.
The same goes for data sharing and lineage, and as a result, governance capabilities concentrate in the catalog.

In other words, this concentration of capabilities in the catalog could be seen less as vendors trying to lock customers in, and more as a rational design that emerges from trying to centrally manage access to data.
And I feel that this very rationality is what ends up creating a new kind of switching cost.

On top of that, trying to standardize access control runs head-on into each cloud's IAM and identity platform.
Permission settings from one environment don't carry over to another because the way principals are represented is fundamentally different, and reducing them to a common format isn't easy.
I suspect this gap won't be closed anytime soon.

Does Using an OSS Catalog Avoid Lock-in?

Let's also think about OSS here.

Chapter 9, "Open Access," of Data Governance with Unity Catalog on Databricks, the book I mentioned earlier, explains that because Unity Catalog supports both the Unity REST API and the Iceberg REST API, you can migrate to other catalogs that support these formats, which eases vendor lock-in concerns.
The same chapter also introduces a setup that self-hosts the OSS version of Unity Catalog.

I think this is correct when it comes to data portability.
And if you keep using an OSS catalog, you may also be able to move to another cloud or another operator while keeping the same permission model.

But what's solved in that case is "lock-in to a vendor," not "lock-in to a specific catalog implementation."
When moving from one OSS catalog to another, or to a different managed catalog, you still need to revisit permissions, auditing, sharing, and operations.
It may be closer to reality to say that the lock-in target has simply shifted from "the vendor" to "the catalog implementation."

Portability

Based on all of this, I'd like to split the word "portability" into two.

Data Portability

Can the same data be read and written from a different environment, and can the data be migrated? This is what open table formats solved.
That said, it assumes the storage is under your control and the tables are managed as Iceberg.

Governance Portability

Can the same data be moved to another environment while keeping the same permission structure, the same auditing, the same sharing model, and the same operability?
This is the part that remains unsolved.

"Being able to take your data with you" and "being able to take the way your data is managed with you" are entirely different problems.
Looking only at the former and declaring "lock-in has been solved" seems like evaluating only half the picture.

I think it's closer to reality to say that lock-in hasn't disappeared, but that the layer where lock-in can occur has moved up.
As storage, table format, catalog, and governance separate into distinct layers, the center of gravity has shifted upward by exactly as much as the lower layers have opened up.

Choosing a Catalog Is as Weighty a Decision as Choosing an Engine

Engines also have their own SQL dialects, optimizations, UDFs, and operational features, and migrating between them is by no means easy.
But I feel that changing catalogs, which means rebuilding permission design, audit design, and operations all at once, is a qualitatively different kind of cost from an engine migration.
I think it's fair to treat choosing a catalog as a decision just as weighty as choosing an engine, and in some cases even more so.

That said, it's also true that most of the time you'll use your data platform's standard catalog. Or rather, if you want to make full use of a data platform's features, you're often effectively required to use that product's catalog.

Conclusion

This time, I sorted out the lock-in Iceberg has solved and the lock-in it hasn't, by separating things into layers.

To summarize:

What the Table Spec opens up covers data files, metadata, and the concept of atomically swapping the pointer; how the pointer is kept is not standardized.
The REST Catalog spec standardizes commits and access delegation such as credential vending as the catalog's role, and the official documentation itself positions the catalog as "the single point of access control."
On the other hand, the permission model that decides "who gets which permissions," audit logs, lineage, and the sharing model are all outside the spec and specific to each catalog implementation.
Data portability also has preconditions: the storage must be under your control, and the tables must be managed as Iceberg.
Using an OSS catalog lets you avoid vendor lock-in, but lock-in to the catalog implementation remains.
Lock-in hasn't disappeared; it seems closer to reality to say the layer where lock-in can occur has moved up, from storage and format to catalog and governance.

I think the term "open table format" does keep its promise when it comes to data and metadata.
On the other hand, the catalog and governance layers above them are not guaranteed the same portability.

"Being able to take your data out" and "being able to take your data's management and governance out" are separate problems.
That's exactly why, when evaluating a data platform, I think we need to look not only at the openness of storage and table formats, but also at the portability of the catalog and governance layers above them.

I hope this article is helpful to anyone thinking through catalog selection or governance design.

Building Scalable Context Layers for AI Agents with Elastic and AWS AgentCore

N Chandra Prakash Reddy — Sat, 26 Sep 2026 10:25:57 +0000

I was at AWS Community Day Bengaluru recently on 11 July 2026 and the energy was simply fantastic. There were a lot of great presentations throughout the day, but one presentation really caught my eye. Someshwaran Mohan Kumar, presenting on “Building Scalable Context Layers for AI Agents with Elastic Agent Builder and AWS AgentCore”.

Let’s be honest, we’ve all had an experience with an AI chatbot that sounds very smart one minute and then can’t remember what we were talking about the next. Someshwaran tackled this same pain area. If you are designing AI apps, the principles covered in this webinar are absolute game changers!

So here’s my thorough evaluation of the session, and how we can finally give our AI bots long-term memory.

The Evolution of AI: We Are Beyond Simple Chatbots

To fix AI agents, we first need to understand how we got here. The discussion began with a very essential question: what exactly is a Large Language Model (LLM), and how is it different from an Agent? Someshwaran humorously describes our initial view of LLMs with the popular joke, merely typing away madly at a prompt.

But the landscape has changed quickly during the past few years:

2022: We began by building simple wrappers around chatbots with simple APIs.
Early 2023: We went to prompt chains, employing tools to link a series of tasks together.
Mid 2023: We observed the emergence of tool-calling bots able to communicate with external functions.
2024: Workflow engines added logic loops and human-in-the-loop judgments.
2025 to 2026: We have officially reached the era of multi-agent systems in which specialized AI roles (e.g. Researchers, Writers, Reviewers) work over common states.

This evolution has been supported by extensive foundational research, on which the engineering community has depended. The emergence of Retrieval-Augmented Generation (RAG) for knowledge intensive tasks has been witnessed. We also learned about the “Lost in the Middle” phenomena, showing that LLMs suffer from a U-shaped performance curve – they recall the beginning and the end of a long prompt, but information in the center affects model performance dramatically. To alleviate these limitations, frameworks such as “ReAct” evolved, which combined reasoning traces with real actions in the environment.

The "Ghajini" Problem: Why Your AI Forgets You

Now this is when it gets fun. Even with complex models, we run across a huge obstacle when designing agents. Agents are stateless by default.

Someshwaran calls this the “Ghajini Problem” (or the Goldfish Problem). It's like dialing up your bank's customer service line. You talk for 10 minutes about your entire account history, your present problem, your preferences. The rep is helping you perfect yourself. But hang up and call back the next day and the new representative knows absolutely nothing. You have to start from scratch.

And this happens with AI bots every single time.

Session Starts: The user is shown the whole context, goals, preferences, and history.
Agent Responds: The agent works well, uses the context and offers value.
Session Ends: All is forgotten. The memory is wiped clean.

The bottom line. Users have to repeat themselves a lot. The agents never learn new preferences, there is no continuity at all and personalization is not possible. It is a problem of context and a problem of agent.

The Infrastructure Solution: AWS AgentCore

So how do we solve this? To build an effective agent, the two parts, the “Conversation Layer + Infra Layer” and the “Knowledge + Context Layer” must collaborate.

The presentation walked over building the base infrastructure using AWS AgentCore. This is the strong wrapper you need for your custom bot. AWS AgentCore enables you to make use of essential runtime resources such as Gateway for safe API routing, Identity management, Policy enforcement for guardrails, and Evaluations for quality tracking.

But the real magic is the fifth resource: Memory. AWS AgentCore introduces session summarizers, analyst preferences and insights natively. It also allows observability to measure memory events API requests, latency and faults.

Code Walkthrough: Building with Strands SDK

We then use the Strands SDK to actually construct this out. Someshwaran gave a good glimpse of how simple the entry point is for an agent developed on top of this architecture.

This is what the code looks like to spin up a memory aware agent:

# src/main.py - Agent Entrypoint

from strands import Agent
from bedrock_agentcore.runtime import BedrockAgentCoreApp

app = BedrockAgentCoreApp()

@app.entrypoint
async def invoke(payload, context):
    agent = Agent(
        model = bedrock_model,
        tools = mcp_client.tools + es_tools,
        system_prompt = ANALYST_PROMPT,
        session_manager = memory_manager
    )

    response = agent(payload['query'])
    return { 'response': response }

See the @app.entrypoint wrapper? This means your method is a handler compatible with AgentCore, where the memory manager is automatically injected before and after each conversational round.

The Context Solution: Elasticsearch

An agent needs to know the domain really well to be useful and AgentCore keeps track of the communication history. This is where Elasticsearch, the specific Context Layer, comes in.

You might be asking yourself “Is Elasticsearch and AgentCore Memory doing the same thing?” They are not competing; they are complementing. AgentCore Memory is for discussion only (episodic memory, summaries, user preferences). Elasticsearch is, alternatively, domain-centric (RAG over live operational data, hybrid search and semantic insight storage).

Together you obtain an agent that knows exactly what is happening in your business processes right now and remembers the user.

Bridging the Gap with MCP

But to be able to safely hook the agent into this huge pool of Elasticsearch data, we need a standardized means to communicate context. Enter MCP (Model Context Protocol).

Someshwaran explains hilariously accurately with a real-world comparison why MCP exists – comparing how outsiders save phone contacts vs. how Indians save contacts. A foreigner might just save a number as “Alex” or “Mark”. Meanwhile, the Indian contact list reads “Tina Airtel”, “Pappu Jio”, “Crush Airtel.” We naturally provide metadata and context to everything to make it easily searchable. MCP achieves just that for AI models . It fills the gap to standardize passing of information and tools across .

The Before and After: Code Simplification

The effect of MCP is huge for the developers. Let’s look at the difference Someshwaran presented.

Before MCP: You had to build brittle, hard-coded integration code. To simply look for a server error we had to manually configure the client, hardcode the index and build sophisticated JSON match queries.

# Before
from elasticsearch import Elasticsearch

es = Elasticsearch(
    "https://my-elastic.example.com",
    api_key="MY_API_KEY"
)

def search_logs(error_code: str):
    return es.search(
        index="weblogs-*",
        query={
            "match": {
                "status": error_code
            }
        }
    )

results = search_logs("500")
print(results)

After MCP: The code is beautifully clean. You start the agent, you define the bedrock model, and you just pass mcp_client.tools. The Elastic MCP server offers tools (eg execute_esql or search_index) dynamically to the agent.

# After
from strands import Agent

agent = Agent(
    model=bedrock_model,
    tools=mcp_client.tools  # discovered from Elastic MCP server
)

response = agent("""
Find the top 5 countries generating 5xx errors this week
and show the trend over time.
""")

print(response)

Instead of writing database queries you just ask the agent a natural language inquiry and it does the rest using the tools it dynamically discovered.

Architecture Deep Dive and Demo

The architecture makes sense when it’s assembled. The Strands Agent orchestrator takes a user request and coordinates LLM processing with Amazon Bedrock, historical context with AgentCore Memory, and tool access with the MCP Client. The MCP Client connects to Elastic Cloud Serverless securely to query live indices.

The live demo showed this architecture using an Elastic AI Agent UI. The user only asked “Describe what data I have available” and the agent took the autonomy to list the 8 indices present in the cluster, automatically categorizing them into CRM & Customer Data and Marketing Data.

Key Takeaways

So here's the bottom line on building your own context-aware AI agents:

Memory turns a stranger into a trusted partner: Think of going to your favorite neighborhood coffee shop every morning. When the waitress remembers you enjoy an oat milk latte with no sugar, you immediately feel loved. Without memory, your agent is like a stranger who forgets your face the second you walk out the door. AWS AgentCore gives your AI a friendly memory of user preferences and interaction history, spanning sessions.
Context works best as a team effort: Don't ask one database to do everything. Think about ordering dinner on an app like Swiggy. AgentCore is like a saved profile, it remembers your dietary preferences and previous orders. Elasticsearch is the restaurant kitchen and delivery fleet – it’s keeping tabs on what ingredients are in stock, current order status and road conditions right now. Together, they ensure sure the user always gets the proper result.
Standardize how your tools connect: Remember when every phone brand had a separate charging pin and you required a drawer full of adapters? It used to be as messy connecting tools to AI bots. The Model Context Protocol (MCP) is an universal USB-C cable. Your agent can plug directly into Elasticsearch tools, no hard-coded bespoke code needed.
If you are building a startup: Now you can deploy customer care agents that remember earlier discussions, no more users needing to re-explain their concerns from scratch.
For engineering and operations teams: You can deploy internal AI assistants who automatically analyze live production logs, and troubleshoot faults, saving your team hours of manual querying.

Conclusion

By the conclusion of the day, this seminar changed how I think about modern AI architecture entirely. We’re finally out of the era of stateless, forgetful chatbots, and into the age where software can actually remember who we are, and grasp what’s occurring in our systems in real time.

Constructing a memory-aware AI was once thought of as a hard problem, accessible only to specific research labs. But as Someshwaran has showed with AWS AgentCore and the Strands SDK, the infrastructure is already in place to handle the heavy lifting for us. Combining conversational memory with the instant search capabilities of Elasticsearch allows us to create AI applications that give true compounding value over time.

If you want to roll up your sleeves and get your hands dirty with these architecture patterns, check out the resources over at elastic.co/search-labs. The tools are there – it’s our turn to construct!

About the Author

As an AWS Community Builder, I enjoy sharing the things I've learned through my own experiences and events, and I like to help others on their path. If you found this helpful or have any questions, don't hesitate to get in touch! 🚀

🔗 Connect with me on LinkedIn

References

Event: AWS Community Day Bengaluru

Speaker: Someshwaran Mohan Kumar

Topic: Building Scalable Context Layers for AI Agents with Elastic and AWS AgentCore

Date: July 11, 2026

Also Published On

AWS Builder Center

Hashnode

Gemma 4 on Amazon SageMaker: QAT Weights Decode 2.05x Faster Than bf16 on One L4

xbill — Fri, 25 Sep 2026 21:55:07 +0000

This article gives a short background on Amazon SageMaker real-time endpoints, then measures Gemma 4 E2B's quantization-aware trained (QAT) checkpoint against the full-size bf16 release on the same NVIDIA L4 endpoint. A suite of Python MCP tools is built to simplify management of the vLLM hosted deployment.

https://github.com/xbill9/sagemaker-gemma


Models	`google/gemma-4-E2B-it` (bf16) and `google/gemma-4-E2B-it-qat-w4a16-ct` (QAT, 4-bit weights)
Hardware	SageMaker `ml.g6.xlarge`, 1x NVIDIA L4, 24 GB
Region	`us-east-2`
Software	AWS vLLM SageMaker container, vLLM 0.30.0
Result	QAT decodes at 105.1 tok/s against 51.3, and serves 1077.25 tok/s at 16 parallel requests against 619.1, with the same score on 40 checked questions

SageMaker Hosting in Five Minutes

SageMaker real-time inference is three objects, created in order:

Object	What it holds
Model	A container image, its environment variables, and an IAM role
Endpoint config	Which model runs on which instance types, and how many instances
Endpoint	The running HTTPS service, billed per instance-hour while it exists

Requests go through aws sagemaker-runtime invoke-endpoint, signed with your AWS credentials. SageMaker health-checks the container, routes traffic to it and writes its log to CloudWatch under /aws/sagemaker/Endpoints/<name>.

The container here is the vLLM build AWS publishes for SageMaker. It reads vLLM settings from SM_VLLM_ environment variables, downloads the model from Hugging Face at start-up, and accepts OpenAI-style chat bodies. Switching checkpoints is one variable: SM_VLLM_MODEL.

SageMaker JumpStart also lists Gemma 4 as ready-made packages:

aws sagemaker list-hub-contents --hub-name SageMakerPublicHub --hub-content-type Model \
  --query "HubContentSummaries[].HubContentName" --output text | tr '\t' '\n' | grep gemma-4

huggingface-llm-gemma-4-31b-it-nvfp4
huggingface-vlm-gemma-4-12b-it
huggingface-vlm-gemma-4-26b-a4b-it
huggingface-vlm-gemma-4-31b-it
huggingface-vlm-gemma-4-31b-it-fp8-block
huggingface-vlm-gemma-4-e2b-instruct
huggingface-vlm-gemma-4-e4b-it

None of the seven is a Google QAT checkpoint, so this comparison uses the vLLM container with a Hugging Face model ID.

Three account limits shape every deployment:

Quota is per instance type, per region, counted in instances. This account holds 1 for each single-L4 type in us-east-1, us-east-2 and us-west-2.
Capacity is separate. us-east-1 and us-west-2 each left an L4 request in Creating for about 30 minutes and then returned InsufficientInstanceCapacity. us-east-2 placed one within minutes every time it was asked.
A fallback list (InstancePools in the endpoint config) holds quota for every type in it, so two endpoints cannot share one region's single-L4 quota.

With one L4 per region to work with, the two checkpoints ran one after the other on the same instance type in the same region.

Where Do I Start?

The deployment itself, from quota check to teardown with the aws CLI and the MCP server, is the first article in this pair: https://dev.to/aws-builders/gemma-4-on-an-amazon-sagemaker-endpoint-aws-cli-nvidia-l4-and-an-mcp-server-2c9d

This article starts from a working endpoint and changes one thing: the checkpoint.

At This Point You Should Have…

The repository above, with aws login done and mcp 2.x installed
A SageMaker quota of at least 1 for ml.g6.xlarge in a region with L4 capacity
The full-size endpoint from the first article deployed as gemma-4-e2b

What QAT Changes

Google trains the QAT checkpoint with 4-bit weights in the loop, then exports it in the compressed-tensors format vLLM reads natively. The -w4a16-ct suffix means 4-bit weights, 16-bit activations. Google publishes the same model in four QAT forms; only this one loads in vLLM:

Checkpoint	vLLM
`-qat-w4a16-ct`	🟢 loads, 4-bit weights
`-qat-q4_0-unquantized`	⚠️ stored at 16-bit, no memory saving
`-qat-q4_0-gguf`	❌ GGUF, for llama.cpp and Ollama
`-qat-mobile-*`	❌ on-device formats

Deploying it is the first article's Steps 4 to 6 with two variables changed:

export NAME=gemma-4-e2b-qat
export MODEL_ID=google/gemma-4-E2B-it-qat-w4a16-ct

vLLM names the format when it starts:

quantization=compressed-tensors

What the Engine Allocates

The container log records the weights, the load time and the KV cache vLLM builds from what is left of the L4's memory:

Model loading took 9.75 GiB memory and 82.751806 seconds
GPU KV cache size: 723,484 tokens, Maximum concurrency for 8,192 tokens per request: 88.32x

Model loading took 8.01 GiB memory and 66.190265 seconds
GPU KV cache size: 867,999 tokens, Maximum concurrency for 8,192 tokens per request: 105.96x

	bf16	QAT	QAT / bf16
Weights (GiB)	9.75	8.01	0.82
KV cache (tokens)	723,484	867,999	1.2
Weight load (s)	82.75	66.19
Create to `InService` (min)	9.9	10.1

The 4-bit weights save 18% of GPU memory. The checkpoint's own header shows why: only the transformer body is 4-bit.

Part of the QAT file	GB	Share	Stored as
Per-layer embedding	4.698	56.5%	BF16
Vocabulary embedding	1.611	19.4%	BF16
Transformer body	1.056	12.7%	packed 4-bit
Audio tower	0.614	7.4%	BF16
Vision tower	0.337	4.1%	BF16

The memory the smaller weights free goes to the KV cache.

How the Measurement Works

compare.py runs the same three measurements against each endpoint, at temperature 0, through the same aws sagemaker-runtime invoke-endpoint call:

Decode. Fixed-length replies of 16 and 512 tokens (ignore_eos), five of each. The decode rate is (512 − 16) / (median time at 512 − median time at 16), which cancels the aws CLI start-up and the network round trip.
Parallel requests. 1, 4 and 16 requests at once, 256 tokens each, two batches per level. Throughput is total output tokens divided by the batch's wall time.
Answers. 40 fixed questions with exact answers: 15 two-digit multiplications, 15 three-number sums, 10 capitals. Scored by regular expression.

python3 compare.py measure docs/runs/2026-09-25-qat-vs-bf16 gemma-4-e2b-qat@us-east-2

wrote docs/runs/2026-09-25-qat-vs-bf16/measure-gemma-4-e2b-qat.json
{
  "decode_tokens_per_second": 105.1,
  "per_call_fixed_cost_seconds": 0.562,
  "load_tokens_per_second": {
    "1": 85.35,
    "4": 328.7,
    "16": 1077.25
  },
  "quality": "37/40"
}

combine computes every ratio from the two result files:

python3 compare.py combine compare.json measure-gemma-4-e2b.json measure-gemma-4-e2b-qat.json

                               gemma-4-e2b     gemma-4-e2b-qat  ratio
weights_gib                           9.75                8.01  0.82
kv_cache_tokens                     723484              867999  1.2
load_seconds                     82.751806           66.190265  
decode_tokens_per_second              51.3               105.1  2.05
load_c1_tokens_per_second             45.6               85.35  1.87
load_c4_tokens_per_second            171.6               328.7  1.92
load_c16_tokens_per_second           619.1             1077.25  1.74
quality_correct                         37                  37  
identical answers: 35/40

Decode Speed

	bf16	QAT
Decode (tokens/s)	51.3	🥇 105.1
512-token reply, fastest (s)	10.491	🥇 5.392
512-token reply, slowest (s)	10.661	🥇 5.526
Per-call client cost (s)	0.633	0.562

QAT decodes 2.05x faster. The slowest QAT reply finished in about half the time of the fastest bf16 one, so the gap is far larger than the spread between repeats.

Each decode step reads every transformer layer's weights from GPU memory, so decode speed follows how many bytes those layers take. Those are the layers the QAT export stores at 4 bits. The embeddings are looked up one row per token, so their 16-bit size costs memory and little time.

Parallel Requests

Requests at once	bf16 tok/s	QAT tok/s	QAT / bf16
1	45.6	🥇 85.35	1.87
4	171.6	🥇 328.7	1.92
16	619.1	🥇 1077.25	1.74

QAT leads at every level. The ratio narrows at 16, where a batch shares each weight read across more requests and the per-token saving counts for less. The single-request figures sit below the decode rate because each call also pays the aws CLI start-up.

Does It Still Answer Correctly?

Questions	bf16	QAT
Multiplication (15)	15	15
a + b − c (15)	12	12
Capitals (10)	10	10
Total (40)	37	37

35 of the 40 answers are identical character for character. The other five are all sums, and each model misses three of them:

Question	Expected	bf16	QAT
876 + 608 − 558	926	❌ 1026	✅ 926
257 + 388 − 290	355	✅ 355	❌ 655
782 + 571 − 534	819	❌ 829	✅ 819
470 + 238 − 983	−275	✅ −275	❌ −285
150 + 937 − 139	948	❌ 918	❌ 1010

The two models make the same number of mistakes on different questions. Forty questions are enough to show a large loss and too few to measure a small one.

Re-Measured in A-B-A Order

Each endpoint runs on its own physical instance, and the two ran 15 minutes apart. To check that the gap belongs to the checkpoint, the bf16 endpoint was deployed a second time after QAT and measured again:

Run	Start (UTC)	Decode tok/s	16 at once tok/s	Correct
bf16	18:36	51.3	619.1	37
QAT	18:50	105.1	1077.25	37
bf16 again	19:10	51.5	625.15	37

The two bf16 runs agree within 1.21% on every speed figure and give identical answers to all 40 questions. The second bf16 instance loaded its weights in 82.09 s against 82.75 s the first time.

Compare to Other Deployments

The same bf16 model on the same GPU has been measured on two other platforms in this series, with vllm bench serve at 128 output tokens:

Platform	1 request tok/s
Cloud Run, NVIDIA L4	49.63
EC2 `g6.2xlarge`, NVIDIA L4	46.09
SageMaker `ml.g6.xlarge`, NVIDIA L4	45.6

The three sit within 9% of each other. SageMaker's figure includes the aws CLI start-up in every call; its decode rate with that removed is 51.3.

On a Tesla T4, QAT decoded 1.79x faster than bf16 for the same model. On the L4 the ratio is 2.05x.

The methods differ: the other runs used vllm bench serve with random prompts, other vLLM versions and their own hosts. Read the rows as a shape.

And Price/Performance?

ml.g6.xlarge in us-east-2 is $1.1267 an hour on demand:

aws pricing get-products --region us-east-1 --service-code AmazonSageMaker \
  --filters Type=TERM_MATCH,Field=instanceName,Value=ml.g6.xlarge Type=TERM_MATCH,Field=regionCode,Value=us-east-2

ml.g6.xlarge USE2-Host:ml.g6.xlarge 1.1267000000 Hrs | $1.1267 per Hosting ml.g6.xlarge hour in US East (Ohio)

The same hourly price buys twice the tokens. Per million output tokens (arithmetic):

Requests at once	bf16 $/M	QAT $/M
1	6.86	🥇 3.67
4	1.82	🥇 0.95
16	0.51	🥇 0.29

The endpoint bills while it exists, busy or idle, so these figures hold only while it is kept busy.

So, Which One?

	bf16	QAT
Decode speed	51.3 tok/s	🥇 105.1 tok/s
16 requests at once	619.1 tok/s	🥇 1077.25 tok/s
GPU memory for weights	9.75 GiB	🥇 8.01 GiB
Checked answers	37 / 40	37 / 40
Cost per million tokens at 16	$0.51	🥇 $0.29

On an L4 SageMaker endpoint, the QAT checkpoint is the default choice for Gemma 4 E2B: the same instance, one changed environment variable, about twice the tokens per dollar, and no measured change in answers. Keep bf16 as the reference when a task's accuracy needs a larger evaluation than 40 questions.

What Stops the Meter

delete_endpoint removes the endpoint, its config and its model:

🗑️ `gemma-4-e2b-qat` in `us-east-2`
- endpoint: deleted
- endpoint-config: deleted
- model: deleted

A Creating endpoint refuses deletion, so a deploy runs until it reaches InService or Failed before it can be removed.

Summary

The goal of this article was to measure what Gemma 4 E2B's QAT checkpoint changes on a SageMaker L4 endpoint. The key to the solution was changing only SM_VLLM_MODEL between two deployments on the same instance type, and measuring decode speed with the per-call client cost removed. The measured results were:

🟢 QAT decodes at 105.1 tok/s against bf16's 51.3, 2.05x
🟢 At 16 requests at once QAT serves 1077.25 tok/s against 619.1, 1.74x
🟢 Both score 37 of 40 on checked questions, with 35 identical answers
🟢 A second bf16 deployment after QAT reproduced the first within 1.21%
⚠️ The 4-bit export saves 18% of weight memory, because the embeddings stay at 16-bit
⚠️ Getting an L4 took three regions: us-east-1 and us-west-2 returned InsufficientInstanceCapacity after about 30 minutes each

Scope: one account, SageMaker ml.g6.xlarge with one NVIDIA L4 in us-east-2, vLLM 0.30.0 from the AWS container, three deployments on 2026-09-25 each on its own instance, measured in bf16, QAT, bf16 order. Prompts were short; long-prompt behaviour was not measured. Decode used five replies per length, parallel throughput two batches per level, and quality 40 questions at temperature 0. Every request went through the aws CLI from one client machine.

The strategy for using MCP for SageMaker deployment and benchmarking was validated with an incremental step by step approach.

References

Repository: https://github.com/xbill9/sagemaker-gemma
Part one, deploying Gemma 4 to SageMaker: https://dev.to/aws-builders/gemma-4-on-an-amazon-sagemaker-endpoint-aws-cli-nvidia-l4-and-an-mcp-server-2c9d
Gemma 4 E2B QAT w4a16: https://huggingface.co/google/gemma-4-E2B-it-qat-w4a16-ct
Gemma 4 E2B: https://huggingface.co/google/gemma-4-E2B-it
Gemma 4 on a Tesla T4, QAT vs bf16: https://dev.to/gde/gemma-4-on-a-tesla-t4-qat-weights-decode-179x-faster-than-bf16-2fi4
2B Gemma 4 on Cloud Run with an NVIDIA L4: https://dev.to/gde/2b-gemma-4-deployment-with-cloud-run-nvidia-l4-mcp-sdk-2x-and-claude-code-4ml3
SageMaker real-time inference: https://docs.aws.amazon.com/sagemaker/latest/dg/realtime-endpoints.html
AWS Deep Learning Containers: https://github.com/aws/deep-learning-containers
compressed-tensors: https://github.com/neuralmagic/compressed-tensors

Gemma 4 on an Amazon SageMaker Endpoint: AWS CLI, NVIDIA L4, and an MCP Server

xbill — Fri, 25 Sep 2026 21:50:14 +0000

This article provides a step by step deployment guide for Gemma 4 E2B to an Amazon SageMaker hosted GPU enabled system. A suite of Python MCP tools is built to simplify management of the vLLM hosted deployment with Claude Code.

https://github.com/xbill9/sagemaker-gemma

What is this project trying to Do?

This project serves Gemma 4 E2B from a SageMaker real-time endpoint on one NVIDIA L4 GPU, using the vLLM container AWS publishes for SageMaker. Every AWS call is a plain aws CLI command, so each step can be run by hand or by the MCP server.

A SageMaker real-time endpoint is a managed HTTPS inference server. SageMaker places the container on a GPU instance, health-checks it, routes requests to it and writes its logs to CloudWatch. There is no instance to patch, no security group to open and no load balancer to build.

Where do I start?

The strategy for starting MCP development for model management is a incremental step by step approach.

First, the basic development environment is setup with the required system variables and a working Claude Code configuration.

Then, the Python MCP server is brought up over stdio and validated with Claude Code in the local environment. The deployment follows as eight steps, each shown as the raw aws command and the MCP tool that runs it.

At This Point You Should Have…

An AWS account and the AWS CLI v2, signed in with aws login
A SageMaker endpoint quota of at least 1 for a single-L4 instance type (ml.g6.xlarge, ml.g6.2xlarge or ml.g6.4xlarge)
Python 3.11 or newer with mcp 2.x
Claude Code or Gemini CLI installed and working
jq for reading JSON replies

Setup the Basic Environment

Clone the repository and install the one requirement into the system Python:

git clone https://github.com/xbill9/sagemaker-gemma
cd sagemaker-gemma
python3 -m pip install -r requirements.txt
cp .env.example .env

.env is gitignored. It holds the settings every tool reads:

cat .env

# Copy to .env (gitignored) and edit. sm.py reads it, so the MCP server, CLI and Makefile all see it.
AWS_REGION=us-east-2
MODEL_ID=google/gemma-4-E2B-it
INSTANCE_TYPE=ml.g6.xlarge
ENDPOINT_NAME=gemma-4-e2b
ROLE_NAME=sagemaker-gemma-execution-role
MAX_MODEL_LEN=8192
# Leave empty to use the newest SageMaker vLLM image in the region.
IMAGE_URI=
# Fallback instance types, highest priority first (same GPU keeps runs comparable).
INSTANCE_POOLS=ml.g6.xlarge,ml.g6.2xlarge,ml.g6.4xlarge

Gemma 4 is Apache-2.0 and ungated on Hugging Face, so no Hugging Face token is needed.

Model Management Tool with MCP Stdio Transport

The simplest MCP transport is stdio: the client launches the server as a local process and talks to it over stdin and stdout. In this project Claude Code is the MCP client. The server is one file, server.py, on the MCP Python SDK 2.x:

mcp = MCPServer(RIG_NAME)
READ_ONLY = ToolAnnotations(readOnlyHint=True, idempotentHint=True)
WRITE = ToolAnnotations(destructiveHint=False)
DESTRUCTIVE = ToolAnnotations(destructiveHint=True)

Every tool carries one of the three annotations, so a client can tell a status check from a deploy from a delete.

The tools do no AWS work themselves. They call sm.py, which runs each request as an aws CLI subprocess:

def aws(*args: str, region: str | None = REGION, parse: bool = True) -> Any:
    """Run `aws <args> --output json` and return the parsed result."""
    cmd = ["aws", *args, "--output", "json"]

The CLI renews an aws login session on its own, so a server that stays up for hours keeps working credentials. sm.py also drops any AWS_SESSION_TOKEN the server inherits from its parent process, because a static token expires inside a long-running server and outranks the login session.

Running the Python Code

The project can be linted:

make lint

All checks passed!
16 files already formatted

and tested:

make test

----------------------------------------------------------------------
Ran 16 tests in 0.010s

OK

The tests replace the aws subprocess with a fake, so they run offline with no credentials. One of them compares the registered tool set and annotations against a fixed list: a tool that failed to register, or a delete that lost its destructive flag, fails the suite. ✅

Test the Protocol by Hand

A client speaks JSON-RPC over stdio. Hold stdin open with sleep, or the server sees end-of-input and exits before it answers:

{ printf '%s\n' \
  '{"jsonrpc":"2.0","id":1,"method":"initialize","params":{"protocolVersion":"2025-06-18","capabilities":{},"clientInfo":{"name":"probe","version":"0"}}}' \
  '{"jsonrpc":"2.0","method":"notifications/initialized"}' \
  '{"jsonrpc":"2.0","id":2,"method":"tools/list","params":{}}'; sleep 3; } \
  | python3 server.py 2>/dev/null

Summarised:

1 sagemaker-gemma
2 ['check_quotas', 'delete_endpoint', 'deploy_endpoint', 'find_vllm_image', 'get_deployment_config', 'get_endpoint_logs', 'get_endpoint_status', 'get_help', 'list_endpoints', 'query_model', 'verify_model_health']

🟢 The server answers the handshake and lists 11 tools.

Claude Code .mcp.json

Claude Code reads .mcp.json in the project directory and launches the server with the system python3:

{
  "mcpServers": {
    "sagemaker-gemma": {
      "command": "python3",
      "args": ["/home/xbill/sagemaker-gemma/server.py"]
    }
  }
}

Gemini CLI reads the same entry from .gemini/settings.json.

Validation with Claude Code

claude mcp get sagemaker-gemma

sagemaker-gemma:
  Scope: Project config (shared via .mcp.json)
  Status: ✔ Connected

From inside Claude Code, get_help returns the resolved settings and the order of work:

### sagemaker-gemma

Serving `google/gemma-4-E2B-it` with vLLM on a **SageMaker real-time endpoint**, managed
entirely through the aws CLI.

| Setting | Value |
| --- | --- |
| Region | `us-east-2` |
| Endpoint | `gemma-4-e2b` |
| Instance types (priority order) | `ml.g6.xlarge`, `ml.g6.2xlarge`, `ml.g6.4xlarge` |
| Max model length | `8192` |
| Image | `newest SageMaker vLLM image (find_vllm_image)` |
| Execution role | `sagemaker-gemma-execution-role` |

**Order of work:** check_quotas → deploy_endpoint → get_endpoint_status until
`InService` (about 10 minutes once an instance is placed) → verify_model_health
→ query_model → delete_endpoint.

The steps below follow that order. Each one shows the raw CLI command; deploy_endpoint runs Steps 2 to 6 in one call.

export AWS_REGION=us-east-2 AWS_PAGER=
export NAME=gemma-4-e2b
export MODEL_ID=google/gemma-4-E2B-it

Step 1 — Check the Endpoint Quota

SageMaker endpoint quotas are per instance type, per region, and count instances.

aws service-quotas list-service-quotas --service-code sagemaker \
  --query "Quotas[?QuotaName=='ml.g6.xlarge for endpoint usage'].Value"

[
    1.0
]

The check_quotas tool reads the same quota for each fallback type across the US regions:

| Instance | us-east-1 | us-east-2 | us-west-1 | us-west-2 |
| --- | ---: | ---: | ---: | ---: |
| `ml.g6.xlarge` | 1 | 1 | - | 1 |
| `ml.g6.2xlarge` | 1 | 1 | - | 1 |
| `ml.g6.4xlarge` | 1 | 1 | - | 1 |

Regions with quota for at least one of these types: us-east-1, us-east-2, us-west-2.

A - means the type is not offered in that region. A 0 means a quota increase request first.

🔎 Tip: One Quota Call per Region

list-service-quotas pages through every SageMaker quota in the region, and the Service Quotas API is rate limited per account. Twelve calls at once, one per type per region, came back as:

An error occurred (TooManyRequestsException) when calling the ListServiceQuotas operation (reached max retries: 2)

check_quotas makes one call per region, one region at a time, and filters the types from that one result. sm.py also sets AWS_RETRY_MODE=adaptive so the CLI backs off and retries.

Step 2 — Find the vLLM Container

The AWS vLLM repository holds pinned release tags such as 0.30.0-gpu-py312-cu130-ubuntu24.04-sagemaker-v1.1, floating aliases such as 0.30-gpu-py312, and -soci index tags. Older release lines receive patch rebuilds, so the most recent push can carry an older vLLM. Sort the pinned tags by version:

TAG=$(aws ecr describe-images --registry-id 763104351884 --repository-name vllm \
  --query "imageDetails[].imageTags[]" --output text | tr '\t' '\n' \
  | grep -E -- '^[0-9]+\.[0-9]+\.[0-9]+-.*-sagemaker-v[0-9]+\.[0-9]+$' | sort -V | tail -1)
export IMAGE=763104351884.dkr.ecr.$AWS_REGION.amazonaws.com/vllm:$TAG
echo $IMAGE

763104351884.dkr.ecr.us-east-2.amazonaws.com/vllm:0.30.0-gpu-py312-cu130-ubuntu24.04-sagemaker-v1.1

The find_vllm_image tool applies the same rule. The image runs vLLM 0.30.0.

Step 3 — Create the Execution Role

SageMaker assumes this role to pull the image and write logs. It is created once per account.

aws iam create-role --role-name sagemaker-gemma-execution-role \
  --assume-role-policy-document '{"Version":"2012-10-17","Statement":[{"Effect":"Allow","Principal":{"Service":"sagemaker.amazonaws.com"},"Action":"sts:AssumeRole"}]}'
aws iam attach-role-policy --role-name sagemaker-gemma-execution-role \
  --policy-arn arn:aws:iam::aws:policy/AmazonSageMakerFullAccess
export ROLE=$(aws iam get-role --role-name sagemaker-gemma-execution-role --query Role.Arn --output text)

On a second run create-role reports EntityAlreadyExists; get-role still sets ROLE. deploy_endpoint creates the role only when it is missing.

Step 4 — Create the Model

A SageMaker model pairs an image with its settings. The container turns each SM_VLLM_ variable into the matching vLLM flag: SM_VLLM_MAX_MODEL_LEN becomes --max-model-len.

aws sagemaker create-model --model-name $NAME --execution-role-arn $ROLE \
  --primary-container "{\"Image\":\"$IMAGE\",\"Environment\":{
    \"SM_VLLM_MODEL\":\"$MODEL_ID\",
    \"SM_VLLM_MAX_MODEL_LEN\":\"8192\",
    \"SM_VLLM_GPU_MEMORY_UTILIZATION\":\"0.9\"}}"

{
    "ModelArn": "arn:aws:sagemaker:us-east-2:<account-id>:model/gemma-4-e2b"
}

Step 5 — Create the Endpoint Config With Fallback Instance Types

The endpoint config says where the model runs. InstancePools lists up to five instance types in priority order, and SageMaker places the first one with a free instance. All three types below carry one L4, so the model sees the same GPU whichever is placed.

aws sagemaker create-endpoint-config --endpoint-config-name $NAME \
  --production-variants "[{\"VariantName\":\"AllTraffic\",\"ModelName\":\"$NAME\",
    \"InitialInstanceCount\":1,
    \"InstancePools\":[{\"InstanceType\":\"ml.g6.xlarge\",\"Priority\":1},
                       {\"InstanceType\":\"ml.g6.2xlarge\",\"Priority\":2},
                       {\"InstanceType\":\"ml.g6.4xlarge\",\"Priority\":3}],
    \"ContainerStartupHealthCheckTimeoutInSeconds\":1800,
    \"ModelDataDownloadTimeoutInSeconds\":1800}]"

{
    "EndpointConfigArn": "arn:aws:sagemaker:us-east-2:<account-id>:endpoint-config/gemma-4-e2b"
}

The two 1800-second timeouts give the container time to download the weights and compile before SageMaker's health check gives up.

🔎 Tip: A Fallback List Holds Quota for Every Type in It

With one Gemma endpoint running on ml.g6.xlarge from the three-type list above, a second endpoint asking for ml.g6.2xlarge in the same region was refused:

ResourceLimitExceeded: The account-level service limit 'ml.g6.2xlarge for endpoint usage' is 1 Instances, with current utilization of 1 Instances and a request delta of 1 Instances.

With a quota of 1 per type, a second endpoint goes in another region, or uses types outside the first endpoint's list.

Step 6 — Create the Endpoint

aws sagemaker create-endpoint --endpoint-name $NAME --endpoint-config-name $NAME

{
    "EndpointArn": "arn:aws:sagemaker:us-east-2:<account-id>:endpoint/gemma-4-e2b"
}

Billing starts when an instance is placed. aws sagemaker wait endpoint-in-service --endpoint-name $NAME blocks until it is ready; get_endpoint_status reports the same state and the instance type that was placed:

✅ `gemma-4-e2b` in `us-east-2`: **InService**
- Instance: `ml.g6.xlarge`

The container log is in CloudWatch, and get_endpoint_logs tails it:

aws logs tail /aws/sagemaker/Endpoints/$NAME --follow

Start-up on the L4, in minutes after create-endpoint:

Phase	Minutes
Weights loaded, 9.75 GiB in 82.75 s	7.4
KV cache sized, 723,484 tokens	9.2
`InService`	9.9

🔎 Tip: A Creating Endpoint Cannot Be Deleted

create-endpoint cannot be undone until the endpoint settles. A delete-endpoint sent while it is Creating is refused:

aws: [ERROR]: An error occurred (ValidationException) when calling the DeleteEndpoint operation: Cannot update in-progress endpoint "arn:aws:sagemaker:us-east-2:<account-id>:endpoint/gemma-4-e2b".

The endpoint finishes starting, bills from the moment its instance is placed, and can be deleted once it reaches InService or Failed. Check the model ID and instance types before Step 6.

🔎 Tip: Capacity Is Separate From Quota

A quota of 1 lets you request one instance; the region still has to have one free. In us-east-1, two requests for L4 instances each stayed in Creating for about 30 minutes and then failed:

Unable to provision requested ML compute capacity due to InsufficientInstanceCapacity error. Please retry using a different ML instance type or after some time.

The same request in us-east-2 placed an instance at once. While SageMaker waits for capacity, no container starts and the CloudWatch log group never appears, which is how get_endpoint_logs tells a capacity wait from a slow model load. No charge accrues during the wait. When it fails, repeat Steps 4 to 6 in another region where Step 1 shows the same quota.

Step 7 — Cross Check the Deployed Model

The vLLM container accepts an OpenAI chat body on invoke-endpoint:

echo '{"messages":[{"role":"user","content":"Why is the sky blue?"}],"max_tokens":256}' > req.json
aws sagemaker-runtime invoke-endpoint --endpoint-name $NAME \
  --content-type application/json --body fileb://req.json out.json
jq -r '.choices[0].message.content' out.json | head -1
jq -c '{model,usage}' out.json

{
    "ContentType": "application/json",
    "InvokedProductionVariant": "AllTraffic"
}
The sky is blue due to a phenomenon called **Rayleigh scattering**. This process is caused by how sunlight interacts with the Earth's atmosphere.
{"model":"google/gemma-4-E2B-it","usage":{"prompt_tokens":15,"total_tokens":271,"completion_tokens":256,"prompt_tokens_details":null,"completion_tokens_details":null}}

verify_model_health sends one short request and checks for a reply:

✅ model=`google/gemma-4-E2B-it` tokens=2 wall=0.749s reply='ok'

Step 8 — Teardown

The endpoint bills by the hour until it is deleted. Deleting the config and the model as well leaves nothing behind. delete_endpoint runs all three:

aws sagemaker delete-endpoint --endpoint-name $NAME
aws sagemaker delete-endpoint-config --endpoint-config-name $NAME
aws sagemaker delete-model --model-name $NAME

Each command prints nothing and exits 0. list_endpoints confirms the account is clear:

### 0 endpoint(s) matching `gemma`

Summary

The goal of this article was to deploy Gemma 4 E2B to an Amazon SageMaker real-time endpoint with the AWS CLI and manage it from an MCP server. The key to the solution was the AWS vLLM SageMaker container, which turns the deployment into three create- calls and makes the endpoint answer OpenAI-style chat requests. The deployment results were:

🟢 Eight CLI steps take an account from quota check to a serving endpoint and back to nothing
🟢 The endpoint reached InService 9.9 minutes after create-endpoint, with the weights using 9.75 GiB of the L4's 24 GB
🟢 The MCP server runs every step from Claude Code, marks each tool read-only, write or destructive, and passes its tests offline
🟢 InstancePools gives one endpoint config several fallback instance types
⚠️ L4 capacity varied by region: us-east-1 refused twice, about 30 minutes each, while us-east-2 placed an instance at once
⚠️ A fallback list holds quota for every type in it

Scope: one account, ml.g6.xlarge with one NVIDIA L4, vLLM 0.30.0 from the AWS container, google/gemma-4-E2B-it at full precision, deployed in us-east-2 on 2026-09-25. Start-up times are from a single deployment.

The strategy for using MCP for SageMaker deployment was validated with an incremental step by step approach.

References

Repository: https://github.com/xbill9/sagemaker-gemma
Gemma 4 E2B on Hugging Face: https://huggingface.co/google/gemma-4-E2B-it
AWS Deep Learning Containers: https://github.com/aws/deep-learning-containers
SageMaker real-time inference: https://docs.aws.amazon.com/sagemaker/latest/dg/realtime-endpoints.html
CreateEndpointConfig API: https://docs.aws.amazon.com/sagemaker/latest/APIReference/API_CreateEndpointConfig.html
MCP Python SDK: https://github.com/modelcontextprotocol/python-sdk
vLLM: https://docs.vllm.ai/

Two Iceberg Clients, One Protocol: Where the Time Goes

xbill — Fri, 25 Sep 2026 16:40:41 +0000

This article provides a step by step comparison of the Rust and Python clients for Apache Iceberg REST catalogs. It times both clients on the same operations against the same tables, then breaks one request down to see where the time goes.

https://github.com/xbill9/lakehouse-iceberg-2026

What is this project trying to Do?

Two Apache clients talk to the same catalogs:

iceberg-catalog-rest 0.10.1, the Rust client, built in release mode
pyiceberg 0.12.0, the Python client, on Python 3.14.7

They run against three catalogs:

Apache Polaris 1.7.0 in Docker on the same machine, so there is no network time
Google BigLake and Microsoft OneLake over the internet

Three things are measured: how long each call takes, how long each client takes to start, and where the time in one call is spent.

The benchmark measures speed only. It does not check that answers are correct, and a client that lacks an operation you need is the wrong choice however fast it is.

Why Not Just Count Features?

The Rust client supports 13 of the 25 catalog endpoints tested, and pyiceberg supports 21. Anyone can count that from the two repositories, and it only holds for these two versions.

How long a call takes is in neither repository, so that is what this article measures.

Where do I start?

The strategy for comparing the two clients is an incremental step by step approach.

First, both clients are timed against a local catalog, where only the client's own time is measured. Then one request is broken down to find where the time goes. The same benchmark then runs against two managed catalogs, the two clients' requests are compared, and startup time is measured last.

At This Point You Should Have…

Rust 1.94 or newer, and a release build — this run used rustc 1.98.1
Python 3.10+ with pyiceberg 0.12.0
Docker, for the local Polaris catalog
A quiet machine — this one is a 16-core Linux host
Optional: a managed Iceberg REST catalog, for the internet runs

Step 1 — Build Both Clients and Start Polaris

$ git clone https://github.com/xbill9/lakehouse-iceberg-2026
$ cd lakehouse-iceberg-2026/iceberg-conformance && ./polaris-up.sh
$ export POLARIS_CLIENT_ID=root POLARIS_CLIENT_SECRET=s3cr3t
$ cd ../iceberg-rust-client && cargo build --release

The benchmark will not save results from a debug build.

Step 2 — Time Both Clients Locally

To keep the comparison fair:

Each client runs in its own new process. The benchmark script is Python, so running pyiceberg inside it would give Python a head start.
Startup is timed separately. Each process connects and warms up before any call is timed.
The clients take turns. They alternate going first, so a machine that slows down slows both.
Every timing is saved, and the medians are computed from the saved files.

$ python3 bench.py --rounds 4 --iters 15
apache-polaris     6 ops, 2 clients, 4 rounds x 15 iters = 720 samples
$ python3 bench_report.py
wrote evidence/bench-comparison.txt   8 run(s) across 3 catalog(s)

Six read operations both clients support, median time in microseconds, lowest to highest across four runs on two days:

op	rust	pyiceberg	python ÷ rust
list_namespaces	301.3–768.1	883.2–1479.4	1.90x–4.29x
load_namespace	312.2–699.3	784.7–1451.2	2.08x–3.52x
head_namespace	328.4–584.8	1048.6–1362.7	2.33x–3.19x
list_tables	327.0–552.1	1003.2–1551.3	2.56x–3.53x
load_table	4634.1–5764.4	6405.6–7344.8	1.23x–1.38x
head_table	4266.1–5298.0	5167.9–6155.4	1.14x–1.21x

Rust is 1.90x to 4.29x faster on the four small calls, and 1.14x to 1.38x on the two table calls, where the server itself does more work.

The numbers vary a lot between runs. The Rust time for list_namespaces ranged from 301.3 to 768.1 microseconds, a spread of 154.9%. That is why the table shows ranges.

With no network involved, these ratios are as large as the difference can get. Step 4 adds the internet.

Step 3 — Break One Request Down

The same list_namespaces request is timed in stages on one pyiceberg connection: the raw HTTP request, then parsing the JSON, then building Python objects, then the full client call. The same raw HTTP request is then timed from Rust.

$ python3 bench_breakdown.py --iters 120

Across seven runs on the local catalog, over two days:

The raw HTTP request is 90% to 94% of pyiceberg's call time.
The same HTTP request takes 269.8 to 479.7 microseconds from Rust's HTTP library, reqwest, and 864.3 to 1286.4 microseconds from Python's, requests.
Everything pyiceberg does after the HTTP request adds 57.3 to 134.7 microseconds.

So 85% to 90% of the difference between the two clients is the HTTP library. That percentage is calculated from the two ranges above, run by run.

🔎 Tip: Small Differences Are Noise Here

The breakdown tool prints, next to each stage, whether the time that stage added is bigger than the normal run-to-run noise. Only JSON parsing was. How the rest of pyiceberg's 57.3 to 134.7 microseconds splits up cannot be measured at this sample size.

One setting is worth knowing about. requests re-reads proxy settings from the environment on every call unless trust_env=False is set. Turning it off saved between 32.9 and 89.9 microseconds in each of five runs, but that saving was bigger than the noise in only one run.

Step 4 — Time Them Over the Internet

The same benchmark against BigLake and OneLake. Both clients use the same login token, created once before the run:

$ python3 bench.py --only google-lakehouse --rounds 4 --iters 15 --storage opendal
google-lakehouse   6 ops, 2 clients, 4 rounds x 15 iters = 720 samples
$ python3 bench.py --only microsoft-onelake --rounds 4 --iters 15 --storage opendal
microsoft-onelake  6 ops, 2 clients, 4 rounds x 15 iters = 720 samples

Two runs each, median time in milliseconds:

op	BigLake rust	BigLake py	ratio	OneLake rust	OneLake py	ratio
list_namespaces	102.4–103.1	92.2–106.7	0.90x–1.04x	36.0–36.4	35.9–36.0	0.99x–1.00x
load_namespace	199.4–201.6	199.1–200.9	0.99x–1.01x	44.6–45.1	45.3–46.3	1.02x–1.03x
head_namespace	198.4–200.4	199.2–200.0	0.99x–1.01x	44.8–45.9	45.3–46.6	0.99x–1.04x
list_tables	202.7–205.5	203.2–206.1	1.00x	45.0–46.3	45.4–45.6	0.98x–1.01x
load_table	296.4–297.1	281.9–284.3	0.95x–0.96x	36.3–39.0	160.7–167.7	4.12x–4.62x
head_table	201.7–205.0	199.9–201.4	0.98x–0.99x	53.2–54.4	53.4–53.8	0.98x–1.01x

Apart from load_table, every ratio is between 0.90x and 1.04x. The local difference is under a millisecond per call, and a round trip of 36 to 200 milliseconds hides it.

On BigLake, pyiceberg was 4% to 5% faster on load_table in both runs, for a reason this test did not identify.

🔎 Tip: BigLake Allows 75 Catalog Requests a Minute

BigLake limits catalog requests to 75 a minute per project, and answers faster bursts with HTTP 429. The benchmark discards any timing that did not return success, and bench_breakdown.py --pace-ms waits between requests. The limit is visible with:

$ gcloud alpha services quota list --service=biglake.googleapis.com \
    --consumer=projects/<project> \
    --format="table(metric,consumerQuotaLimits[0].unit,consumerQuotaLimits[0].quotaBuckets[0].effectiveLimit)"
biglake.googleapis.com/irc_catalog_requests          1/min/{project}       75
biglake.googleapis.com/irc_read_requests             1/min/{project}       600

Step 5 — Compare the Two Clients' Requests

OneLake's load_table took 4.12x to 4.62x longer through pyiceberg, while every other OneLake call was even. The two clients send different requests.

pyiceberg sends the header X-Iceberg-Access-Delegation: vended-credentials by default (catalog/rest/__init__.py:881). It asks the catalog to include a temporary storage credential with the table. The Rust client sends no such header.

bench_delegation.py times the same pyiceberg request with and without the header:

$ python3 bench_delegation.py --only microsoft-onelake --iters 40

row                                  p50       p90
GET, header sent                   146.6     235.5
GET, header removed                 38.1      41.3
whole load_table                   148.5     182.4
whole load_table, no header         36.6      42.4

With the header, OneLake adds a storage credential to the response and takes 108.5 ms longer. Without it, pyiceberg's load_table takes 36.6 ms, inside the Rust client's 36.3–39.0 ms. On Polaris and BigLake the header made no measurable difference.

So pyiceberg pays 108.5 to 111.9 ms and gets a credential for reading the table's files. The Rust client pays nothing and gets no credential. The companion article on the Rust client covers what happens when it then tries to read the files.

Step 6 — Measure Startup Time

Startup here means starting a new process, connecting, and answering each of the six operations once. Median of five starts per client per run:

catalog	rust	pyiceberg	python takes longer by
Polaris, local, 4 runs	16.4–26.0 ms	525.8–557.9 ms	507.2–535.1 ms
BigLake, 2 runs	1376.3–1410.3 ms	1911.8–2016.7 ms	535.5–606.4 ms
OneLake, 2 runs	955.9–996.2 ms	1723.5–1772.4 ms	767.6–776.2 ms

Over the internet both clients wait on the same round trips, so the ratio drops from 21.4x–33.6x locally to 1.4x on BigLake and 1.8x on OneLake. The extra half second stays. It matters for a CLI, a Lambda function or a short-lived agent, and hardly at all for a service that starts once.

Where the half second goes, median of five, across seven local runs:

step	time
start Python, `python3 -c pass`	9.0–11.2 ms
`import pyiceberg.catalog.rest`	a further 337.9–366.3 ms
connect: fetch the catalog config and log in	6.8–12.4 ms

Most of it is importing the library. Connecting to the catalog is the smallest part.

On OneLake the import took 338.9 and 341.4 ms, the same as locally, and fetching the config took 447.3 and 493.8 ms. Both clients fetch the config; only pyiceberg pays for the import.

How This Compares to Published Lambda Numbers

Cold Starts Are Dead measured AWS Lambda start times of 88.3 ms for Python 3.13 on arm64 (106.2 ms on x86_64) and 14.1 ms for Rust (17.0 ms), at 512 MB, with hello-world functions and no libraries.

The local Python startup here is about six times that: 525.8 to 557.9 ms against 88.3 ms. The difference is mostly the 337.9 to 366.3 ms spent importing pyiceberg.

Compare and Contrast

	🦀 iceberg-catalog-rest	🐍 pyiceberg
`list_namespaces`, local, µs	🥇 301.3–768.1	883.2–1479.4
Small calls over the internet	tie, 0.90x–1.04x	tie
HTTP library	🥇 `reqwest`	`requests`, 85%–90% of the difference
Extra startup time	🥇 —	507.2–776.2 ms
Of which, importing the library	—	337.9–366.3 ms
`load_table` on OneLake, ms	🥇 36.3–39.0, no credential	160.7–167.7, with a storage credential
Requests storage credentials	no	🥇 yes, by default
Endpoints supported	13 of 25	🥇 21 of 25

So, Which One?

For a long-running service, pick by features. Over the internet the per-call difference disappears, and startup happens once. Choose the client that supports the operations and login method you need.

For a CLI, a Lambda function or a short-lived agent, startup decides it. Python adds half a second or more per process on every catalog tested.

Summary

The goal of this article was to measure how fast the Rust and Python Iceberg REST clients are. The key to the solution was running both as separate processes against the same catalogs, timing startup separately, and breaking one request down to see where the time goes. The results were:

🟢 Locally, Rust is 1.90x to 4.29x faster on small calls, and 85% to 90% of that difference is the HTTP library
🟢 Over the internet the difference disappears: 0.90x to 1.04x on BigLake and OneLake
⚠️ Python starts 507.2 to 776.2 ms slower on every catalog, mostly from importing the library (337.9 to 366.3 ms locally)
⚠️ OneLake's 4x on load_table comes from pyiceberg requesting a storage credential by default, which the Rust client never does
❌ pyiceberg was 4% to 5% faster on BigLake's load_table, cause unknown

Scope: iceberg-catalog-rest 0.10.1 in a release build with rustc 1.98.1 and rustls 0.23.45, and pyiceberg 0.12.0 on Python 3.14.7 with requests 2.34.2 over OpenSSL 3.5.7, on one 16-core Linux host. Apache Polaris 1.7.0 in Docker on the same machine with local file storage: four benchmark runs on 2026-09-17 and 2026-09-18, and seven breakdown runs of 80 to 120 requests each. Google BigLake and Microsoft OneLake over the internet on 2026-09-18: two benchmark runs each. Each benchmark run is 720 timed calls across six read operations; startup is the median of five starts per client per run. No writes were timed. Managed catalogs do not report a version, and each internet run is one region at one point in time.

The strategy for comparing two Iceberg REST clients by speed was validated with an incremental step by step approach.

References

What One Rust Client Can Reach Across Seven Iceberg Catalogs

xbill — Fri, 25 Sep 2026 16:20:54 +0000

This article provides a step by step guide to pointing the Apache Rust client for Iceberg REST catalogs at seven catalogs and recording what works. A Python script runs a small Rust program against each catalog and saves every result.

https://github.com/xbill9/lakehouse-iceberg-2026

What is this project trying to Do?

An earlier article tested what seven Iceberg REST catalogs support. This one asks from the other side: if you write a lakehouse tool in Rust today, will it work with the catalog you already pay for, and what do you need to set up? The client is iceberg-catalog-rest 0.10.1 from the Apache Iceberg project, and the catalogs, tests and test table are the same as in that article:

Apache Polaris 1.7.0, running locally as the reference
Google BigLake and Microsoft OneLake, over the internet
AWS Glue and AWS S3 Tables, over the internet
Databricks Unity and Snowflake Horizon, not run here

The result is a short one. This client implements 13 of the 25 operations the earlier article tested, it logs in to five of the seven catalogs, and on every catalog it logs in to, all 13 answer. Two lines of Cargo.toml stand between it and those five. The other two are AWS, they require a signature this client cannot produce, and the Rust answer to that is two more catalog crates that do not implement quite the same operations.

What Does Each Result Mean?

The test suite is 33 checks covering 25 of the 35 operations in the Iceberg REST specification. Each check gets one of five results:

result	meaning
`OK`	the client sent the request and got an answer
`FAILED`	the client sent the request and got an error
`IMPLICIT`	the client sends this request on its own, and there is no way to call it directly
`NOT-EXPRESSIBLE`	the client has no method for this operation
`NOT-ISSUED`	the client supports it, but it is a write, and that run only reads

Where do I start?

The strategy for testing the client is an incremental step by step approach.

First, a local Polaris catalog is started and the client's source is read to list which operations it supports. The test program is then built and run against Polaris, reads first and writes after, then against the managed catalogs one login method at a time, and finally it tries to read a table's files.

At This Point You Should Have…

Rust 1.94 or newer and cargo — this run used rustc 1.98.1
Docker, for the local Polaris catalog
Python 3.10+ with pyyaml, for the test scripts
Optional: logins for any managed catalog — gcloud for BigLake, az for OneLake, the AWS CLI for Glue and S3 Tables

Step 1 — Start Polaris

Polaris runs locally with permissive settings. Anything that fails here is a problem in the test scripts, so it is fixed before any cloud catalog is tested:

$ git clone https://github.com/xbill9/lakehouse-iceberg-2026
$ cd lakehouse-iceberg-2026/iceberg-conformance
$ ./polaris-up.sh
$ export POLARIS_CLIENT_ID=root POLARIS_CLIENT_SECRET=s3cr3t

Step 2 — Build the Test Program

The versions are pinned exactly, so every result applies to one release:

iceberg = "=0.10.1"
iceberg-catalog-rest = "=0.10.1"
iceberg-storage-opendal = { version = "=0.10.1", features = ["opendal-gcs", "opendal-azdls"] }
reqwest = { version = "0.12", default-features = false, features = ["json", "rustls-tls"] }

The last two lines add a TLS backend and the cloud storage backends. What You Add Beyond the Client, near the end, says what happens without each of them.

$ cd ../iceberg-rust-client
$ cargo build --release
    Finished `release` profile [optimized] target(s) in 1m 41s

Step 3 — List What the Client Supports

Each of the 33 checks is matched to a method in the client's published source code, with the file and line number, and check_refs.py confirms all 25 of those lines still sit inside the function they name. Counting each endpoint once, since five checks use the same update_table endpoint:

13 of 25 distinct endpoints are expressible through this client.
Counted per probe the figure is 19 of 33, which is the same fact
weighted by how many probes paper 1 happened to point at one endpoint.

The 12 endpoints it cannot reach split two ways:

Missing — 11 endpoints. All seven view operations, scan planning, metrics reporting, the separate credentials endpoint, and commitTransaction. The client builds eight URLs in total (catalog.rs:177-215), and none of them is a view, a scan plan, a metrics report or a transaction.
Stubbed — 1 endpoint. Catalog::update_namespace exists, but returns the error "Updating namespace not supported yet!" (catalog.rs:659) and sends nothing.

Two endpoints it does reach, it reaches only in part, and both are counted among the 13: load_table cannot ask for ?snapshots=all, and list_namespaces handles paging internally and never sends pageSize. It also reaches registerTable, which is one of the ten operations the earlier article did not cover, so it sits outside this count of 25.

Line numbers quoted here were read by hand and are archived, with the lines themselves, in evidence/rust-source-citations.txt.

Step 4 — Run the Reads Against Polaris

$ python3 run_rust.py --storage opendal
apache-polaris     1 implicit, 14 not-expressible, 11 not-issued, 7 ok
wrote evidence/rust-run-apache-polaris.json

All 7 supported read operations work. The one IMPLICIT result is the config request the client sends when it first connects, and the 11 NOT-ISSUED are writes this run does not send.

Step 5 — Run the Writes Against Polaris

The client has a method for each of those 11, and the read-only run sends none of them. A method that compiles can still fail on the wire, so this step sends them. Polaris is local, permissive and disposable, so the writes run there and on no other catalog, and every request goes through a small logging proxy that records what the client sent beside what came back:

$ python3 run_writes.py
apache-polaris     11 ok, 1 unsupported, 0 failed
wrote evidence/rust-write-surface.txt   22 request(s)

  probe                          verdict      ms     endpoint
  create_namespace               OK           60     reachable
  update_namespace_props         UNSUPPORTED  0      unsupported
  create_table                   OK           64     reachable
  commit_table                   OK           112    reachable
  commit_remove_properties       OK           119    reachable
  commit_add_schema              OK           116    reachable
  commit_set_current_schema      OK           116    reachable
  commit_upgrade_format_version  OK           99     reachable
  rename_table                   OK           3      reachable
  drop_table_purge               OK           16     reachable
  drop_table                     OK           3      reachable
  drop_namespace                 OK           1      reachable

All eleven work, and the scratch namespace is dropped at the end. The one refusal is the stub from Step 3, and the proxy log shows why it takes 0 ms: no request to the properties endpoint appears anywhere in the run.

🔎 Tip: The Proxy Log Shows Three Things

A properties commit always sends both update kinds. Setting a property and removing one produce the same pair, set-properties and remove-properties, because the crate's properties action builds both every time (update_properties.rs:95). The two differ in their contents.

Adding a column is one request carrying two updates, with a requirement attached:

  POST    200  /v1/quickstart_catalog/namespaces/irc_probe_rust_1790009108/tables/t1
          updates: add-schema, set-current-schema
          requirements: assert-current-schema-id

Every commit re-reads the table first, so one property change is a GET and then a POST. That reload is also why a transaction commits against the table's current state, which is open upstream as apache/iceberg-rust #3134.

The log also confirms one thing: a two-level namespace goes out as ...%1Fchild, the unit separator the specification asks for.

Step 6 — Match the Login to Each Catalog

The client can log in with a token, with an OAuth2 client ID and secret, or with fixed extra headers. It has no AWS request signing.

login	catalogs	with this client
OAuth2	Polaris	built in
token from an environment variable	Unity	built in
key-pair JWT	Horizon	built in: a `credential` with no colon is sent as `client_secret` with no `client_id` (`catalog.rs:238`), which is what Horizon expects
`gcloud` or `az` login	BigLake, OneLake	a token created outside the client, which the client cannot renew
AWS SigV4 signing	Glue, S3 Tables	not supported

Glue and S3 Tables require SigV4 on the catalog requests themselves. The specification does not describe that — its security schemes are OAuth2 and bearer tokens, and the signing in it covers storage access — but pyiceberg signs anyway, with rest.sigv4-enabled, rest.signing-name and rest.signing-region, so the Python REST client reaches all seven catalogs and the Rust one reaches five. The request to add it, apache/iceberg-rust #1236, has been open since April 2025.

Fixed headers look like a way round it, and they are worth one measurement: a signature minted for GET /v1/config and passed as a static header got a 200 on that request and a 403 on the next, on Glue and S3 Tables both, with all 7 checks refused through the client. A signature covers the request it signs.

The token the client does hold is refreshed by regenerate_token(), which only knows how to repeat an OAuth2 login, so a long-running program on BigLake or OneLake mints its own.

Step 7 — Reach AWS Through the Other Two Crates

Rust does reach both AWS catalogs. The same Apache project publishes iceberg-catalog-glue and iceberg-catalog-s3tables, both 0.10.1, which call the AWS APIs directly and sign as the AWS SDK does. All three crates implement the same Catalog trait, so a program written against dyn Catalog swaps between them by changing a dependency.

What changes with the dependency is the set of operations that answers. Reading each crate's impl Catalog for block:

  Catalog trait method     rest       glue       s3tables
  update_namespace         refused    sent       refused
  drop_table               sent       sent       refused
  register_table           sent       sent       refused

The other 12 trait methods are sent by all three. Each refusal says why:

  rest       update_namespace         'Updating namespace not supported yet!'
  s3tables   drop_table               'drop_table is not supported for S3Tables; use purge_table instead'
  s3tables   register_table           'Registering a table is not supported yet'
  s3tables   update_namespace         'Update namespace is not supported for s3tables catalog'

One of those four is the service speaking: S3 Tables requires a purge, which the earlier article measured from the wire, so that refusal is the catalog's own rule carried faithfully by the crate. The other three are crate-level gaps, and they land at runtime, since all three crates satisfy the same trait and compile the same way.

So a Rust tool covering all seven catalogs carries three catalog implementations and an operation set that varies by which one is loaded. In Python, one client covers all seven.

Step 8 — Point It at a Managed Catalog

$ python3 run_rust.py --only google-lakehouse --only microsoft-onelake --storage opendal
google-lakehouse   1 implicit, 14 not-expressible, 11 not-issued, 7 ok
wrote evidence/rust-run-google-lakehouse.json
microsoft-onelake  1 implicit, 14 not-expressible, 11 not-issued, 7 ok
wrote evidence/rust-run-microsoft-onelake.json

The same result as Polaris: all 7 supported read operations work on both, over the internet, with a token minted by gcloud and by az.

Step 9 — Read the Table's Files

load_table sets up file access but does not read anything, so each run also reads the table's metadata file through the client. The output lists the names of the settings the storage library received, never their values:

apache-polaris
  fileio config keys: ...
  storage credential keys among them: none
  read: ok, scheme file, 5614 bytes, 21 ms
google-lakehouse
  fileio config keys: ...
  storage credential keys among them: none
  read: ok, scheme gs, 5497 bytes, 451 ms
microsoft-onelake
  fileio config keys: ...
  storage credential keys among them: none
  read: FAILED after 47066 ms
  error: Unexpected => Failure in doing io operation, source: Unexpected
  (persistent) at read, context: { timeout: 10 } => io timeout reached

Local files and Google Cloud Storage read fine, the second one on the Google login already on the machine. No catalog passed along a storage credential, because the client never asks for one: it sends no X-Iceberg-Access-Delegation header, so the storage library finds a login on its own.

Azure is where that runs out. The Azure backend, opendal-service-azdls 0.57.0, has code to use your az login and is never given a way to run the az command (backend.rs:297), which leaves an account key, a SAS token or a service principal secret — and OneLake has no account key. Asking for a credential does not change it: OneLake hands one out, and with the delegation header passed as a static header the read still timed out, after 47181 ms. The client reads the credential from the response (types.rs:226) and copies only the response's config settings into file access (catalog.rs:455). Both halves are open upstream, as #2931 and #1442.

What You Add Beyond the Client

	needed for	what happens without it
a `reqwest` TLS feature	any `https://` catalog	every request fails before any response: 7 failed, 0 ok, no HTTP status. Upstream #2888
`iceberg-storage-opendal`	loading any cloud-stored table	`load_table` refuses: "StorageFactory must be provided for RestCatalog"
a storage login in your environment	reading any table file	the read fails or hangs; catalog-issued credentials are read and unused
a SigV4 signer	Glue and S3 Tables over REST	no login at all, and Step 7's two crates are the way round it

The first two are one line each. iceberg-catalog-rest 0.10.1 declares reqwest with TLS switched off and offers no feature to switch it on, so a build that never names a TLS backend fails on the first https:// request; Cargo merges features across dependencies, so your own line fixes it, and a project already using reqwest with TLS will never see it. The core iceberg 0.10.1 crate ships two storage factories, local files and memory (io/storage/local_fs.rs:330, io/storage/memory.rs:250), with its README pointing at the other crate on line 70.

Compare and Contrast

catalog	login	ok	failed	table files
Apache Polaris	🟢 OAuth2	7 reads, 11 writes	0	🟢 read
Google BigLake	🟢 `gcloud` token	7 reads	0	🟢 read, on the machine's Google login
Microsoft OneLake	🟢 `az` token	7 reads	0	❌ timed out
AWS Glue	❌ SigV4	0	7	—
AWS S3 Tables	❌ SigV4	0	7	—
Databricks Unity	token, not run	—	—	—
Snowflake Horizon	key-pair JWT, not run	—	—	—

Behind those numbers every catalog that answered showed the same shape: 14 endpoints the client has no method for, and every endpoint it does implement returning an answer.

Summary

The goal of this article was to point the Apache Rust Iceberg REST client at seven catalogs and record what works and what it takes. The key to the solution was matching every test to a line in the client's source code, running the same tests against each catalog, and logging every request the client sent. The results were:

🟢 Every endpoint the client implements answered on every catalog it could log in to: 7 reads on Polaris, BigLake and OneLake, and 11 writes on Polaris, with 0 failures
🟢 13 of 25 endpoints are implemented; of the other 12, 11 are missing and 1 is a stub that sends no request
⚠️ Two lines of Cargo.toml stand in front of that: a TLS backend (upstream #2888) and iceberg-storage-opendal for cloud storage
❌ Glue and S3 Tables require SigV4, which this client cannot send, so a Rust tool covering all seven catalogs loads three catalog crates where Python loads one — and those three disagree on three of the 15 Catalog methods, at runtime (upstream #1236, open since April 2025)
❌ OneLake's files could not be read: the Azure backend cannot use an az login, and a credential from the catalog is read and then ignored (upstream #2931, #1442)

Scope: iceberg-catalog-rest 0.10.1 with iceberg 0.10.1, iceberg-storage-opendal 0.10.1 and reqwest 0.12.28 with rustls-tls, built with rustc 1.98.1. Source read 2026-09-04, versions captured 2026-09-17, catalog runs 2026-09-18, write run 2026-09-21, one run each from one machine in one region. Polaris 1.7.0 ran in Docker with permissive settings and local file storage, and the writes ran there alone, so those results describe one permissive server. Glue and S3 Tables were tested only with the signature experiment, and Unity and Horizon were not run at all, their rows coming from the source. The tests check that each operation answers; the answers themselves are not checked. Three of the seven catalogs were on trial accounts in the earlier article, and managed catalogs do not report a version.

The strategy for testing what one Rust Iceberg client can reach was validated with an incremental step by step approach.

References

I Followed the n8n AWS Docs and It Broke at the First Command

Sarvar Nadaf — Fri, 25 Sep 2026 14:09:50 +0000

You SSH into a fresh EC2 box, run sudo dnf install -y docker, then the very next command from the guide you're following:

docker compose up -d

And it dies:

docker: 'compose' is not a docker command.

I hit this on camera. Followed the steps, watched n8n never come up, and spent a few minutes convinced I'd broken something. I hadn't. Almost every "self-host n8n on AWS" tutorial has this exact hole in it, because the authors tested on Ubuntu or a VPS where Docker installs differently. On Amazon Linux 2023, the default AMI for EC2, the command they tell you to run gives you half of what you need.

This post is the fix, and the full path from a bare EC2 instance to your first n8n login. Two containers, one paste-ready compose file, no reverse proxy yet (that's the hardening step, and it's its own article). By the end you have n8n on Postgres running on a box you own.

The gotcha, up front

On Amazon Linux 2023:

sudo dnf install -y docker

installs the Docker engine. It does not install the Compose v2 plugin. They're separate now. Compose stopped being a standalone docker-compose binary years ago and became a plugin that lives under Docker's CLI, and dnf's docker package doesn't bundle it. So the engine runs fine, docker run works, and then docker compose throws 'compose' is not a docker command because the plugin isn't there.

On Ubuntu you'd install docker.io plus docker-compose-plugin from Docker's apt repo and never notice. On AL2023 the plugin is on you. Here's the whole install, plugin included:

sudo dnf install -y docker
sudo mkdir -p /usr/local/lib/docker/cli-plugins
sudo curl -fsSL \
  https://github.com/docker/compose/releases/download/v2.29.7/docker-compose-linux-x86_64 \
  -o /usr/local/lib/docker/cli-plugins/docker-compose
sudo chmod +x /usr/local/lib/docker/cli-plugins/docker-compose
sudo systemctl enable --now docker

That third command is the one the other guides skip: it drops Docker's official Compose plugin binary into the directory the CLI actually looks in. Now both answer:

sudo docker --version
# Docker version 25.0.14, build 0bab007
sudo docker compose version
# Docker Compose version v2.29.7

Two version banners is how you know the box is actually ready. If you only check the first one, you find out the plugin is missing at docker compose up, which is the worst time to find out.

One catch on the download URL: it ends in x86_64, which is right for a t3 (Intel) box. If you launched a Graviton/ARM instance (t4g and friends), grab the aarch64 binary instead by swapping the filename to docker-compose-linux-aarch64. The wrong architecture installs cleanly and then fails with an exec-format error the moment you run it, which sends you hunting in the wrong place. Pick the binary that matches your instance. And v2.29.7 is just the version I pinned here; check Docker's releases and use the current one if you'd rather not lag.

Launch the box first

Before any of that, you need the instance:

AMI: Amazon Linux 2023.
Size: t3.small or larger. n8n plus Postgres want about 2 GB of RAM. Skip t2.micro and t3.micro (1 GB) unless you like watching containers get OOM-killed mid-run.
Security Group: SSH (22) from your IP only. For this private first test you can open 5678 to your IP only too. Do not open 5678 to 0.0.0.0/0. An open n8n editor on the public internet is an editor anyone can find and claim.
SSH in: ssh -i your-key.pem ec2-user@<PUBLIC_IP>

Everything below runs on that box as ec2-user.

Why bother self-hosting at all

n8n is a credential aggregator. One instance can hold your Stripe key, your database password, your Slack token, your Google OAuth, all in one place. On n8n Cloud that pile lives on someone else's server, priced per seat, capped on executions. Self-hosting moves it onto a box inside your own AWS account: no seat fees, no execution caps, your data stays home, and you pick the version.

The trade is honest. You now own the patching, the backups, and the security. This article gets it running. The hardening article (next in the series) closes the door behind it.

The stack: two containers

That's the whole thing.

n8n runs the editor and the workflow engine.
Postgres stores workflows and executions so they survive a restart.

Not SQLite. n8n defaults to SQLite, which is fine for a five-minute look, but it handles concurrent executions poorly and migrating off it later is an afternoon you won't enjoy. Start on Postgres.

Clone the repo so you have the compose file on the box:

git clone https://github.com/simplynadaf/self-host-n8n-on-ec2.git
cd self-host-n8n-on-ec2/compose

Here's the compose file:

services:
  n8n:
    image: n8nio/n8n:1.123.64
    restart: unless-stopped
    ports:
      - "5678:5678"
    environment:
      - N8N_SECURE_COOKIE=false
      - N8N_DIAGNOSTICS_ENABLED=false
      - N8N_PERSONALIZATION_ENABLED=false
      - N8N_ENCRYPTION_KEY=change-me-to-a-long-random-string-please
      - DB_TYPE=postgresdb
      - DB_POSTGRESDB_HOST=postgres
      - DB_POSTGRESDB_DATABASE=n8n
      - DB_POSTGRESDB_USER=n8n
      - DB_POSTGRESDB_PASSWORD=change-me-strong-db-password
    depends_on:
      - postgres

  postgres:
    image: postgres:16-alpine
    restart: unless-stopped
    environment:
      - POSTGRES_DB=n8n
      - POSTGRES_USER=n8n
      - POSTGRES_PASSWORD=change-me-strong-db-password
    volumes:
      - pg_data:/var/lib/postgresql/data

volumes:
  pg_data:

Three lines decide whether this works.

The image is pinned to 1.123.64, not latest. Pinning makes the build reproducible, and this version patches a real issue: CVE-2026-65589, an info-disclosure bug where credentials passed as custom headers in LLM sub-nodes could land in execution records. Run latest and you're one silent restart away from a version you didn't choose.

N8N_ENCRYPTION_KEY encrypts every credential n8n stores. Set a real 32-plus character random string and save it somewhere safe right now. Lose it and every saved credential is unrecoverable. A restored backup without this key is a database full of workflows whose logins can't be decrypted.

The DB password appears twice (DB_POSTGRESDB_PASSWORD in the n8n service, POSTGRES_PASSWORD in postgres) and the two values must match. If they don't, n8n can't reach its own database and the container just restarts in a loop while you wonder why the editor never loads.

Bring it up

Edit the two passwords and the encryption key, then:

sudo docker compose up -d
sudo docker compose ps

First run pulls both images and starts them. Give it twenty to forty seconds, then check n8n is answering:

curl -s -o /dev/null -w 'n8n -> HTTP %{http_code}\n' http://localhost:5678
# n8n -> HTTP 200

200 means n8n is up and serving. Once Compose was actually installed, this came back green on the first try.

First login

Open http://<PUBLIC_IP>:5678 in a browser. A fresh instance shows the setup wizard. Create your owner account with an email and a strong password, submit, and you land on the canvas.

Do this immediately after the box comes up, not tomorrow. The first account created on a fresh n8n becomes the owner, and until someone submits that form, it's open to whoever reaches it first. On a private Security Group that's only you. It's still a habit worth keeping.

One line in the compose file explains itself here: N8N_SECURE_COOKIE=false. That's only so first login works over plain HTTP on a raw IP while testing. It's a development shortcut, not a keeper. In production n8n sits behind HTTPS and this setting goes away.

Before this goes anywhere near the internet

The compose here publishes port 5678 directly. That's fine for a private test where the Security Group only lets your IP in, but it falls apart on the open internet, where certificate transparency logs announce every new HTTPS host within minutes and scanners find fresh boxes fast.

Before you point a domain at this or widen the Security Group:

Keep n8n patched (1.123.64 or newer).
Stop publishing 5678. Use expose so it's only reachable inside the Docker network.
Put Caddy in front for automatic TLS, and restrict the editor to your admin IP. Leave only /webhook/* public.
Move the encryption key and DB password into AWS Secrets Manager, read through a least-privilege IAM role.
Security Group: allow 22 (your IP), 80, 443. Never 5678 to 0.0.0.0/0.

That whole checklist is the hardening article in this series. If this box is going public, that's your next read.

Stop and start without losing data

Your data lives in the pg_data volume, so you can stop the stack safely:

sudo docker compose down     # stop, keep the data
sudo docker compose up -d    # start again

down stops the containers and keeps the volume, so your workflows and account are still there on the next up. To also wipe the data, that's down -v, and only when you mean it.

What you have now

n8n running on an EC2 box you control, backed by Postgres.
A pinned, patched image instead of a moving latest target.
An encryption key you actually set and saved.
A clear line for what to do before this faces the internet.

And you know the one thing most AWS n8n guides get wrong: on Amazon Linux 2023, installing Docker does not install Compose, and the fix is one curl into the plugin directory.

Where the series goes

Get it running (this one): bare EC2 to first login, past the Compose gotcha.
Harden it: TLS, closed editor port, Secrets Manager, least-privilege IAM.
Give it a brain: wire this same n8n to Amazon Bedrock and build a real AI agent on the canvas, model running in your account, no OpenAI key anywhere.

The compose file, the install and verify scripts, and the full setup notes are in the repo: github.com/simplynadaf/self-host-n8n-on-ec2.

Fn::GetStackOutput — Référencer les outputs CloudFormation entre comptes AWS

sylvainbruas — Fri, 25 Sep 2026 06:57:29 +0000

Dans un article précédent sur le VPC partagé, nous avons mis en place une architecture où le compte réseau possède et gère le Amazon Virtual Private Cloud (VPC) — subnets, tables de routage, NAT Gateways, AWS Transit Gateway — et les autres comptes AWS (par équipe ou par projet) viennent y déployer leurs workloads en utilisant les subnets partagés via AWS Resource Access Manager (RAM).

Cette architecture est élégante du point de vue de la gouvernance, mais elle soulève immédiatement une question pratique : comment les templates AWS CloudFormation des équipes font-ils référence aux ressources du compte réseau ? Le VpcId, les SubnetId, les security groups de base — tout ça vit dans le compte réseau, dans une stack que l'équipe réseau maintient.

Jusqu'à très récemment, les options disponibles étaient toutes imparfaites : hard-coder les IDs (fragile), passer par AWS Systems Manager (SSM) Parameter Store (overhead opérationnel), ou utiliser Fn::ImportValue (limité au même compte et même région). AWS vient de changer la donne avec Fn::GetStackOutput, une nouvelle fonction intrinsèque qui résout exactement ce problème.

Le contexte : VPC partagé en multi-compte

Rappelons rapidement l'architecture cible décrite dans l'article sur le VPC partagé. L'organisation AWS est découpée en plusieurs comptes :

Compte Réseau : propriétaire du VPC, des subnets, du Transit Gateway et de toute la couche réseau
Comptes Équipes : chaque équipe (ou projet) a son propre compte AWS ; leurs ressources (Amazon Elastic Compute Cloud (EC2), AWS Lambda, Amazon Relational Database Service (RDS), Amazon Elastic Container Service (ECS)…) sont déployées dans les subnets partagés depuis le compte réseau

Le compte réseau déploie une stack NetworkStack qui contient toutes les ressources réseau et expose des Outputs (VpcId, SubnetIds, NatGatewayId…). Les comptes équipes veulent utiliser ces valeurs dans leurs propres stacks CloudFormation.

Le problème : référencer des ressources cross-compte avant Fn::GetStackOutput

Avant l'introduction de Fn::GetStackOutput, trois approches coexistaient, chacune avec ses limitations :

1. Hard-coding des IDs

La solution la plus simple et la plus dangereuse. On récupère les IDs manuellement et on les copie dans les templates ou dans des fichiers de paramètres.

# Avant — fragile et non reproductible
Parameters:
  VpcId:
    Type: String
    Default: "vpc-0abc123def456789"  # ← IDs en dur
  SubnetPrivateA:
    Type: String
    Default: "subnet-0123456789abcdef0"

Le problème : si le compte réseau recrée le VPC (migration, incident, refactoring), tous les templates des équipes doivent être mis à jour manuellement. C'est une bombe à retardement.

2. SSM Parameter Store cross-compte

L'équipe réseau stocke les valeurs dans SSM, et les équipes consomment via des SSM dynamic references.

# Stack réseau — écriture dans SSM
VpcIdParam:
  Type: AWS::SSM::Parameter
  Properties:
    Name: /network/vpc-id
    Value: !Ref MyVpc

# Stack équipe — lecture depuis SSM cross-compte (impossible nativement)
VpcId: "{{resolve:ssm:/network/vpc-id}}"  # ✗ ne fonctionne pas cross-compte

Les SSM dynamic references ne fonctionnent pas en cross-compte. L'équipe réseau devrait publier dans SSM dans chaque compte équipe, ce qui génère un overhead opérationnel considérable.

3. Fn::ImportValue — limité au même compte

Fn::ImportValue est la fonction native CloudFormation pour partager des outputs entre stacks. Elle crée une référence forte (le stack producteur ne peut pas être supprimé tant qu'une stack consommatrice existe), mais elle est strictement limitée au même compte et même région.

# ✗ Ne fonctionne pas cross-compte
VpcId: !ImportValue "NetworkStack-VpcId"

Aucune de ces approches ne résout proprement le cas du VPC partagé en multi-compte. C'est pour ça que Fn::GetStackOutput est une avancée significative.

Fn::GetStackOutput : la solution native

Fn::GetStackOutput est une fonction intrinsèque CloudFormation qui permet de référencer les outputs d'une autre stack, potentiellement dans un autre compte AWS ou une autre région. Elle résout la valeur au moment du déploiement (create ou update), en appelant l'API DescribeStacks de CloudFormation.

Syntaxe

La fonction accepte quatre paramètres :

Paramètre	Obligatoire	Description
`StackName`	Oui	Nom de la stack productrice
`OutputName`	Oui	Identifiant logique de l'output (pas le nom d'export)
`Region`	Non	Région AWS de la stack productrice
`RoleArn`	Non	ARN du rôle IAM à assumer pour l'accès cross-compte

Comment CloudFormation résout la valeur

Le mécanisme est simple : au moment du déploiement, CloudFormation assume le rôle AWS Identity and Access Management (IAM) spécifié dans le compte réseau, appelle cloudformation:DescribeStacks sur la NetworkStack, récupère la valeur de l'output demandé, et l'injecte dans le template en cours de déploiement.

Mise en pratique : templates CloudFormation

Stack productrice — compte réseau

L'équipe réseau déclare ses outputs normalement. Pas besoin de Export — c'est une différence importante avec Fn::ImportValue qui requiert un nom d'export.

# NetworkStack — compte réseau (111111111111)
AWSTemplateFormatVersion: '2010-09-09'
Description: Infrastructure réseau partagée

Resources:
  SharedVpc:
    Type: AWS::EC2::VPC
    Properties:
      CidrBlock: "10.0.0.0/16"
      EnableDnsHostnames: true
      EnableDnsSupport: true
      Tags:
        - Key: Name
          Value: shared-vpc-production

  SubnetPrivateA:
    Type: AWS::EC2::Subnet
    Properties:
      VpcId: !Ref SharedVpc
      CidrBlock: "10.0.1.0/24"
      AvailabilityZone: !Select [0, !GetAZs ""]

  SubnetPrivateB:
    Type: AWS::EC2::Subnet
    Properties:
      VpcId: !Ref SharedVpc
      CidrBlock: "10.0.2.0/24"
      AvailabilityZone: !Select [1, !GetAZs ""]

  SubnetPrivateC:
    Type: AWS::EC2::Subnet
    Properties:
      VpcId: !Ref SharedVpc
      CidrBlock: "10.0.3.0/24"
      AvailabilityZone: !Select [2, !GetAZs ""]

Outputs:
  VpcId:
    Description: "ID du VPC partagé"
    Value: !Ref SharedVpc

  SubnetPrivateA:
    Description: "Subnet privé — AZ A"
    Value: !Ref SubnetPrivateA

  SubnetPrivateB:
    Description: "Subnet privé — AZ B"
    Value: !Ref SubnetPrivateB

  SubnetPrivateC:
    Description: "Subnet privé — AZ C"
    Value: !Ref SubnetPrivateC

Notez l'absence de bloc Export dans les outputs — Fn::GetStackOutput référence directement le nom logique de l'output (VpcId, SubnetPrivateA…), pas un nom d'export.

Stack consommatrice — compte équipe

# AppStack — compte équipe A (222222222222)
AWSTemplateFormatVersion: '2010-09-09'
Description: Application déployée dans le VPC partagé

Parameters:
  NetworkAccountId:
    Type: String
    Default: "111111111111"

Resources:
  AppSecurityGroup:
    Type: AWS::EC2::SecurityGroup
    Properties:
      GroupDescription: "SG pour l'application"
      VpcId:
        Fn::GetStackOutput:
          StackName: NetworkStack
          OutputName: VpcId
          RoleArn: !Sub "arn:aws:iam::${NetworkAccountId}:role/GetStackOutputRole"

  AppLoadBalancer:
    Type: AWS::ElasticLoadBalancingV2::LoadBalancer
    Properties:
      Type: application
      Subnets:
        - Fn::GetStackOutput:
            StackName: NetworkStack
            OutputName: SubnetPrivateA
            RoleArn: !Sub "arn:aws:iam::${NetworkAccountId}:role/GetStackOutputRole"
        - Fn::GetStackOutput:
            StackName: NetworkStack
            OutputName: SubnetPrivateB
            RoleArn: !Sub "arn:aws:iam::${NetworkAccountId}:role/GetStackOutputRole"
        - Fn::GetStackOutput:
            StackName: NetworkStack
            OutputName: SubnetPrivateC
            RoleArn: !Sub "arn:aws:iam::${NetworkAccountId}:role/GetStackOutputRole"
      SecurityGroups:
        - !Ref AppSecurityGroup

La syntaxe est identique pour chaque référence cross-compte. L'OutputName est exactement le nom déclaré dans le bloc Outputs de la stack réseau.

Configuration IAM : le rôle GetStackOutputRole

C'est la pièce centrale de la sécurité du dispositif. L'équipe réseau crée un rôle IAM dans son compte, que CloudFormation (depuis les comptes équipes) peut assumer pour lire les outputs.

Voici le template CloudFormation pour créer le rôle IAM dans le compte réseau :

# À déployer dans le compte réseau — peut faire partie de NetworkStack
GetStackOutputRole:
  Type: AWS::IAM::Role
  Properties:
    RoleName: GetStackOutputRole
    AssumeRolePolicyDocument:
      Version: "2012-10-17"
      Statement:
        - Effect: Allow
          Principal:
            AWS:
              - "arn:aws:iam::222222222222:root"  # Compte Équipe A
              - "arn:aws:iam::333333333333:root"  # Compte Équipe B
          Action: sts:AssumeRole
    Policies:
      - PolicyName: DescribeNetworkStack
        PolicyDocument:
          Version: "2012-10-17"
          Statement:
            - Effect: Allow
              Action:
                - cloudformation:DescribeStacks
              Resource:
                - !Sub "arn:aws:cloudformation:${AWS::Region}:${AWS::AccountId}:stack/NetworkStack/*"

Points importants sur ce rôle :

La permission cloudformation:DescribeStacks est suffisante — pas besoin d'accès aux ressources réseau elles-mêmes
Le Resource est restreint à NetworkStack uniquement, pas à toutes les stacks du compte réseau
La trust policy liste explicitement les comptes autorisés — on peut affiner avec un Principal par rôle d'exécution CloudFormation plutôt que par compte (:root)

Référence forte vs référence faible

C'est la différence fondamentale entre Fn::ImportValue et Fn::GetStackOutput :

Caractéristique	`Fn::ImportValue`	`Fn::GetStackOutput`
Portée	Même compte + même région	Cross-compte, cross-région
Nécessite un Export	Oui	Non
Type de référence	Forte	Faible
Suppression du stack producteur	Bloquée	Autorisée
Propagation automatique des changements	Non	Non

Une référence faible signifie que si l'équipe réseau supprime ou recrée la NetworkStack, CloudFormation ne l'en empêchera pas. Les stacks consommatrices ne le sauront pas immédiatement — elles échoueront uniquement lors de leur prochain déploiement.

Pour compenser, deux mesures sont recommandées :

Activer la protection contre la suppression sur NetworkStack :

aws cloudformation update-termination-protection \
  --enable-termination-protection \
  --stack-name NetworkStack \
  --region eu-west-1

Documenter les dépendances dans les READMEs et les runbooks — les références Fn::GetStackOutput n'apparaissent pas dans la console CloudFormation du compte réseau.

Intégration avec AWS CDK

AWS Cloud Development Kit (CDK) supporte nativement Fn::GetStackOutput pour les références cross-compte et cross-région. Avec la clé de contexte @aws-cdk/core:defaultCrossStackReferences :

strong (défaut) : utilise Fn::ImportValue pour les références dans le même compte/région
weak : utilise toujours Fn::GetStackOutput
both : génère les deux types simultanément (migration progressive)

Pour notre architecture SharedVPC, le comportement weak est le bon choix : on est obligatoirement en cross-compte.

// cdk.json
{
  "context": {
    "@aws-cdk/core:defaultCrossStackReferences": "weak"
  }
}

Ce qu'il faut retenir

Fn::GetStackOutput permet de référencer les outputs d'une stack CloudFormation dans n'importe quel compte AWS et n'importe quelle région, sans avoir besoin d'un nom d'export
Pour le pattern VPC partagé, c'est la solution idéale : l'équipe réseau expose ses outputs (VpcId, SubnetIds…), les équipes applicatives les consomment directement depuis leurs stacks
La résolution se fait au moment du déploiement via cloudformation:DescribeStacks — un rôle IAM dans le compte réseau est nécessaire pour les accès cross-compte
La référence est faible (pas de blocage de suppression) — compensez avec la protection contre la suppression sur la NetworkStack et une documentation explicite des dépendances
La valeur est figée au moment du déploiement. Si l'équipe réseau modifie un output, les stacks consommatrices doivent être mises à jour explicitement pour récupérer la nouvelle valeur

Fn::GetStackOutput ne remplace pas Fn::ImportValue pour les cas dans le même compte — la référence forte garde tout son intérêt quand on veut une garantie d'intégrité référentielle. Mais pour le cross-compte, c'est désormais l'option native et la plus propre.

Jev After Eight Days of Independent Tests: Level With Mid-Price LLMs, Behind the Frontier

xbill — Thu, 24 Sep 2026 14:26:03 +0000

This article provides a review of the independent evidence on TypeSafe's Jev, the open models built to replace it, and the prior art behind both, as of September 23, 2026. Every figure below is traced to a primary source, and re-scored from committed per-item outputs wherever the author published them. This is a snapshot eight days after launch, and the arXiv preprints it cites are days old and unrefereed.

On accuracy, Jev sits level with mid-price LLMs and 6.5 to 11.5 points behind the frontier in the cleanest comparison. Out of the box its probabilities are the best calibrated of the models measured on familiar English tasks and are wrong in both directions off them, and one temperature fitted on 50 to a few hundred labels fixes most of the error.

https://github.com/xbill9/gemma4-dev/tree/main/jev

What Is Jev?

Jev is a hosted model from TypeSafe AI, launched on September 15, 2026. It answers typed questions about a piece of text: choice picks one option from a list, score places the text on an ordered scale, and noul returns the probability that a statement is true. It writes no text.

The API is POST /v1/systemone at api.typesafe.ai. The price is $0.042 per million input tokens, and output is free. Access is through a waitlist, OpenRouter and Vercel's AI Gateway.

TypeSafe calls this category a System One model, after Kahneman's fast and intuitive system. The company raised a $40M seed round led by DCVC, and its co-founders are Diogo Almeida (CEO), Sasha Sheng (COO) and Erik Gafni (CTO).

The read itself is an old technique: end the prompt where the answer would start, take the model's scores for only the allowed label tokens, and apply a softmax over them. Any open model can be read this way; vLLM exposes it through logprob_token_ids, SGLang through /v1/score, and Featherless AI's SimpleJev wraps it as a Jev-compatible server.

What Counts as Evidence Here?

The review covers 14 arXiv preprints read in full, 104 GitHub evaluation repositories, 33 dev.to and Medium posts, TypeSafe's own pages and documentation, and press and forum coverage. Each figure carries one of three levels of support.

Re-scored: recomputed from committed per-item outputs, and it matched
Checked: the source's arithmetic recomputed from its published tables, with no per-item data available
Reported: the author's statement, which cannot be checked from outside

Studies are graded A (human or public labels, at least 200 items, stated method, committed outputs), B (real but small, synthetic, LLM-labelled or unreproducible) or C (a demo or a restatement).

By that rubric, the posts split sharply. Of 33 dev.to and Medium posts read in full, 5 are careful measurements on human or public labels, 11 are small, synthetic or scored by agreement between models, and 17 restate vendor claims. Of the 104 GitHub repositories, 33 grade A, 57 grade B, 5 grade C, and 9 are tools. Of the 14 arXiv preprints, 10 call TypeSafe's hosted Jev.

Who ran a study matters too. The studies with committed per-item outputs and no product in the race are Ibrahim and Zaki, Janardhan, sanand0, SamuelSacco and OmarMujahid, the last the most favourable to Jev; Rafe and Das also has none, and withholds its per-item outputs. Every's review was written by a launch partner. LangChain, which ships the langchain-typesafe integration, reported 100% on 500 judgments that are 5 test cases repeated 100 times (not re-checked). And the "15.9% faster pipeline" headline came from a TypeSafe employee's three-case demo. Bespoke Labs sells Nimble, and the replica authors test their own models. Matt Mastracci wrote the DiffusionGemma PR he compares with Jev, and Google's Gemma account shared his thread. heiko-hotz, whose SNIPS comparison includes Gemma 4 31B, works at Google. synthorai, which found Jev behind flash LLMs, runs a gateway that sells access to the models it compared.

Where Does Jev Land on Accuracy?

Level with mid-price LLMs and behind the frontier, at a fraction of the frontier's price. That trade is the product's design, and the numbers below show its size.

The largest pre-registered study, Ibrahim and Zaki's replication of a social-science annotation suite with 7,977 human-labelled items (arXiv 2609.24574), has Jev behind the best of 19 LLMs on 14 of 15 tasks, by a median 11.6 macro-F1 points. That comparator is the best model per task, picked after the results; within Jev's price band, Gemma 4 31B and Qwen3 235B score 3 and 2 F1 points above it. Its one per-item file checked, the 266-item dialect task, re-scores exactly to the paper's table.

Manjunath Janardhan's 200-item, six-model comparison on BANKING77, BoolQ, Yelp and ChaosNLI re-scores exactly from its committed logs.

Model	Accuracy	Calibration error (ECE)
🥇 Claude Fable 5.1	84.0%	0.064
🥈 GPT-6 Astra	79.0%	0.119
🥉 DeepSeek V4.1 Flash	76.0%	0.138
MiniMax M3	75.5%	0.112
Kimi K3	74.5%	0.119
Jev	72.5%	0.161

The two frontier leads (+11.5 and +6.5 points) exclude zero in their 95% intervals. Kimi K3, MiniMax M3 and DeepSeek V4.1 Flash sit within noise of Jev.

Banking77 is the one task many authors ran independently. Eight runs, one of them JevBench's own first-party run, put Jev at 0.753 to 0.840, with a median of 0.809. In every study that ran LLMs beside it, the best of them was ahead.

What Does Jev Do Well?

Binary and few-class decisions, fast, with answers that always fit the schema.

Spam: 98.33% on 18,514 emails, level with TF-IDF logistic regression at 98.39%, and it held up on newer mail where the trained baseline fell to 72.5%; the question wording was tuned on labelled errors from the same sets
Reranking: level with a dedicated reranker on eight retrieval sets, 0.692 against 0.691
Monitoring code for backdoors: AUROC of 0.976 and 0.970, catching about 90% of backdoors at 2% false positives; a best-of-5 attack cut detection to 60%
Schema compliance: zero invalid answers across 23,703 decision-model calls, Jev's included, in the largest study
Repeatability: 1.33% of answers changed between identical passes in one study, 2.2% in the phishing benchmark, and 0 of 96 repeats in another
Language: German cost 0.5 points on MASSIVE
Speed and price: about 0.1 s of server time, at $0.042 per million input tokens

Are Its Probabilities Calibrated?

They rank answers usefully; out of the box they beat most alternatives on familiar English tasks and miss badly off them.

TypeSafe publishes no calibration error, reliability plot, Brier score or log loss for Jev on any dataset. Its only confidence-to-accuracy figure is one cookbook of 60 SEC filings: 90% right at confidence 0.9 or above, 40% below.

The independent picture depends on the comparison.

On Bespoke Labs' 13-subset public suite, Jev's median ECE is 0.071, lower than Nimble-9B's on 11 of 13 subsets, and lower than every Gemma 4 read in the companion article (0.114 to 0.180) before any fitting.
Against LLMs that write their confidence as a number, Jev's own probabilities usually win. Its median ECE of 0.157 on the social-science tasks beats 16 of 19 LLMs.
Once every model gets one fitted temperature, 15 of those LLMs beat it.
Against LLMs that return full probability distributions, Jev's calibration error is the highest in Janardhan's run (0.161) and in sanand0's nine-model Banking77 pilot (0.138).

The direction of the error changes with the data. On public multi-class sets Jev is overconfident: on GoEmotions, labels it scored between 0.80 and 0.95 matched the human label 15% of the time. On crash narratives and synthetic items it is under-confident. Within one judge study, refit temperatures ran from 0.65 to 4.45, and its authors conclude that no single temperature fits.

The API also rounds every probability to 0.01. In one committed sample, 70.4% of choice probabilities came back as exactly 0, and noul is clamped between 0.01 and 0.98.

🔎 Tip: Fit a Temperature on 50 to a Few Hundred of Your Own Labels

A small refit fixes most of the calibration error, for Jev and for any open model read the same way.

A two-parameter refit on civil_comments took Jev's ECE from 0.16–0.21 to under 0.025 on a held-out half
Transferring a fitted slope and refitting only the intercept on 50 labels cut mean ECE by 74%
Out-of-fold Platt scaling cut Jev's ECE on crash narratives by 3.35x
In the companion article, one temperature fitted on 50 labels brought plain Gemma 4 26B's median ECE from 0.180 to 0.080 on the public suite, against Jev's 0.071 as shipped

The open replicas report the same order of correction: Nimble fitted 2.179, Luce 2.5 to 3.3 on unseen tasks. Luce's rule is to ship with 100 to 300 real labelled items and fit the temperature on them.

How Much Faster and Cheaper Is It?

Speed runs from 0.5x, slower than a local Gemma, to 12.1x faster; cost runs from 0.6x, dearer, to 478x cheaper, depending on the comparison model, the network path and where the clock runs.

Comparison	Source	Faster by	Cheaper by
GPT-5.6 Luna, reasoning off, client clock	OmarMujahid	1.04x	4.3x
DeepSeek, Banking77	sanand0	—	1.18x
Haiku 4.5, phishing verdict	jev-phishing-bench	2.9x	12x
Mid-size open model, 47 account reviews	devopsdaily	12.1x	7.4x
Claude Fable 5.1, via OpenRouter	Janardhan	9.9x	478x
19 LLMs, social-science tasks (median)	arXiv 2609.24574	—	18.6x
DeepSeek, 6G edge orchestration	arXiv 2609.23136	1.3x	0.6x (Jev dearer)
DeepSeek, edge orchestration, fees per correct completion	arXiv 2609.22753, same lab	—	3.2–3.4x
Local Gemma 4 26B-A4B Q4 on a mini-PC	ikkun1222	0.5–0.65x (Jev slower)	—

The server itself is fast: about 105 ms in one run, and about 76 ms once the network round trip is subtracted in another. Most of the cost gap comes from output being free. In devopsdaily's workload 83% of the old bill was output, and per input token the two prices differed by 1.31x. In the companion article, plain Gemma 4 26B on one EC2 L4 at full load cost at most $5.43 per million decisions, against Jev's $5.54 at the same 132-token prompts.

Where Does It Break?

On wording, language, and questions whose answer is absent from the input.

Option names move answers. With the question, input, rubric and option set held fixed, swapping which rubric sits behind "no" and "yes" changed 32.5% of Jev's answers, against about 2% with neutral names, and AUROC fell from 0.81 to 0.58 (arXiv 2609.26758, n = 1,200; reported). The paper is graded C as evidence about Jev: it names no dataset or Jev version, its gold labels likely come from a teacher LLM, its code is unreleased, and no generative LLM was given the same swap.

Bare labels mislead it. A router given option names with no descriptions sent all 40 hard tasks to the cheap model, at a median confidence of 0.96. One-line option descriptions fixed 37 of 40.

Language shift costs accuracy. Russian XNLI dropped from 88.3% to 77.3% with ECE tripling, and Spanish cost 3 to 6 points.

Jev is most confidently wrong where the answer does not follow from the input. With no "unknown" option, every answer on KoBBQ bias questions is wrong by design, and Jev gave them 0.79 confidence. On a fair die its choice probabilities averaged 83% against a true 17%, while noul gave 19%. On heart-risk data its probabilities ran about 3x too high, and a chat LLM on the same 5,000 people ranked risk slightly better (AUC 0.79 against 0.77).

Questions batched in one request cannot see each other's answers. One reviewer's request returned "suspend" for an account the same response classified as a developer sending tests.

TypeSafe's own list of known weaknesses names literal reading, arithmetic and counting, dates, long irrelevant input, adversarial content and phrasing: "refund" scored 0.72 and "not a refund" 0.47 on the same ticket.

What Does TypeSafe Claim, Checked?

193.6x faster and 444.6x cheaper. The source is TypeSafe's four in-house workflow evals, which the launch post calls "on the higher end of real world gains". TypeSafe discloses that the workflows "were made by individuals on our model capabilities team, so some bias could exist", and that its LLM comparators ran through its own probability adapter, which it says is "slower and more expensive". The comparator is never named. Checked against the published summary points, 444.6x fits only Jev against Opus 5 in the workflow setup. Averaged over all eight workflow setups, the figures are 97.8x faster and 149.2x cheaper. Timings came from laptops on the US West Coast, with no case counts, run counts or variance.

Eval accuracy. Reference labels are "an average of the responses of GPT-6 Astra and Claude Fable 5.1, both at high thinking", so accuracy there means agreement with two LLMs. Jev scores 67.8%, equal to Sonnet 5, with GPT-5.6 Sol leading at 74.1%. On invoice processing Jev scores 61.8% against Sol's 79.1%. TypeSafe states that it chose to publish no public-benchmark results.

"0% hallucination". TypeSafe's launch post: "Our number is not empirical. Schema matching is guaranteed." Every answer is a valid option; the wrong-but-valid rate is the accuracy figures above.

RLCD. Reinforcement learning for calibrated decisions has no paper, patent or method description. The CEO said on Hacker News that the architecture is "close to the chest for now, but we have talked about writing a paper." On Latent Space he agreed that all of Jev's training data is synthetic. Model size and base are undisclosed.

Pricing. The launch post says "We can't prove it isn't subsidized", and the home-page FAQ says "We can serve Jev profitably at our current prices." How long the launch price holds is an open question for anyone building on it.

The frontier comparison. On Hacker News a commenter described Jev as "basically a zero-shot classifier" that classifies "as accurately (they claim) as a frontier-level LLM", and the CEO replied "exactly right!" The independent accuracy results above are the test of the second half.

The founder. TypeSafe's pages say Diogo Almeida "co-invented RLHF". He is not an author of Christiano et al. (2017), which introduced it; he is a primary author of InstructGPT (arXiv 2203.02155), which applied it to instruction following, and on Latent Space he separates the two.

How the claims travelled:

Claim as circulated	Primary source
194x faster and 445x cheaper against GPT-6 Astra (Tom's Hardware)	Astra is a reference labeller with no scored setup; 444.6x fits Opus 5
Coverage of arXiv 2609.26758: Jev flips 70.4 of 100 answers	The paper gives 70.4 for an open ModernBERT head; Jev's is 32.5%
"Its own employee measured 15.9% on a real pipeline" (Cherry Creek News)	3 cases of a demo support ticket; costs modelled from an estimated price

Several self-published evaluations also state more than their own data supports, in both directions. The full report lists the corrections for Luce, Laya and NanoJev and uses the corrected figures from every study.

What Predates Jev?

Every piece of the mechanism.

Piece	Prior art
Score options by label likelihood	GPT-3 §2.4 (2020); MMLU §4.1 (2021); lm-evaluation-harness `multiple_choice`
Map each label to one token	Verbalizers, Schick & Schütze (arXiv 2001.07676)
Label probabilities carry prior bias	Calibrate Before Use, Zhao et al. (2021)
One fitted temperature fixes overconfidence	Guo et al. (2017); Kadavath et al. (2022), a temperature of 2.5 for an RLHF policy, beside Jev's 2.66, Nimble's 2.179 and Luce's 2.5 to 3.3
Post-training worsens calibration	GPT-4 report, Fig. 8: MMLU ECE 0.007 to 0.074 after PPO
Cross-entropy rewards reporting true probabilities	Gneiting & Raftery (2007): the log score is strictly proper
Cheap model first, defer when unsure	Selective classification (2017); FrugalGPT

The visible contribution is the packaging: a typed API, one hosted endpoint, output priced at zero, and many questions evaluated against one input in a single request. Whether RLCD adds more can be judged once it is published; Jev's native probabilities beating most LLMs' self-reported confidence suggests the training does something.

What About the Open Alternatives?

Each trained replica is built on Qwen or an encoder, and each publishes its own evaluation.

Project	Base	Own headline	Independent evidence
Nimble-9B (Bespoke Labs)	Qwen3.5-9B + LoRA	75.9% vs Jev 77.3% on 13 public sets	none; run by the seller, with raw probabilities (fitted temperature 2.179); its own model trails Jev
Kev (Jared Palmer)	Qwen3.5 + LoRA + pointer head	Kev-9B 0.852 test vs Jev 0.857 dev	none
Luce (scienthoon)	Qwen3-4B + LoRA	86.9% vs Jev 84.7% on 500 kubernetes issues	none
Laya (Convai)	ModernBERT-large 421M	0.766 vs 0.727 after training on the benchmark	ikkun1222: 29.2 / 60.0 / 40.0 on three Japanese tasks; the card's own zero-shot 0.362 sits below its 0.461 majority baseline
SimpleJev (Featherless)	any HF model; demo serves Gemma 4 26B	no accuracy published	none
DiffusionGemma via vLLM PR #57250	one denoise step of `diffusiongemma-26B-A4B-it`	198/201 vs Jev 191/201, "roughly tied"	ywchiu: 32.2% vs Jev 61.4% on synthetic routing; companion article: 75.9% vs Jev 77.3% on the public suite

SimpleJev's README states that its probabilities "are not calibrated probabilities of correctness". The DiffusionGemma headline comes from 201 items built by the PR's author; publishing the items would let others reproduce it.

Labels beat zero-shot wherever they were tried. A 310M Japanese encoder trained on 200 rows beat Jev on news topics by 12 points, 88.8% against 76.8%, and tied it on two sentiment tasks, and fine-tuned models led by 2 to 15 points on five public splits. On phishing, Luce trained on 1,000 labels reached 97.4% against Jev's 62.6% on the same benchmark, though on different items. On that benchmark, a two-line regex scored 91.6%, five narrow Jev questions combined by logistic regression reached 95.0% on a held-out half, and Haiku 4.5 asked the same five questions reached 93.2%, a difference too small to be significant (p = 0.063).

thejevai.com, registered on September 20 and promoted on Hugging Face as the endpoint for API keys, has no stated connection to TypeSafe. TypeSafe's API is api.typesafe.ai.

Compare and Contrast

	Jev	Trained open replica	Plain open model, label scores read
Labels needed	none to run; about 50 to fit a temperature	hundreds to thousands, plus a training run	none to run; about 50 to fit a temperature
Accuracy on human labels	level with Kimi K3, MiniMax M3 and DeepSeek V4.1 Flash; 6.5 to 11.5 behind the frontier	trails or matches Jev generically; beats it when trained on the task's own labels	Gemma 4 26B, 4-bit: 75.3% vs Jev 77.3% on the 3,880-record suite; level on yes/no, 4.5 behind on multiple choice
Calibration	🥇 median ECE 0.071 as shipped on the public suite; none published by the vendor	own evaluations	median ECE 0.180 as shipped, 0.080 after 50 labels
Where it runs	TypeSafe, OpenRouter, Vercel	your hardware	your hardware
Cost	$0.042 per million input tokens; $5.54 per million decisions at 132 tokens	your hardware	at most $5.43 per million decisions on an L4 at full load

So, Which One?

For binary and few-class decisions in English, at high volume, with no infrastructure to run and no labels yet, Jev is fast and cheap, its accuracy sits with mid-price LLMs, and its probabilities are the best calibrated out of the box on familiar English tasks. Plan on labelling 50 to a few hundred decisions to set thresholds and fit its probabilities.

For multi-class work with many options, decisions where the rule lives outside the input, or non-English text, where Jev loses accuracy, a model trained on your own labels won wherever one was tried; for non-English that evidence is one Japanese study.

For data that cannot leave your network, any open model read by label scores gives the same typed interface. In the companion article plain Gemma 4 26B trails Jev by 2.1 points overall and 4.5 on multiple choice, and needs about 50 labels to come within 0.01 of Jev's calibration.

What Is Still Unmeasured?

The published Gemma data points so far:

Gemma 4 31B beats Jev's median F1 in the pre-registered social-science study, 0.611 against 0.581
Gemma 4 31B scores 98.86% on SNIPS against Jev's 97.14% after its output's code fences were stripped; 80.57% before
Gemma 4 26B-A4B trails Jev by 11.6 points on Japanese news topics and ties it on two sentiment tasks
Gemma 4 E4B, stating its confidence as text, scores 81.0% against Jev's 95.5% on synthetic emails
Plain Gemma 4 26B read by its label probabilities, in the companion article: on four public tasks (sst2, AG News, DAIR Emotion, tweet_eval irony) and on Bespoke Labs' 3,880-record, 13-subset suite, 2.1 points behind Jev overall

Still unpublished: Jev's behaviour across model versions, calibration across difficulty on real data, and whether a generative LLM shows the same option-name sensitivity.

Summary

The goal of this article was to establish what independent evidence shows about TypeSafe's Jev and its open alternatives. The key to the solution was tracing every figure to its primary source and re-scoring from published per-item outputs wherever they existed. The results were:

🟢 Jev returns a valid option on every call, and its server time is about 0.1 s
⚠️ Accuracy sits with mid-price LLMs: 72.5% against 74.5–76.0% for Kimi K3, MiniMax M3 and DeepSeek V4.1 Flash in one six-model study, and a median 11.6 F1 behind the best LLM per task in the largest pre-registered one
🟢 Strong on binary and few-class work: 98.33% on 18,514 emails, and level with a dedicated reranker
🟢 Best calibrated out of the box on familiar English tasks: median ECE 0.071 on the 13-subset public suite
🟢 50 to a few hundred of your own labels and one fitted temperature fix most of its calibration error, and most of an open model's
⚠️ Measured speed gains run from 0.5x to 12.1x and cost gains from 0.6x to 478x; TypeSafe's 193.6x and 444.6x have no named comparator
⚠️ No calibration metric is published by TypeSafe, and the direction of its calibration error changes by domain
⚠️ Swapping option names changed about a third of its answers in one unreplicated study
⚠️ Plain Gemma 4 26B read by label probabilities (companion article) trails Jev by 2.1 points on the 3,880-record suite, is level on yes/no, 4.5 behind on multiple choice, and needs 50 labels to come within 0.01 of Jev's calibration
❌ RLCD has no published method, and the model's size, base and training data are undisclosed

Scope: sources read on September 23, 2026, eight days after launch: 14 arXiv preprints, 104 GitHub repositories, 33 dev.to and Medium posts, and TypeSafe's own pages; the companion article's public-suite run finished on September 24 UTC. Every accuracy, calibration and timing figure here is a third party's measurement, marked in the full report as re-scored, checked or reported, except the Gemma figures in Compare and Contrast and the Summary, which are the author's own measurement in the companion article; that article's Jev figures come from Bespoke Labs, which sells a competing model. Every run that states a version used jev-1.13.0. Sources were gathered and audited with AI assistance (Claude), and every figure was checked against its source, cited in the full report, except LangChain's, marked not re-checked there. The author is a Google Developer Expert and an AWS Community Builder; neither Google nor AWS funded, reviewed or saw this work. The author has no relationship with TypeSafe, Bespoke Labs or any replica reviewed.

The strategy for using primary sources to evaluate Jev was validated with an incremental step by step approach.

References

Full evidence review with every source: https://github.com/xbill9/gemma4-dev/blob/main/jev/reports/Jev%20independent%20evidence%20review.md
Companion measurement article: https://dev.to/gde/plain-gemma-4-26b-vs-jev-on-one-ec2-l4-21-points-behind-overall-level-on-yesno-45-behind-on-15k6
TypeSafe launch post: https://typesafe.ai/blog/introducing-system-one-models-and-jev
TypeSafe models and pricing: https://docs.typesafe.ai/models
TypeSafe evals: https://evals.typesafe.ai/
Jev 1.13 known weaknesses: https://docs.typesafe.ai/model-jaggedness/jev-1.13
Ibrahim & Zaki, arXiv 2609.24574: https://arxiv.org/abs/2609.24574
Rafe & Das, arXiv 2609.24052: https://arxiv.org/abs/2609.24052
Li et al., arXiv 2609.26550: https://arxiv.org/abs/2609.26550
Option naming, arXiv 2609.26758: https://arxiv.org/abs/2609.26758
Janardhan, jev-frontier-bench: https://github.com/manjunathshiva/jev-frontier-bench
Nimble public benchmarks: https://github.com/bespokelabsai/nimble/blob/main/docs/PUBLIC_BENCHMARKS.md
jev-phishing-bench: https://github.com/anisselbd/jev-phishing-bench
Luce: https://github.com/scienthoon/luce
OmarMujahid, jev-decision-bench: https://github.com/OmarMujahid/jev-decision-bench
bitnovus, jev-spam-eval: https://github.com/bitnovus/jev-spam-eval
SamuelSacco, jev-exploration: https://github.com/SamuelSacco/jev-exploration
sanand0, BANKING77 pilot: https://sanand0.github.io/llmevals/jev/
heiko-hotz, jev-evaluation: https://github.com/heiko-hotz/jev-evaluation
ywchiu, jev_benchmark: https://github.com/ywchiu/jev_benchmark
ikkun1222, Jev against a 310M encoder: https://dev.to/ikkun1222/jev-vs-a-310m-encoder-i-trained-myself-750-rows-three-tasks-two-different-winners-242e
SimpleJev: https://github.com/featherless-ai/simple-jev
vLLM PR #57250: https://github.com/vllm-project/vllm/pull/57250
Matt Mastracci, Jev against DiffusionGemma: https://x.com/mmastrac/status/2100626193943052784
Guo et al., On Calibration of Modern Neural Networks: https://arxiv.org/abs/1706.04599
Kadavath et al., Language Models (Mostly) Know What They Know: https://arxiv.org/abs/2207.05221
Brown et al., GPT-3: https://arxiv.org/abs/2005.14165
Hendrycks et al., MMLU: https://arxiv.org/abs/2009.03300
Schick & Schütze, verbalizers: https://arxiv.org/abs/2001.07676
Zhao et al., Calibrate Before Use: https://arxiv.org/abs/2102.09690
Gneiting & Raftery, Strictly Proper Scoring Rules, Prediction, and Estimation, Journal of the American Statistical Association 102 (2007)
Geifman & El-Yaniv, Selective Classification: https://arxiv.org/abs/1705.08500
Chen et al., FrugalGPT: https://arxiv.org/abs/2305.05176
GPT-4 technical report: https://arxiv.org/abs/2303.08774
InstructGPT: https://arxiv.org/abs/2203.02155
Christiano et al. 2017: https://arxiv.org/abs/1706.03741

Architecting an Enterprise Network on AWS Cloud WAN

Kasun de Silva — Thu, 24 Sep 2026 04:31:22 +0000

How to design a multi-region, multi-account AWS network on Cloud WAN, from segments and attachments to single-hop inspection, and the lessons from running one in production.

Most enterprise AWS networks grow the same way. One VPC becomes fifty. One region becomes three or four. A Transit Gateway appears in each region, they get peered, and a team spends a growing share of its week keeping route tables in agreement across all of them. It works, and it gets a little more brittle every quarter.

AWS Cloud WAN is the answer to that pattern. I have run it in production for a network of a few hundred accounts and several hundred VPCs across four regions, and this post is the guide I wish I had at the start: how regions connect, how segments carve up the network, how inspection works, how to attach VPCs at scale, and what to expect operationally. It is a design guide, not a tutorial. There is no console clicking here.

The problem Cloud WAN solves

A Transit Gateway is regional. Run in four regions and you run four of them, peer them in a mesh, and keep the route tables on all four telling a consistent story. Every new VPC is an attachment, a route table association, propagation into the right tables, and often a static route or two. Every new environment or partner multiplies that.

Two problems dominate at scale.

Routing intent is spread everywhere. No single document says "test can reach shared services, and nothing else." That intent lives in route tables in four regions, and you prove it by reading all of them.

Inspection is double-hop. Traffic between VPCs in different regions goes through a firewall leaving the source region and another entering the destination region. Stateful firewalls need to see both directions of a flow, and with independent regional Transit Gateways the only way to guarantee symmetry is to inspect at both ends. Twice the traversals, twice the logs, two rule sets that must agree.

Cloud WAN removes both problems, and that is the reason to adopt it.

Cloud WAN in one paragraph

Cloud WAN gives you a global network, which holds a core network that AWS operates for you. The core network has an edge in each region you choose, and AWS connects those edges over its backbone. You attach VPCs, Transit Gateways, Direct Connect gateways and VPNs to the edges. Every attachment lands in a segment, a routing domain that spans all edges. The whole thing, every segment, every rule about which segment can reach which, and where traffic gets inspected, is described in one versioned JSON document: the core network policy.

If Transit Gateway is a router you configure, Cloud WAN is a WAN you describe.

Reading the diagram

Columns are regional edges. Everything inside a column is attached to that edge.
Rows are segments. A segment exists on every edge, so cards in the same row reach each other across regions for free.
Green links are the shared services hub reaching into each environment. Two-way, never transitive.
Dashed orange arrows are internet egress. Every VPC drains to its region's inspection VPC, which holds the NAT and internet gateway.
Solid orange paths are two production flows to another cloud, inspected once each and exiting through the nearest Direct Connect landing.
The VPNs have no arrows because their path is the same as the orange one, with the VPN in place of the gateway.

The rest of the post explains each of these in turn.

Regions: edges, not peerings

Each region you list in the policy becomes a core network edge. AWS peers every edge to every other edge. You never create an inter-region peering attachment, and you never maintain inter-region routes. When a VPC attaches in Sydney, its CIDR is learned by the edges in Virginia and Oregon automatically, inside whichever segment it joined. Inter-region routing is just intra-segment routing.

The default behaviour is region affinity. Traffic stays in its home region wherever it can and crosses the backbone only when the destination or a policy says so. That gives predictable paths by default, and you override it deliberately when you want a particular region to handle something, such as inspection or egress.

An edge is a router you never see. Every attachment in a region terminates on it, it holds one route table per segment, and AWS peers it to every other edge. It has no internet gateway and no firewall of its own. Anything that needs either has to be a VPC you attach to it, which is why the inspection VPCs carry both jobs below.

Each edge needs its own ASN from a range you reserve in the policy. Pick a private range, keep it out of anything you peer with, and never change it afterwards.

Segments: the routing intent, written down

Segments are where the design lives. A segment is a routing domain that exists on every edge. Attachments in the same segment can reach each other across regions. Attachments in different segments cannot, unless the policy shares routes between them.

An enterprise segment model that works well has four groups.

Group	Purpose	Isolation
Environment	One segment each for test, staging and production workloads	Isolated. A workload segment sees itself and shared services, nothing else.
Shared services	Central logging, DNS, egress, platform tooling	Shared into every environment segment across all regions.
External	VPNs, SaaS platforms and partners that do not fit the internal pattern	One per environment. Isolated, so external parties never see each other.
Inter-cloud	Footprints in other public clouds	One per environment. Isolated, routed through inspection.

Three rules make this model hold.

Shared services is a hub, not a bridge. Every environment segment can reach shared services, but sharing is not transitive. Test cannot reach production by going through shared services. The policy shares one segment's routes into another explicitly, and you only share in one direction: into the hub.

Split external and inter-cloud by environment. One external segment is simpler. Three means a test VPN endpoint can never be a path into production, and inspection rules for each environment stay separate.

Use segment isolation for anything third-party. Isolation is a segment attribute. Set it and attachments inside the segment cannot reach each other, only what you share in. Every partner VPN becomes an island that reaches exactly what you routed to it.

Written out, the reachability of the whole network fits in one table. This is what the policy encodes, and it applies identically on every edge.

Segment	Can reach	Isolated	Inspected
Test	Test and shared services, all regions	Yes	Egress only
Staging	Staging and shared services, all regions	Yes	Egress only
Production	Production and shared services, all regions	Yes	Egress only
Shared services	Every environment segment, all regions	No	Egress only
External (per environment)	Its own environment, via inspection	Yes	Every flow
Inter-cloud (per environment)	Its own environment, via inspection	Yes	Every flow

The segment count stays small. Twelve to fifteen segments covers most enterprises. If you find yourself creating a segment per team, you are rebuilding route tables under a new name.

Attachments: make joining the network boring

At a few hundred accounts, attaching a VPC has to be self-service or your network team becomes a ticket queue. The pattern:

Share the core network to every member account with AWS Resource Access Manager. One share, one organisational unit.
Let the VPC owner create the attachment in their own account, through whatever internal platform or Terraform module they already use.
Have the attachment carry a tag naming its segment. The tag is the only thing the owner controls.
Write an attachment policy in the core network policy that matches the tag and places the attachment in the segment. No human chooses a segment for a VPC.
Let route propagation do the rest. The VPC's CIDR appears in the segment's route table on every edge.

Turn attachment acceptance off for internal segments. If your network team manages both the core network and the accounts that attach to it, a manual approve step adds friction and no security, because segment placement is already decided by the policy, not by whoever created the attachment. Keep acceptance on for external segments where a third party could be on the other end.

Enable appliance mode on attachments to inspection VPCs, and only those. Appliance mode pins both directions of a flow to the same availability zone, which stateful firewalls need. Workload VPCs do not need it and it costs you a cross-AZ hop if you turn it on everywhere.

Inspection: one hop, not two

An inspection VPC is a VPC with a stateful firewall in it, attached to the core network, that traffic is steered through. Cloud WAN steers that traffic with network function groups.

You put inspection VPCs into a group, then in the policy you say which segment-to-segment flows must go via that group. That is a send-via rule, and it is how you inspect east-west traffic between environments, or between an environment and a partner. There is also send-to, which sends a segment's traffic to a group, typically a default route, and is how you do centralised egress.

Egress deserves a sentence of its own because it surprises people. Every segment, shared services included, gets a send-to rule that installs a default route towards the group, and the inspection VPC holds the NAT and internet gateway. Nothing else on the network has a way out. A workload VPC with its own NAT gateway is a hole in the design, so do not build one.

The win is single-hop inspection. Because the core network sees the whole flow end to end, it can send both directions through the same firewall and keep the flow symmetric even when source and destination are in different regions. An inter-region flow crosses one firewall, not two. Fewer traversals, less log volume, one rule set to reason about, and far simpler troubleshooting.

Which region's firewall handles a cross-region flow? The policy decides. You can prefer the region closest to the source, set an explicit priority order, or override per edge. That lets you route a small region's traffic through a neighbour's firewall until the small region justifies its own, and fail over when a firewall needs maintenance.

Design the groups per environment. One group for production inspection, one for lower environments, one for egress. Sharing a firewall between production and test saves money and blurs a boundary you spent the segment model building.

Hybrid and multi-cloud: same edges, same segments

Everything that is not a VPC attaches to the same core network and lands in a segment like anything else.

Other clouds connect over a Direct Connect gateway attachment. Many enterprises deliver those circuits through a connectivity provider rather than dedicated fibre at every site. Land them in the inter-cloud segments and send all their traffic via inspection. Land the same cloud in two regions. Both gateways advertise the same prefixes into the segment, each edge prefers the nearer one, and if a circuit drops its routes withdraw and every edge converges on the other. The two gateways never talk to each other, and a VPC in Oregon does not cross the Pacific to reach a cloud it can exit to locally.
VPNs and partners land in external segments, one per environment, each isolated so partners never share a routing domain.
Existing Transit Gateways can attach as route table attachments. That is a bridge, not a native attachment type, and it is how you keep older things running while you migrate them.

The result is one place to look. A partner VPN, a link to another cloud and a production VPC are rows in the same attachment list, placed by the same policy.

Migrating from Transit Gateway without a big bang

Cloud WAN and Transit Gateway coexist well, and that is the migration.

Build the core network beside the Transit Gateways. Peer each regional Transit Gateway to its local edge. The Transit Gateways' prefixes propagate into Cloud WAN, so Cloud WAN already knows how to reach everything.
Attach VPCs without moving traffic. Each VPC gets a Cloud WAN attachment and its CIDR propagates, but its route tables still point at the Transit Gateway. Verify the routes are learned on every edge before going further.
Cut over with managed prefix lists. Add Cloud WAN routes to VPC route tables using a prefix list, so a growing set of destinations moves to Cloud WAN while the Transit Gateway stays the default. You move routes per environment, per region, and roll back by editing a prefix list.
Flip the default. Once a VPC's traffic is all on Cloud WAN, point its default route at the core network and remove the Transit Gateway attachment.

Go test, then staging, then production. Run Route Analyzer before and after every step to confirm paths, and run synthetic connectivity tests across attachments to catch what Route Analyzer cannot.

Operating it: what to know before you commit

The design above is the good news. These are the things that will shape your day-to-day.

Policy changes take minutes. Applying a new policy version, and rolling one back, takes minutes rather than seconds. On a core network carrying production that is a long time to watch a change land. Test in a lower environment first, and know exactly which version you are rolling back to before you press the button.

Everything is one document. That is the strength and the hazard. A mistake in one attachment rule affects every region at once. Treat the policy like production code: Terraform, pull requests, plan output reviewed, applied by a pipeline, never edited in the console.

The console diff gets noisy. A large policy does not always make a small change look small in the change view. Review the diff in source control first.

Permissions cut across accounts and services. The roles for the team that runs the core network, and for the automation that creates attachments in member accounts, take real least-privilege work. Start that with your security team early, not the week before production cutover.

Observability is on you. Cloud WAN gives you route tables, attachment state and a Route Analyzer. It does not give you flows. Ship VPC flow logs from every attachment and every inspection VPC into one queryable store. That is where you will actually see what the network is doing.

Cost is per edge and per attachment. You pay an hourly rate for each core network edge and each attachment, plus data processing. There is no per-request billing. A dormant region with an edge still costs money, so only list regions you actually use.

Is it worth it?

For a multi-region, multi-account network, yes. You replace a mesh of regional Transit Gateways and their route tables with one policy. You halve firewall traversals for inter-region traffic. Attaching a VPC becomes a tag. And "what can test reach?" is answered by a file instead of a scavenger hunt through four regions.

Cloud WAN is still young and it shows in change velocity and tooling. But the model is right. Describe the WAN, and let AWS run it.

If you are running a Transit Gateway mesh and weighing this up, I am happy to compare notes. Find me on blog.kasun.dev.

Cómo darle memoria semántica a un agente de IA sin base de datos vectorial

Kevin Lupera — Thu, 24 Sep 2026 03:05:33 +0000

Puedes darle a un agente de IA serverless una memoria durable y basada en significado usando búsqueda vectorial nativa de DynamoDB y embeddings de Amazon Bedrock, sin una base de datos vectorial dedicada y sin pipeline de sincronización. En esta demo el embedding vive junto al ítem, un solo PutItem escribe ambos, y buscar por significado es una única llamada a SearchVectors. Medido cara a cara en us-east-1, esta ruta nativa fue ~2.5x más rápida en escrituras y ~1.4x más rápida en búsquedas (p50) que S3 Vectors para la memoria en la ruta caliente de un agente.

📦 Clona y ⭐ strands-dynamo-vectors

Este es el fallo del que trata todo el post. A un agente recién creado le preguntas por un dato que le contaron a un agente anterior. Nunca vio esa conversación, así que no tiene con qué responder — echa mano de una tool, no encuentra nada, y termina pidiéndote que se lo repitas:

[session 2] user: What's my production cluster called and its region?

Tool #1: retrieve_context
Could you please provide the reference or context key that holds the production
cluster details? Alternatively, I can try to retrieve this information from other
available sources if you direct me.

Esa es una ejecución real de demo.py, no un experimento mental. Esto golpea a cualquiera que corra agentes en AWS Lambda, donde el contenedor muere entre invocaciones y cada turno es, en la práctica, un agente nuevo.

Lo que vas a aprender:

Las tres capas de memoria: por qué state, session y memoria a largo plazo no son lo mismo.
Cómo construir memoria semántica: enchufar la búsqueda vectorial nativa de DynamoDB al Strands Harness como backend de memoria.
El trade-off: latencia y costo medidos de DynamoDB Vectors vs un store dedicado (S3 Vectors).

¿Por qué un agente "olvida" entre turnos?

Piensa en tu agente como un asistente personal al que reemplazan por otra persona distinta cada vez que le hablas. No es una metáfora de Lambda: es literalmente lo que pasa. El contenedor que tenía la última conversación desapareció, así que entra un asistente nuevo sin cuaderno y sin memoria.

Para arreglar el "olvido" primero tienes que darte cuenta de que son tres problemas distintos, cada uno con su propio ciclo de vida. Mezclarlos es el bug que hace que un agente "recuerde lo que no debe y olvide lo que sí".

Capa	Qué es	Duración	¿Va al modelo?
State	key/value que usan tu app y tus tools	entre requests	No — inyéctalo tú si hace falta
Session	el historial de una conversación	hasta que dejes de reanudar ese id	Sí — es el contexto
Memoria a largo plazo	datos durables recuperados por significado	entre ejecuciones sin relación, para siempre	Se inyecta antes de cada turno

En términos del asistente: state es la nota adhesiva sobre el escritorio, session es el cuaderno de la reunión de hoy, y la memoria a largo plazo es que el asistente de verdad recuerde que prefieres desplegar por las mañanas, incluso semanas después y aunque lo digas con otras palabras.

El Strands Harness (su constructor de agentes ya ensamblado, create_harness) expone exactamente dos interruptores para esto:

create_harness(session={"id": "..."})       # reanudar ESTA conversación
create_harness(memory={"stores": [store]})  # llevar datos entre TODAS las ejecuciones

Fíjate en que son ortogonales. Session sobrevive al teardown de Lambda; la memoria sobrevive a todo, incluidas conversaciones que no comparten ninguna palabra.

Todo el punto de la búsqueda vectorial de DynamoDB es que el vector vive junto al ítem. Una sola tabla es a la vez tu store operacional y tu índice vectorial, sin un segundo servicio.

Contrasta las dos formas. RAG significa Retrieval-Augmented Generation: darle al modelo datos relevantes antes de que responda.

Enfoque	Piezas	Pipeline de sincronización	Lecturas
Vector DB dedicada	DynamoDB + Streams + Lambda worker + OpenSearch/Pinecone	Sí — Streams → Lambda → índice externo	latencia por consistencia eventual
DynamoDB Vectors (esta demo)	Solo DynamoDB (ítem + vector)	Ninguno	milisegundos de un dígito en la región

Paso 1: Crear la tabla con un índice vectorial nativo

Declara el índice vectorial al crear la tabla, no después. Agregarlo a una tabla que ya tiene ítems dispara un backfill que pagas en tiempo de reloj. Esta es la declaración del índice de setup_table.py:

ddb.create_table(
    TableName=TABLE_NAME,
    AttributeDefinitions=[
        {"AttributeName": "pk", "AttributeType": "S"},
        {"AttributeName": "sk", "AttributeType": "S"},
    ],
    KeySchema=[
        {"AttributeName": "pk", "KeyType": "HASH"},
        {"AttributeName": "sk", "KeyType": "RANGE"},
    ],
    BillingMode="PAY_PER_REQUEST",  # on-demand es OBLIGATORIO para índices vectoriales
    VectorIndexes=[
        {
            "IndexName": INDEX_NAME,
            "VectorAttribute": {"AttributeName": VECTOR_ATTRIBUTE},
            "SearchSchema": [
                {"AttributeName": "pk", "SearchSchemaElementType": "HASH"}
            ],
            "Projection": {"ProjectionType": "ALL"},
            "Dimensions": EMBED_DIMS,          # DEBE coincidir con el modelo de embeddings (1024)
            "DistanceFunction": DISTANCE_FUNCTION,  # COSINE
        }
    ],
)

Por qué importa: dos campos deben coincidir o la búsqueda devuelve silenciosamente nada útil — Dimensions debe igualar la salida de tu modelo de embeddings (Titan V2 = 1024), y DistanceFunction debe coincidir con cómo generaste los vectores (COSINE para embeddings de longitud unitaria).

Ejecútalo:

python setup_table.py

[setup] region=us-east-1 table=strands-agent-memory
[setup] create-table sent for 'strands-agent-memory' with vector index 'memory-vector-idx' (1024 dims, COSINE)
[setup] waiting for table to become ACTIVE ...
[setup]   table=ACTIVE index=ACTIVE backfilling=None
[setup] confirming the search endpoint serves the index ...
[setup]   searchable on attempt 1
[setup] DONE. The table is ready for writes and semantic search.

Por qué importa: que DescribeTable reporte ACTIVE es necesario pero no suficiente — el endpoint de búsqueda puede ir con retraso. El script demuestra que está listo lanzando una llamada real a SearchVectors en un bucle de reintentos, así que cuando imprime searchable on attempt N, el índice de verdad sirve consultas.

Paso 2: Convertir texto en vector con Titan

DynamoDB almacena y busca vectores, pero no los genera. Tu código llama a Bedrock. Este es el helper completo de embeddings de embeddings.py:

def embed(text: str) -> list[float]:
    """Devuelve el embedding de Titan V2 para text como lista de floats."""
    response = _bedrock.invoke_model(
        modelId=EMBED_MODEL_ID,  # amazon.titan-embed-text-v2:0
        contentType="application/json",
        accept="application/json",
        body=json.dumps(
            {"inputText": text, "dimensions": EMBED_DIMS, "normalize": True}
        ),
    )
    return json.loads(response["body"].read())["embedding"]

Por qué importa: normalize=True produce vectores de longitud unitaria, que es lo que espera la búsqueda COSINE. Y como es tu proceso el que llama a Bedrock, cada escritura semántica y cada búsqueda son dos llamadas facturables — una a Titan, una a DynamoDB — que es también por qué el rol de IAM necesita bedrock:InvokeModel, no solo permisos de DynamoDB.

Paso 3: Implementar el store de memoria que el Harness entiende

El Harness trae un file store cuya búsqueda es solapamiento de tokens por palabra clave: matchea palabras. Nosotros queremos que matchee significado. El store implementa el protocolo strands.memory.MemoryStore; los dos métodos que importan son add() y search(), de ddb_memory_store.py:

async def add(self, content: str, metadata: dict | None = None) -> str:
    """Genera el embedding del dato y lo guarda como vector buscable en DynamoDB."""
    key = f"memories/{uuid.uuid4().hex[:8]}"
    await self._storage.write(
        key,
        content.encode("utf-8"),
        vector=embed(content),   # primero el embedding, luego una escritura guarda ambos
        metadata=metadata,
    )
    return key

async def search(self, query: str, options: dict | None = None) -> list[MemoryEntry]:
    """Genera el embedding de la query y devuelve las memorias más similares por significado."""
    results = await self._storage.search(
        SearchQuery(
            vector=embed(query),
            top_k=self.max_search_results,
            pk=self._partition,       # acota la búsqueda a un solo tenant
            include_values=False,
        )
    )
    # ... mapea los resultados a MemoryEntry(content=..., metadata={"score": r.score})

Por qué importa: son async, tienes que hacerles await. Un write sin await no hace nada silenciosamente y tu tabla queda vacía. Ese await que falta es la trampa que más tiempo me costó (más abajo).

Paso 4: Demostrar el recall por significado y luego conectarlo al agente

Guarda un dato en una "conversación" y luego haz una pregunta sin relación que no comparta ninguna palabra de contenido. De demo.py:

fact = "The user prefers deployments to happen on Tuesday mornings."
await store.add(fact, metadata={"kind": "preference"})
await asyncio.sleep(2)  # deja que el índice vectorial se asiente (consistencia eventual)

query = "when does this user like to ship releases?"
results = await store.search(query)

[write] stored in some conversation: 'The user prefers deployments to happen on Tuesday mornings.'
[search] asked in ANOTHER conversation, by meaning: 'when does this user like to ship releases?'
  score=0.6665  The user prefers deployments to happen on Tuesday mornings.

Por qué importa: la query dice "ship releases", el dato dice "deployments"; la query no dice nada de "Tuesday". Un Query por prefijo nunca encuentra esto. El índice vectorial sí, porque busca significado, no texto — el score de coseno quedó en 0.6665.

Ahora enchufa ese mismísimo store a un agente Harness completo como su backend de memoria — esta es la juntura documentada (el seam):

store = DynamoDBVectorMemoryStore(
    TABLE_NAME, region_name=REGION, partition=DEMO_USER,
    index_name=INDEX_NAME, max_search_results=3,
)
agent = create_harness(
    model=MODEL, session=False,
    memory={"stores": [store]},   # <-- el seam: el Harness posee la inyección + la tool search_memory
)
reply = await agent.invoke_async(
    "I'm planning next week's release. Is there any timing preference on file for me?"
)

[agent] user: I'm planning next week's release. Any timing preference on file?
[agent] agent: You've indicated a preference for deployments to happen on Tuesday
mornings. This is noted for planning next week's release.

Por qué importa: el Harness sigue siendo dueño del manager de memoria. Llamó a search_memory, sacó la preferencia de DynamoDB y la inyectó en el contexto antes de que el modelo respondiera. Misma tabla, mismo índice — ahora moviendo a un agente real.

Lo que me costó tiempo

Los métodos add()/search() son async, y strands-dynamodb-storage (v0.1.x) es joven. Escribí un store, corrí la demo y obtuve una tabla vacía sin ningún error. Una escritura sin await devuelve una corrutina y no hace nada silenciosamente — sin excepción, sin fila.

💡 Si tu búsqueda semántica no devuelve nada y no hay error, revisa que cada await self._storage.write(...) y await store.add(...) esté realmente con await. Un await que falta es un no-op silencioso, no un crash.

Lo segundo: que DescribeTable diga ACTIVE no significa que el índice ya vaya a responder una llamada a SearchVectors. Haz polling con una búsqueda real (como hace setup_table.py) en lugar de confiar en el estado de la tabla.

¿Cuánto cuesta y es más rápido?

Los mismos 30 embeddings de Titan V2 (1024 dims) escritos en cada store, luego 30 búsquedas (TopK=5) en cada uno, midiendo solo alrededor de la llamada al store. Un solo escritor, ejecutado desde fuera de AWS, así que los números incluyen el round trip cliente↔región; dentro de una VPC son más bajos. Los datos crudos están en metrics.json.

Store	write p50	write p95	search p50	search p95
DynamoDB Vectors	121.05 ms	130.24 ms	122.09 ms	140.87 ms
S3 Vectors	304.36 ms	401.06 ms	167.15 ms	257.27 ms

Sobre el costo: los vectores viven en el almacenamiento normal de DynamoDB — los primeros 25 GB-mes son gratis, luego ~$0.25/GB-mes; el equivalente a una demo de memorias son kilobytes. Las escrituras y búsquedas se facturan por GB (la búsqueda tiene un mínimo de 1 KB), y el artículo de referencia midió US$0.0004 por 100 memorias + 30 búsquedas. Recuerda: cada operación semántica son dos llamadas facturables (Titan + DynamoDB).

Medido en una ejecución de un solo escritor, sin concurrencia, desde fuera de AWS, septiembre de 2026. Tómalos como orden de magnitud, no como un SLA.

Cuándo usarlo (y cuándo no)

No es "cuál es mejor", es qué representan tus vectores.

✅ Memoria de agente, usuarios, productos, recomendaciones, señales de fraude: datos operacionales que mutan seguido y se leen en la ruta caliente. DynamoDB Vectors — una escritura, cero pipeline de sincronización, lecturas de milisegundos de un dígito en la región.
❌ PDFs, docs, wikis, corpus de RAG: embed una vez, consulta muchas, enormes y mayormente inactivos. Usa S3 Vectors — optimizado para el menor costo de almacenamiento (~90% más barato para ese patrón).

¿Cómo hago la limpieza?

Con PAY_PER_REQUEST no hay cargo por capacidad inactiva, pero el almacenamiento factura por GB-mes mientras la tabla exista. Borra todo:

python cleanup.py

Esto borra la tabla de DynamoDB (su índice vectorial se va con ella), el bucket y el índice de S3 Vectors creados por el benchmark, y los archivos locales de sesión bajo ./.agent.

FAQ

¿Necesito una base de datos vectorial aparte para la memoria del agente? No. Si los vectores son operacionales — creados y leídos en la ruta caliente, como la memoria de un agente — la búsqueda vectorial nativa de DynamoDB los mantiene junto al ítem, así que no hay un segundo servicio ni pipeline de sincronización.

¿La partition key es un límite de seguridad? No. El pk acota y acelera una búsqueda, pero cualquiera con dynamodb:SearchVectors sobre el índice puede consultar cualquier partición. El aislamiento real entre tenants vive en IAM — tablas o índices separados si necesitas aislamiento fuerte.

¿Qué versión mínima del SDK necesito? boto3/botocore ≥ 1.43.64. SearchVectors de DynamoDB llegó al modelo de servicio del AWS SDK el 2026-08-04; los SDKs anteriores no exponen client.search_vectors.

Puntos clave

La memoria son tres cosas, no una. State, session y memoria a largo plazo tienen duraciones distintas; la mayoría de las necesidades de "memoria" eran solo una session sobreviviendo al teardown de Lambda.
Los vectores operacionales viven junto a su ítem. La búsqueda vectorial nativa de DynamoDB elimina la vector DB dedicada y su pipeline de sincronización para la memoria del agente.
Ajusta el store al perfil del dato. DynamoDB Vectors para datos calientes que mutan; S3 Vectors para bases de conocimiento grandes y mayormente inactivas.

Tu agente ya no tiene que ser el asistente al que reemplazan cada mañana y que tiene que pedirte que le repitas todo: puede recordar de verdad lo que le contaste, incluso semanas después, incluso con otras palabras.

¿Qué necesita recordar tu agente entre conversaciones, y lo estás guardando como state, session o memoria semántica? Cuéntame en los comentarios.

Referencias

¡Gracias!

Plain Gemma 4 26B vs Jev on One EC2 L4: 2.1 Points Behind Overall, Level on Yes/No, 4.5 Behind on Multiple Choice

xbill — Thu, 24 Sep 2026 02:00:40 +0000

This article provides a step by step guide to measuring Gemma 4 26B as a Jev-style decision model on an AWS EC2 L4 GPU, and compares a plain read of its label probabilities against DiffusionGemma's one-step read and against Jev's published results. The measurement was pre-registered, and every per-item output is committed.

On Bespoke Labs' 3,880-record public suite, plain Gemma 4 26B trails Jev 1.13.0 by 2.1 points overall, shows no measurable difference on yes/no questions, and trails by 4.5 points on multiple choice. Jev is better calibrated out of the box, and one temperature fitted on 50 labels brings Gemma's median calibration error within 0.01 of Jev's. Against DiffusionGemma, the plain read is level on accuracy, worse calibrated before fitting and similar after, and 1.9 to 5.1 times faster per decision.

https://github.com/xbill9/gemma4-dev/tree/main/jev

Why Measure This?

A Jev-style decision model answers a typed question with a probability for each allowed option, in one forward pass, with no generated text. TypeSafe's Jev does this as a hosted service. Any open model can do it: end the prompt where the answer starts, read the scores of the allowed label tokens, and apply a softmax over those.

A companion review of the independent evidence on Jev found two open questions for Gemma. Plain Gemma read this way had no published accuracy or calibration result. DiffusionGemma, read through vLLM PR #57250, had been described by Google's Gemma account on September 18 as "yielding well-calibrated decision distributions", with no published measurement behind it.

This run answers both on the same GPU, the same prompts, the same label tokens and the same scoring code, using community 4-bit (AWQ) builds of both models. A third run puts both, and Gemma 4 E4B, on the public suite where Bespoke Labs has published results for Jev, so the Gemma numbers sit beside Jev's on identical records.

At This Point You Should Have…

An AWS account with quota for one G-family instance, and the AWS CLI logged in
A security group, subnet and instance profile with AmazonSSMManagedInstanceCore for remote commands
Python 3 with transformers and pybase64 for the client side
The repository cloned: git clone https://github.com/xbill9/gemma4-dev and cd gemma4-dev/jev

Step 1 — Pre-Register the Measurement

The models, image, serving flags, data, metrics and comparisons are written down and committed before any model call. The file is PREREGISTRATION.md.

git show -s --oneline 3c67f62 d94471d a5cda09 894323e adf0c16

3c67f62 jev: pre-registration for the L4 AWQ run; option-order variant, label-count calibration curve, EC2 user-data
d94471d jev: run 2026-09-23-l4-awq — Gemma 4 26B label logits vs DiffusionGemma one-step reads, both AWQ 4-bit on one EC2 L4; per-item outputs, summary, host evidence
a5cda09 jev: pre-registration addendum (latency pass on-instance, off-label tokens, E2B/E4B bf16 arms); 20-split calibration analysis; archived run log and throughput arithmetic
894323e jev: plain arm builds its prompt from the served model's own chat template (26B unchanged, checked); E4B/E2B deviation recorded
adf0c16 jev: Bespoke Labs public-suite run — rebuild with checksum checks, runner via the proxy's Jev parser, scorer matched to Nimble's definitions, instance driver, generic launcher; pre-registration addendum

The latency and small-model runs were added in one pre-registration addendum after the first run, and the public-suite run in another; each was committed before its own first call, and each deviation from the plan is recorded there. The pre-registration commits to publishing every result, including any where either arm does worse.

Step 2 — Pick a Matched Pair of Checkpoints

An NVIDIA L4 has 24 GB of memory, so both 26B models run at 4 bits. Both checkpoints come from the same uploader and apply the same quantization settings to both models, so what differs is the model and how it is read.

	Plain arm	Diffusion arm
Checkpoint	`cyankiwi/gemma-4-26B-A4B-it-AWQ-4bit`	`cyankiwi/diffusiongemma-26B-A4B-it-AWQ-INT4`
Format	compressed-tensors, 4-bit, group 32, symmetric	same
Size	17.19 GB	17.22 GB

Both arms share one set of serving flags.

Step 3 — Launch One L4 on EC2

The security group opens port 8000 to one address only.

aws ec2 create-security-group --region us-east-1 --group-name jev-eval-sg \
  --description "jev eval: vLLM 8000 from one IP" --vpc-id <vpc-id> --query GroupId --output text
aws ec2 authorize-security-group-ingress --region us-east-1 --group-id <sg-id> \
  --protocol tcp --port 8000 --cidr <your-ip>/32

The instance boots the regional GPU Deep Learning AMI and runs aws/user-data.sh, which pulls a vLLM nightly that contains PR #57250 and serves one model at a time.

aws ec2 run-instances --region us-east-1 --image-id <gpu-dlami> --instance-type g6.xlarge \
  --subnet-id <subnet-id> --security-group-ids <sg-id> \
  --iam-instance-profile Name=<profile> --user-data file://aws/user-data.sh \
  --block-device-mappings 'DeviceName=/dev/sda1,Ebs={VolumeSize=100,VolumeType=gp3,DeleteOnTermination=true}' \
  --tag-specifications 'ResourceType=instance,Tags=[{Key=Name,Value=jev-eval},{Key=ManagedBy,Value=jev}]'

i-0c5b14e913b4c1019 g6.xlarge   us-east-1a

A g6.xlarge has the same L4 as the larger sizes with 16 GB of host memory, so user-data.sh adds a 16 GB swapfile on hosts under 30 GB; both 17 GB checkpoints loaded with it in place.

Step 4 — Serve the Plain Arm

One flag set serves both arms: Triton attention, eager mode, prefix caching, 2048-token context, 16 sequences and 32 logprobs. The diffusion arm adds a 64-token canvas.

/opt/jev/serve.sh ar

The client waits on /v1/models until the model answers.

READY after ~480s
['cyankiwi/gemma-4-26B-A4B-it-AWQ-4bit']

That covers the image pull, the 17 GB download and the load. The host record from the instance:

NVIDIA L4, 595.91.07, 23034 MiB
image: vllm/vllm-openai:nightly-e9757321527ca1ecd514c07c1418dd2c53da3d19 digest: sha256:f75fec992c293dd41042745f8aa71bfdf12b188b6a01e0baf075e4bca5d297e5
vllm 0.29.1rc1.dev573+ge97573215 torch 2.13.0+cu130

Step 5 — Check the Read Before the Full Run

Both arms use the prompt, answer template, label tokens and probability code from the PR's own structured_server.py. run_eval.py confirms both chat templates render the identical prefix before it sends anything. A five-example run checks the path end to end.

python3 run_eval.py --arm autoregressive --upstream http://<host>:8000 \
  --model cyankiwi/gemma-4-26B-A4B-it-AWQ-4bit --run smoke --limit 5 --concurrency 2

autoregressive sst2: 5/5 (5s)
autoregressive ag_news: 5/5 (1s)
autoregressive emotion: 5/5 (0s)
autoregressive irony: 5/5 (0s)

Each record stores the label probabilities, labels_returned, which confirms every label came back with a real probability, and label_mass, the share of the model's probability that landed on the allowed labels.

Step 6 — Run Both Arms

The data is 300 labelled examples each from sst2, AG News, DAIR Emotion and tweet_eval irony, stratified with a fixed seed. sst2 and irony carry annotators' labels, AG News its source's news categories, and DAIR Emotion labels taken from the hashtags on each tweet. Each arm runs once as written, and the three choice tasks run once more with their options listed in reverse order.

python3 run_eval.py --arm autoregressive --upstream $U --model $PLAIN --run 2026-09-23-l4-awq --concurrency 8
python3 run_eval.py --arm autoregressive --upstream $U --model $PLAIN --run 2026-09-23-l4-awq --concurrency 8 --variant reversed
/opt/jev/serve.sh diffusion
python3 run_eval.py --arm diffusion --upstream $U --model $DIFF --run 2026-09-23-l4-awq --concurrency 4
python3 run_eval.py --arm diffusion --upstream $U --model $DIFF --run 2026-09-23-l4-awq --concurrency 4 --variant reversed
wc -l results/2026-09-23-l4-awq/diffusion-*.jsonl

    300 results/2026-09-23-l4-awq/diffusion-ag_news.jsonl
    300 results/2026-09-23-l4-awq/diffusion-ag_news--reversed.jsonl
    300 results/2026-09-23-l4-awq/diffusion-emotion.jsonl
    300 results/2026-09-23-l4-awq/diffusion-emotion--reversed.jsonl
    300 results/2026-09-23-l4-awq/diffusion-irony.jsonl
    300 results/2026-09-23-l4-awq/diffusion-sst2.jsonl
    300 results/2026-09-23-l4-awq/diffusion-sst2--reversed.jsonl
   2100 total

The diffusion arm reads each example four times with different noise in the answer slots, as the PR's proxy does. The DiffusionGemma server came up in about 300 seconds with the image already pulled.

Step 7 — Score It

python3 score.py --run 2026-09-23-l4-awq

score.py writes SUMMARY.md and summary.json beside the records. Every figure below comes from those two files. Readouts: plain is one read of plain Gemma. diffusion is the proxy's automatic readout, the mean of four DiffusionGemma reads on the 96% to 100% of examples where its rule re-reads and one read on the rest; the label-count tables use the four-read mean throughout.

How Accurate Is Each One?

Level on three tasks, plain ahead on AG News.

Task	Always-majority	Plain Gemma	DiffusionGemma	Difference (95% range)
sst2	51.0%	95.0%	93.3%	−1.7 (−4.0 to +0.7)
AG News	25.0%	🥇 86.7%	83.0%	−3.7 (−6.7 to −0.7)
DAIR Emotion	35.0%	58.3%	60.3%	+2.0 (−1.0 to +5.0)
tweet_eval irony	60.3%	89.7%	87.3%	−2.3 (−5.7 to +1.0)

Throughout, level means the 95% range of the difference includes zero, and 🥇 in an accuracy table marks a lead whose range excludes it; in Compare and Contrast 🥇 marks the better value. Only the AG News difference excludes zero.

Are the Raw Probabilities Calibrated?

DiffusionGemma's are closer. Plain Gemma's are strongly overconfident.

Task	Plain Gemma ECE	DiffusionGemma ECE	Difference (95% range)
sst2	0.051	0.045	−0.005 (−0.021 to +0.020)
AG News	0.129	0.119	−0.010 (−0.030 to +0.020)
DAIR Emotion	0.386	0.261	−0.125 (−0.154 to −0.089)
tweet_eval irony	0.098	0.043	−0.055 (−0.077 to −0.014)

ECE is expected calibration error over 15 bins: the gap between how confident a model says it is and how often it is right. DiffusionGemma is better calibrated on emotion and irony, and level on the other two.

The fitted temperatures show the size of the gap. On the pre-registered single split, one temperature that best fits the labels was 3.48 to 7.24 for plain Gemma once 25 or more labels were used, and 0.73 to 2.36 for DiffusionGemma. A temperature above 1 means the model's probabilities are too extreme.

What Do 50 Labels Buy?

Most of the correction for plain Gemma; a smaller gain for DiffusionGemma, and none on irony.

Each task's examples split into a fitting half and a held-out half. One temperature is fitted on the first N labels of the fitting half and scored on the held-out half. On the pre-registered single split, 50 labels took plain Gemma from 0.040, 0.139, 0.440 and 0.088 to 0.029, 0.064, 0.102 and 0.042 on sst2, AG News, DAIR Emotion and irony, and DiffusionGemma from 0.047, 0.132, 0.311 and 0.055 to 0.025, 0.061, 0.115 and 0.054.

One split of 150 held-out examples moves ECE by several hundredths, so the same fit was repeated over 20 random splits, an analysis added after the run. Means, with the lowest and highest split in brackets:

Task	Plain, 0 labels	Plain, 50 labels	Diffusion, 0 labels	Diffusion, 50 labels
sst2	0.048	0.039 (0.021–0.061)	0.050	0.042 (0.025–0.081)
AG News	0.130	0.066 (0.037–0.132)	0.121	0.085 (0.060–0.138)
DAIR Emotion	0.392	0.089 (0.047–0.161)	0.266	0.111 (0.062–0.155)
tweet_eval irony	0.099	0.060 (0.035–0.130)	0.054	0.066 (0.031–0.116)

With no labels, DiffusionGemma is better calibrated on emotion and irony. With 50, plain Gemma's mean is at or below DiffusionGemma's on all four tasks, and the split-to-split ranges overlap throughout.

What Does DiffusionGemma's Read Do?

About half of its probability lands outside the allowed labels. The median share on the allowed labels per task:

Task	Plain Gemma	DiffusionGemma
sst2	1.000	0.503
AG News	0.999	0.370
DAIR Emotion	1.000	0.392
tweet_eval irony	1.000	0.593

At the answer slot the server returns the allowed labels plus the single most likely token. In the latency run that token was <eos>, the end-of-sequence token, in 40% of reads on sst2, 53% on AG News and 64% on DAIR Emotion, and the in 24% of reads on irony. The proxy rescales the label probabilities to sum to one, so the answer reads as confident either way; label_mass is the field that shows it, and it is worth logging in any deployment.

The proxy re-reads when the answer slot's entropy is above 0.1. It computes that entropy from the returned tokens' full-vocabulary probabilities without rescaling them, so a label holding half the probability on its own already scores 0.35. It re-read 100% of sst2 and AG News examples, 99.0% of DAIR Emotion and 96.3% of irony.

The re-reads barely change the answer: one read and the mean of four are within 0.6 points of accuracy on every task (93.3% and 93.3% on sst2, 83.3% and 83.0% on AG News, 60.7% and 60.3% on DAIR Emotion, 86.7% and 87.3% on irony). How much the four reads disagree separates right from wrong well on sst2 and irony, AUROC 0.891 and 0.873, and weakly on AG News and DAIR Emotion, 0.685 and 0.635.

Does Option Order Change the Answer?

For both models, by similar amounts. Listing the choice options in reverse order changed:

Task	Plain Gemma	DiffusionGemma
sst2	6 of 300 (2.0%)	7 of 300 (2.3%)
AG News	14 of 300 (4.7%)	21 of 300 (7.0%)
DAIR Emotion	27 of 300 (9.0%)	26 of 300 (8.7%)

How Fast Is Each One?

A second run timed both arms with the client on the instance itself, against localhost, one request at a time, over the first 100 examples of each task. Same image, flags and checkpoints, on a g6.4xlarge: the same L4 with a larger host.

python3 run_eval.py --arm autoregressive --upstream http://localhost:8000 --model $PLAIN --run 2026-09-23-l4-latency --limit 100 --concurrency 1
python3 run_eval.py --arm diffusion --upstream http://localhost:8000 --model $DIFF --run 2026-09-23-l4-latency --limit 100 --concurrency 1 --reads 4 --keep-top 5

| Task | Plain, median | Plain, 90th pct | Diffusion one read, median | Diffusion one read, 90th pct | Diffusion 4 reads (first + 3 parallel), median |
|---|---|---|---|---|---|
| sst2 | 61 | 62 | 118 | 119 | 306 |
| ag_news | 61 | 62 | 119 | 120 | 307 |
| emotion | 61 | 62 | 119 | 120 | 308 |
| irony | 61 | 61 | 121 | 122 | 312 |

Times are in milliseconds. A plain Gemma 26B decision takes 61 ms on one L4 for these prompts, which run 105 to 299 tokens with a median of 132; longer inputs such as full tickets or logs take longer. One DiffusionGemma read takes 1.9 to 2.0 times as long, and the automatic rule's four reads 5.0 to 5.1 times, which is the path it takes on nearly every call. At 61 ms one request at a time, one L4 answers about 16 plain decisions a second before any batching.

🔎 Tip: Fit One Temperature Before You Trust a Threshold

Both models give usable probabilities after one fitted temperature. score.py does it with a grid search on log loss:

def fit_temperature(probs, gold_idx):
    return min(T_GRID, key=lambda t: nll([temper(p, t) for p in probs], gold_idx))

Label 50 real decisions from the task, fit on them, and set any act-or-escalate threshold on the rescaled probabilities. Plain Gemma needs the larger correction, and both reach similar calibration once it is applied.

What About the Smaller Gemma 4 Models?

The label read works with any Gemma 4. DiffusionGemma ships only at 26B-A4B, so the smaller models get the plain arm alone, run in bf16 on the same L4 with the same flags. Their chat templates end at the model turn without the empty thought block the 26B template adds, so their prompt follows their own template. These two arms were added in a pre-registration addendum after the first run, and are exploratory. The 26B column is 4-bit and the others bf16, so the gaps mix model size with precision. All three read letter labels (A to F) with the meanings listed in the prompt, the format of the PR's proxy; smaller models may do better with the label words as the answers themselves.

Task	26B, 4-bit	E4B, bf16	E2B, bf16
sst2	95.0%	94.3%	88.7%
AG News	86.7%	83.7%	30.3%
DAIR Emotion	58.3%	54.0%	53.3%
tweet_eval irony	89.7%	84.3%	72.7%

E4B is 0.7 to 5.4 points behind the 26B, needs a smaller correction (fitted temperature 1.89 to 3.77 at 50 labels, against 4.65 to 7.05, both means over 20 splits), and reaches similar calibration once corrected: 0.042, 0.071, 0.089 and 0.081 against the 26B's 0.039, 0.066, 0.089 and 0.060. It changed more answers when the options were reversed on AG News, 35 of 300 against 14.

E2B answered "world" on 277 of 300 AG News examples, and on 263 when the options were listed in reverse order, so the same option wins whatever its position, and its topic accuracy sits near the 25% majority rate. Its fitted temperatures reach 7.99, the top of the pre-registered search range, so its corrected calibration figures understate what a wider search would reach.

How Does It Compare With Jev?

Bespoke Labs, which makes the open Nimble-9B decision model, ran Jev 1.13.0 through its API on a 13-subset public suite and published the results per subset, with the converters and record ids. The suite holds 3,880 human-labelled records: yes/no questions from BoolQ, PAWS, SQuAD 2.0, Civil Comments and Aegis 2.0; multiple choice from MultiNLI, PubMedQA, VitaminC and MASSIVE intents in English and German; and five-level ratings from HelpSteer2 and SummEval.

A third run, on a g6.xlarge with the same image and flags, rebuilt the suite from the public sources and read it with all three Gemma arms. All 13 rebuilt subsets matched the published checksums, so the Gemma figures and Bespoke Labs' Jev figures come from the same records.

Each record is a Jev request, which the PR's proxy converts with its own Jev parser, so the Gemma arms see the same prompt format as the first run: yes/no questions answered with the word yes or no, multiple-choice options lettered A, B, C with the answer read as a letter, and rating levels numbered from 1. Gemma was read in that format as published, with no prompt tuning, while Jev takes the request in its own format; a different prompt could move the multiple-choice gap in either direction. Scoring uses Bespoke Labs' definitions, and tests/test_suite_scoring.py checks it against Bespoke Labs' own code.

bash nimble_suite/build.sh <workdir>
python3 nimble_suite/run_suite.py --arm autoregressive --upstream http://localhost:8000 \
  --model $PLAIN --records <workdir>/public --run 2026-09-24-l4-suite
python3 nimble_suite/suite_stats.py --run 2026-09-24-l4-suite --run 2026-09-24-l4-suite-e4b

Accuracy pooled over records, with the 95% range in brackets:

Questions	Records	Jev 1.13.0, published	Nimble-9B, published	Plain Gemma 26B	DiffusionGemma 26B	Gemma 4 E4B
All	3,880	77.3% (76.0–78.6)	75.9%	75.3% (73.9–76.6)	75.9% (74.5–77.2)	73.3% (71.9–74.6)
Yes/no	1,399	84.6% (82.6–86.4)	80.1%	84.8% (82.8–86.6)	84.5% (82.5–86.3)	81.9% (79.8–83.8)
Multiple choice	1,848	82.8% (81.1–84.5)	81.1%	78.3% (76.4–80.1)	77.3% (75.4–79.2)	74.8% (72.8–76.8)
Five-level rating	633	45.2% (41.3–49.1)	51.2%	45.5% (41.7–49.4)	52.8% (48.9–56.6)	49.6% (45.7–53.5)

Jev leads plain Gemma by 2.1 points over all records (95% range 0.2 to 4.0) and by 4.5 on multiple choice (2.0 to 7.1). Jev's per-record answers are unpublished, so these ranges compare two independent proportions: pairing would narrow them, and records that share a passage or article widen them, most of all for ratings. The overall gap's range starts at 0.2 points, so it is the least secure of the three; the multiple-choice gap is the firm one. PubMedQA and VitaminC each account for 33 of the 84 records behind it, and PubMedQA has the largest single-subset gap, 77.2% against 64.0%.

On yes/no questions pooled, plain Gemma shows no measurable difference from Jev, 84.8% against 84.6%, a difference anywhere from 2.8 points ahead to 2.5 behind. Within them Jev is ahead on BoolQ by 5.7 points and plain Gemma on Civil Comments and SQuAD 2.0 by 6.0 and 6.4; PAWS, 5.2 points to Jev, is within its range.

On ratings DiffusionGemma leads both Jev, by 7.6 points (2.1 to 13.1), and plain Gemma, on exact-level accuracy; Bespoke Labs advises reading that beside the error of the probability-weighted level, which this article does not report. Plain Gemma and DiffusionGemma are level over all 3,880 records: 187 right only for plain, 211 right only for DiffusionGemma, exact McNemar p = 0.25. On ratings DiffusionGemma was right alone on 97 records against 51, p = 0.0002, and the whole gap comes from SummEval, whose 384 records come from 24 news articles. Records from one article move together, so that p-value overstates the evidence.

Bespoke Labs' open Nimble-9B scores 75.9% over all records, level with the two 26B reads. Gemma 4 E4B trails the 26B by 2.0 points over the whole suite, with single subsets ranging from 9.6 points ahead to 8.0 behind, and trails Jev by 4.1 (2.2 to 6.0).

Median calibration error over the 13 subsets, with Bespoke Labs' 10 bins:

	ECE as shipped	ECE after 50 labels	Brier as shipped
Jev 1.13.0, published	0.071		0.267
Nimble-9B, published	0.109		0.314
Plain Gemma 26B	0.180	0.080	0.359
DiffusionGemma 26B	0.114	0.074	0.290
Gemma 4 E4B	0.173	0.077	0.359

As shipped, Jev has a lower calibration error than plain Gemma on all 13 subsets, than DiffusionGemma on 11 and than E4B on 12. Brier score, which rewards accuracy and calibration together, favours Jev too: plain Gemma and E4B beat it on 2 of 13 subsets, DiffusionGemma on 4. DiffusionGemma's raw calibration error is lower than plain Gemma's on all 13, and its median share on the allowed labels was 66.5% per read on this suite.

After one temperature per subset, fitted on 50 labels from that subset, the Gemma medians sit 0.003 to 0.009 above Jev's as-shipped median. Per subset, fitted plain Gemma is still above Jev on 8 of 13, by up to 0.049, and the Gemma arms are at or below Jev on 4 to 6. The fitted figures are scored on the held-out half of each subset, 72 to 300 records, and ECE reads higher on fewer records, which works against the Gemma columns. Jev's figures are as shipped, and a temperature fitted on Jev's own output could lower them as well.

Matt Mastracci, who wrote vLLM PR #57250, compared DiffusionGemma with Jev on 201 hand-built items on September 17, and Google's Gemma account shared the thread the next day. DiffusionGemma answered 198 correctly and Jev 191; he called the two "roughly tied" and DiffusionGemma "the winner, I think." Seven of Jev's ten errors fell in one set, 89 words drawn from five sentences, and the items, code and per-item outputs are unpublished. On the public records DiffusionGemma is 1.4 points behind Jev, anywhere from 0.4 ahead to 3.3 behind, so both measurements find the two close overall; by question type they part, level on yes/no, 5.5 points behind on multiple choice (3.0 to 8.1) and 7.6 ahead on ratings. Plain Gemma 26B, read the same way without the diffusion step, lands within 0.6 points of DiffusionGemma over the suite (p = 0.25), so this suite shows no accuracy gain from the diffusion step.

His timings show the same cost for the re-reads: 1.9 to 4.6 times one read on his eight sets, against 2.6 times here. One read beat Jev's API on seven of his eight sets and the automatic re-reads lost on all eight; his single reads were timed warm and the re-reads cold, which exaggerates that second gap.

Compare and Contrast

	Jev 1.13.0, published	Plain Gemma 4 26B, label read	DiffusionGemma 26B, one-step read
Accuracy, public suite	🥇 77.3%	75.3%, 2.1 behind Jev; level with DiffusionGemma	75.9%, 1.4 behind Jev, a range that includes a tie
Yes/no, public suite	84.6%	84.8%	84.5%
Multiple choice, public suite	🥇 82.8%	78.3%	77.3%
Five-level rating, public suite	45.2%	45.5%	🥇 52.8%
Median ECE, public suite, as shipped	🥇 0.071	0.180	0.114, lower than plain on all 13 subsets
Median ECE, public suite, after 50 labels		0.080	0.074
Median Brier, public suite, as shipped	🥇 0.267	0.359	0.290
Accuracy, four tasks		🥇 level on three, ahead on AG News	level on three, 3.7 points behind on AG News
Raw calibration, four tasks		overconfident, fitted temperature 3.5 to 7.2	🥇 closer, fitted temperature 0.7 to 2.4
Probability on the allowed labels		🥇 about 100%	37% to 59%, 66.5% on the public suite
Time per decision		🥇 61 ms on the instance	118 to 121 ms for one read, 306 to 312 ms with the automatic rule
Option-order changes, four tasks		2.0% to 9.0%	2.3% to 8.7%
Price per million decisions	$5.54 at 132 input tokens	at most $5.43 on a `g6.xlarge` at full load	at most $31.72
Serving	TypeSafe's hosted API	any vLLM	vLLM with PR #57250

So, Which One?

Between the two Gemma 4 26B reads, for a Jev-style decision service on one L4: plain Gemma read by its label probabilities, plus one temperature fitted on about 50 labels. It is at least as accurate on the four tasks and level with DiffusionGemma over the public suite, faster, and places all of its probability on the answers you allowed.

DiffusionGemma's lower raw calibration error has a range excluding zero on two of four tasks and holds on all 13 public-suite subsets, and it matters when no labels exist at all. With 50 labels the difference is no longer detectable on the four tasks, and on the suite the two split 8 to 5, a count within chance (sign test p = 0.58). On five-level ratings DiffusionGemma scored higher than both Jev and plain Gemma, on few source articles; a rating task is the one place to try both.

Against Jev, on the same public records: plain Gemma 4 26B trails Jev by 2.1 points overall, shows no measurable difference on yes/no questions pooled, and trails it by 4.5 points on multiple choice. Its median calibration error comes within 0.01 of Jev's as-shipped median once one temperature per subset is fitted on 50 labels from that subset; per subset it stays higher on 8 of 13. Jev is the better calibrated with no labels at all.

Where a smaller model has to do, Gemma 4 E4B in bf16 gives up 0.7 to 5.4 points against the 26B on the four tasks and 2.0 points over the public suite. E2B falls apart on four-way topic classification with this prompt.

What Does a Decision Cost?

The run logs give a throughput. Plain Gemma answered 2,100 decisions in 51 seconds, 41.2 a second, at client concurrency 8. DiffusionGemma, reading each example four times, answered 2,100 in 298 seconds, 7.0 a second, at concurrency 4. The client was on a home connection, so these are lower bounds on what the L4 serves.

At the g6.xlarge on-demand price of $0.8048 an hour, that is at most $5.43 per million decisions for plain Gemma and $31.72 for DiffusionGemma with four reads, which ran at half the concurrency, so its figure is the looser bound. TypeSafe prices Jev at $0.042 per million input tokens, which is $5.54 per million decisions at this run's median of 132 input tokens and $12.56 at the longest, 299: about the same as the L4 at full load. The L4 is charged by the hour whether busy or idle, so its per-decision cost holds only at full load. The public suite's longer prompts would raise both the L4's cost and Jev's.

The first run, from launch to termination in under 1.2 hours, cost at most $0.97 of instance time. The second, on a g6.4xlarge at $1.3232 an hour for 1.19 hours, cost $1.57. The public-suite run, on a g6.xlarge for 0.82 hours, cost $0.66, including rebuilding the suite and reading 11,640 records across three models. All three, $3.20 together, exclude the prorated 100 GB volume. All ran on demand; G-family spot capacity in us-east-1 was unavailable at launch time.

Teardown

Every command on the instance went through AWS Systems Manager, so no SSH port was ever open, and port 8000 was open to one address.

aws ec2 terminate-instances --region us-east-1 --instance-ids i-0c5b14e913b4c1019

shutting-down

Then delete the security group, and confirm that nothing tagged ManagedBy=jev remains.

Summary

The goal of this article was to measure Gemma 4 26B as a Jev-style decision model, read two ways, for accuracy and calibration on labelled data, and to set it beside Jev's published results on a public suite. The key to the solution was a matched pair of 4-bit checkpoints on one EC2 L4, identical prompts and label tokens, and a measurement pre-registered before any call. The results were:

⚠️ On Bespoke Labs' 3,880-record public suite, plain Gemma 4 26B trails Jev 1.13.0 by 2.1 points overall (range 0.2 to 4.0) and 4.5 on multiple choice (2.0 to 7.1)
🟢 No measurable difference from Jev on yes/no questions pooled, 84.8% against 84.6%
⚠️ Jev is the best calibrated with no labels: median ECE 0.071 against 0.114 to 0.180 for the Gemma arms
🟢 One temperature fitted on 50 labels brings the Gemma medians to 0.074 to 0.080; per subset plain Gemma stays above Jev on 8 of 13
⚠️ DiffusionGemma 1.4 points behind Jev over the suite (0.4 ahead to 3.3 behind): 5.5 behind on multiple choice, 7.6 ahead on ratings; Mastracci's hand-built items had it 198 against 191, both close overall
🟢 Plain Gemma and DiffusionGemma level on accuracy: plain ahead only on AG News of the four tasks, 75.3% against 75.9% over the suite, p = 0.25
🟢 DiffusionGemma better calibrated as shipped, on emotion and irony and on all 13 suite subsets; after 50 labels the difference is no longer detectable
⚠️ DiffusionGemma places only 37% to 59% of its probability on the allowed labels, and the proxy's rescaling hides it
⚠️ The automatic re-read rule fires on 96% to 100% of examples, costs 2.6 times one read, and moves accuracy by at most 0.6 points
🟢 61 ms per plain decision on the instance, and at most $5.43 per million at full load, level with Jev's $5.54 at this run's prompt length
🟢 Gemma 4 E4B in bf16 0.7 to 5.4 points behind the 26B on the four tasks and 2.0 over the suite
❌ Gemma 4 E2B answered "world" on 277 of 300 AG News examples

Scope: one NVIDIA L4 in us-east-1 on three instances, a g6.xlarge for the accuracy run, a g6.4xlarge for the latency and small-model run and a g6.xlarge for the public suite, vLLM 0.29.1rc1.dev573+ge97573215, the 26B models 4-bit from the same uploader with the same quantization settings and E4B and E2B in bf16, 300 examples per task plus the 3,880-record public suite, one run per arm. Google's reference checkpoints are bf16, and 4-bit quantization may affect a diffusion model differently from an autoregressive one, so results at bf16 may differ for either arm. Latency comes from 100 examples per task with the client on the instance; throughput comes from the first run's client on a home connection. The latency, E4B and E2B arms were added in a pre-registration addendum after the first run, and E4B and E2B are exploratory; the 20-split calibration and the public suite's ranges, paired test, medians and per-subset counts were added after their runs, and the per-subset and pooled suite figures are the pre-registered ones. The suite's E4B arm ran under its own run name, 2026-09-24-l4-suite-e4b, with the same instance, image, flags and records. All four test sets and all 13 suite datasets were published before Gemma 4 and may be in its training data, and the high irony scores may indicate it there; whether they are in Jev's is unknown. No Jev call was made: the Jev and Nimble-9B figures are Bespoke Labs' published results on the same records, from one run of Jev 1.13.0 by a company that publishes a competing model, counting an invalid Jev response as wrong, with Jev's probabilities rounded to two decimals by its API. Code, pre-registration and every per-item output are in the repository. Parts of the analysis and writing were done with AI assistance (Claude); every figure comes from the committed output files.

The strategy for using label probabilities to run Gemma 4 as a decision model was validated with an incremental step by step approach.

References

Code, pre-registration and per-item results: https://github.com/xbill9/gemma4-dev/tree/main/jev
Companion review of the independent evidence on Jev: https://dev.to/gde/jev-after-eight-days-of-independent-tests-level-with-mid-price-llms-behind-the-frontier-1kln
vLLM PR #57250, DiffusionGemma structured reads: https://github.com/vllm-project/vllm/pull/57250
Plain checkpoint: https://huggingface.co/cyankiwi/gemma-4-26B-A4B-it-AWQ-4bit
Diffusion checkpoint: https://huggingface.co/cyankiwi/diffusiongemma-26B-A4B-it-AWQ-INT4
DiffusionGemma model card: https://huggingface.co/google/diffusiongemma-26B-A4B-it
Google's Gemma account on DiffusionGemma's calibration: https://x.com/googlegemma/status/2101069861598482817
Matt Mastracci, Jev against DiffusionGemma on hand-built items: https://x.com/mmastrac/status/2100626193943052784
Guo et al., On Calibration of Modern Neural Networks: https://arxiv.org/abs/1706.04599
Amazon EC2 G6 instances: https://aws.amazon.com/ec2/instance-types/g6/
TypeSafe Jev: https://docs.typesafe.ai/concepts/system-one
Bespoke Labs, Nimble public-suite results for Jev 1.13.0 and Nimble-9B: https://github.com/bespokelabsai/nimble/blob/0e67403/docs/PUBLIC_BENCHMARKS.md