DEV Community: Jangwook Kim

Google A2A 1.0 in Production: Our Multi-Vendor Interoperability and Compatibility Audit

Jangwook Kim — Thu, 24 Sep 2026 00:48:04 +0000

Why We Brought This Tool Into Our Lab

We tested Google's Agent-to-Agent protocol, or A2A, to evaluate communication between independently built AI agents rather than produce another demo. We brought it in to evaluate whether a shared protocol could reduce custom translation code when agents use different request and response formats.

A2A 1.0 needs a compatibility layer

A2A 1.0 handled discovery, task lifecycle, authentication failures, and artifacts between local agents, but pre-1.0 agent cards and methods did not work directly, so a versioned translation proxy remained necessary for backward compatibility.

One agent accepts web requests using Hypertext Transfer Protocol, or HTTP. Those requests use custom data fields encoded in JavaScript Object Notation, or JSON, a format for representing data as text. Another expects a continuous stream of updates. A third returns a job identifier that the client must use to check repeatedly for results. Authentication, cancellation, progress updates, and generated files all use different conventions. The first integration looks manageable; by the fifth, the team has substantial integration code to maintain.

A2A addresses that boundary. It gives agents a shared way to find each other, exchange messages, track work, send updates, cancel tasks, and deliver artifacts such as generated files. It does not standardize an agent's internal model, prompt framework, memory system, or tool implementation.

That distinction mattered in our evaluation. We tested whether A2A let agents exchange work reliably, not whether it chose agents or coordinated their work.

We anchored our test plan to three primary references: the original A2A interoperability announcement, the A2A 1.0 backward-compatibility discussion, and n8n's A2A implementation analysis. We then turned the protocol concepts into tests we could run. Successful discovery meant finding an agent and reading its connection details and capabilities, not proving that it could work with another agent.

Our local test setup contained two Python agents:

Coordinator agent: accepted a research request and delegated document rendering.
Renderer agent: accepted structured data and returned a Markdown artifact.
Compatibility proxy: a service that converted selected pre-1.0 agent descriptions and messages into formats the new client accepted.
Mock identity service: issued fixed bearer tokens, which grant access to whoever presents them, for tests with valid and invalid credentials.
Test client: recorded discovery responses, task states, artifacts, HTTP status codes, and protocol errors.

Agents published an agent card, a document describing how to contact them and which requests they accept. They exchanged structured requests and exposed task records with explicit status information. We inspected requests and responses with the command-line tool curl, the Python HTTP client httpx, and a reverse proxy that forwarded traffic to the agents. We could understand these exchanges without adding monitoring code inside a vendor's agent software.

We also confirmed what A2A did not handle. It did not choose an agent for a task or guarantee that two agents understood a skill the same way. It did not automatically exchange OAuth credentials, which let applications access services with permission, or accept arbitrary older request formats. We still had to decide where requests went, which callers we trusted, and which data formats we accepted. We also needed compatibility tests.

A2A connects agents to each other, while the Model Context Protocol, or MCP, connects an AI application to tools and sources of context. We use MCP primarily between an AI application and tools or context providers. We use A2A between autonomous or semi-autonomous services that expose agent-level capabilities. Combining them is reasonable: an A2A agent can invoke MCP tools internally. Replacing one with the other usually produces an awkward abstraction.

Hands-On Walkthrough: Setup, Execution & Output

We ran the audit in a separate Python environment and limited the software development kit, or SDK, to one major version. The kit supplies code for building software that uses the protocol. We also cloned the sample repository so that our first control run used the published sample structure before we introduced our own coordinator and renderer logic.

Our setup sequence was:

git clone --depth 1 https://github.com/a2aproject/a2a-samples.git
python3 -m venv .venv
. .venv/bin/activate

python -m pip install --upgrade pip
python -m pip install "a2a-sdk>=1.0,<2" httpx uvicorn pytest pytest-asyncio

# We recorded the resolved versions in the audit artifact.
python -m pip freeze > a2a-lab.lock

# We first ran the repository's Python sample following its checked-in README.
# We then started our two-agent fixtures on separate loopback ports.
python lab/renderer_agent.py --host 127.0.0.1 --port 8101 \
  --token renderer-lab-token > logs/renderer.log 2>&1 &

python lab/coordinator_agent.py --host 127.0.0.1 --port 8102 \
  --renderer-url http://127.0.0.1:8101 \
  --renderer-token renderer-lab-token > logs/coordinator.log 2>&1 &

python -m pytest -q tests/test_discovery.py \
  tests/test_task_lifecycle.py \
  tests/test_artifacts.py \
  tests/test_auth.py \
  tests/test_legacy_cards.py

We kept the agents deliberately simple. The coordinator received a text request, converted it into a structured rendering request, and called the renderer through its advertised A2A interface. The renderer emitted a Markdown artifact instead of returning the content only as conversational text.

Our test sequence covered five parts of agent communication:

Discover the renderer's agent card.
Select a communication method the agent supports rather than assuming one in advance.
Submit a message and follow its task state.
Retrieve and validate the resulting artifact.
Repeat the exchange with missing, invalid, and valid credentials.

We separately supplied a fixture, a pre-1.0 agent card used as controlled test data. It listed one service address and did not specify how the client and agent should agree on a protocol version. It also used the older operation names and task fields our legacy client expected. We tested the card directly with the 1.0 client, then repeated the test through our compatibility proxy.

The shortened simulated transcript below comes from our fixture mode, which produces the same output for the same test inputs. We used this output to avoid presenting timings from agents running on the same machine as evidence of production performance.

$ python lab/run_matrix.py --emit-fixture

[discovery] GET http://127.0.0.1:8101/.well-known/agent-card.json
[discovery] HTTP 200
[discovery] name="Effloow Renderer" protocolVersion="1.0"
[discovery] binding="JSONRPC" auth="Bearer"

[auth-negative] send without Authorization header
[auth-negative] HTTP 401 WWW-Authenticate="Bearer"

[auth-negative] send with invalid token
[auth-negative] HTTP 403 error="invalid_token"

[auth-positive] send with renderer-lab-token
[task] id="task-7f31" state="submitted"
[task] id="task-7f31" state="working"
[artifact] name="audit-summary.md" mediaType="text/markdown"
[task] id="task-7f31" state="completed"

[artifact-check] sha256 fixture match: PASS
[context-check] request context preserved: PASS
[terminal-state-check] completed task immutable: PASS

[legacy-direct] old discovery path: HTTP 404
[legacy-direct] top-level url without negotiated interface: REJECTED
[legacy-direct] legacy task method: METHOD_NOT_FOUND
[legacy-direct] sessionId field: SCHEMA_ERROR

[legacy-via-proxy] discovery path fallback: PASS
[legacy-via-proxy] interface normalization: PASS
[legacy-via-proxy] task method translation: PASS
[legacy-via-proxy] sessionId to contextId mapping: PASS

13 passed, 0 failed

The authentication result deserves emphasis. Advertising a bearer or OAuth-compatible security scheme in an agent card did not authenticate anything by itself. We still had to configure who issued tokens, which service could accept them, and when they expired. We also needed request checks that limited each caller's access to tasks.

For the local test, a fixed bearer token was sufficient because we were testing connection setup and failures, not the service that verifies identities. In production, we would use short-lived credentials and check who issued them and which service they were intended for. We would also tie task retrieval and cancellation to a verified caller.

The artifact test also caught more than a normal chat response test. We checked the file type, filename, and associated task. We also compared a checksum—a value calculated from the file's contents; and checked behavior after the task finished. That gave us a concrete contract for generated files rather than relying on text hidden in a message body.

Teams building similar automated tests can use our broader AI tools collection to compare related infrastructure tools. For architecture or migration support, see effloow services.

Compatibility Failures and Implementation Limits

Our 1.0 exchange worked under the expected successful conditions. Compatibility with pre-1.0 assumptions did not work automatically.

The first breakage occurred during discovery. Our legacy fixture requested its known discovery location and expected one top-level service URL. The 1.0 endpoint returned no card at that location, and the newer card required the client to select an interface. The client never reached message submission.

We handled this in the compatibility proxy. It answered requests at both discovery addresses and converted the older service address into the interface format the new client expected. It also kept the original card for troubleshooting. We left both agents' application code unchanged.

The second failure involved method dispatch: how the server selects the operation that handles a request. The legacy client invoked its historical task-oriented method, while our 1.0 endpoint accepted the current message-oriented operation. The server correctly returned a method error. Retrying the identical body under a different method name was not enough because the nested message and task fields also differed.

Our workaround translated the complete request and response pair:

older operation names to their 1.0 equivalents;
older session identifiers to the identifiers now used to group related exchanges;
older message components to the structure the new service accepted;
current task states to the status format the older client expected;
details about generated files to the older response format wherever it could preserve them.

By default, we rejected translations that would discard information. If a 1.0 response contained multiple artifacts or a capability the old client could not represent, our proxy returned an explicit compatibility error instead of silently dropping data. That policy made compatibility failures explicit instead of reporting success when data was missing.

The third issue was advertised capabilities that did not work in every case. Listing streaming, authentication, or a skill in an agent card did not prove that the agent consistently supported it. After discovery, we added conformance checks: requests that tested whether the agent actually supported its advertised capabilities. Our client used the card to choose what to test, not as proof of compatibility.

The fourth issue was the order of task updates. In an early version, our renderer reported completed before it saved the generated file with the task. A fast client could see a terminal state, meaning the task had ended, yet receive no generated files. We changed the renderer to save files reliably before marking the task as finished.

We also prevented finished tasks from changing state. Otherwise, a worker retrying an operation could mark a completed task as working again. A state machine controls which task states can follow others, and using the protocol's labels did not make ours correct.

Cancellation exposed another ambiguity. We could request cancellation, but we could not assume that an underlying model call or external tool stopped instantly. We therefore distinguished between receiving a cancellation request and confirming a canceled terminal state. Our worker checked for cancellation between processing steps and stopped publishing files once it had marked the task as canceled.

Authentication produced predictable but operationally important failures. A card describing a security scheme did not tell our client where its credentials should come from in every deployment. We had to configure that relationship outside the task message. We also had to ensure that logs redacted authorization headers and that redirected requests did not forward tokens to an untrusted host.

Finally, matching version labels did not establish compatibility. Two services can both mention A2A 1.0 yet differ in communication methods, authentication, file formats, streamed updates, or optional features. We now save a record of supported capabilities for each agent and card version, then stop using that record when the card changes.

Our rule: require data to match the expected format inside the core service. Handle known older formats only in a versioned adapter that translates them. Save data from failed exchanges as reusable compatibility test inputs, with sensitive information removed.

Scale, Latency & Cost vs. Alternatives

We did not publish response-time or processing-rate measurements from this audit. Both agents ran on the same machine and performed fixed work, so those measurements would describe our test setup more than A2A in production. Our three sources also did not measure performance under comparable conditions that would support comparisons across products.

We instead evaluated the network path structurally.

A synchronous A2A request keeps the caller waiting while the agent works, adding request handling, data checks, authentication, and conversion into a format the receiver understands. Streaming sends updates over an open connection and must handle receivers that cannot keep up. Asynchronous tasks let the caller continue while work runs, but require stored task state and repeated checks, streamed updates, or notifications. A compatibility proxy adds another network step unless it runs within the application or alongside it on the same machine.

Those costs are usually small compared with running a model or waiting for an external tool, but they can dominate agents that do simple, fixed work. We would not route a sub-process function call through A2A merely to claim protocol consistency.

Option	What we would use it for	Interoperability burden	Operational trade-off	Lock-in profile
A2A 1.0	Agent discovery, delegation, tasks, artifacts, and status	Medium initially, lower across conforming peers	Requires conformance tests, task storage, auth, and version policy	Lower at the agent boundary
MCP	Connecting an AI application to tools and context	Low to medium	Strong fit for tool exposure; not our preferred agent-task lifecycle	Lower at the tool boundary
Custom REST or gRPC	Stable internal services with tightly controlled callers. Representational State Transfer, or REST, organizes web requests around resources such as tasks. gRPC lets one service call functions in another.	Low for the first integration, rising with each peer	Maximum control, but we define every convention for tracking work	High unless other systems can use the same data formats
Workflow webhooks	Webhooks notify another service when an event occurs, allowing teams to trigger automated workflows.	Low for simple triggers	Easy to operate until tracking long-running work becomes complex	Medium and platform-dependent
Queue-specific RPC	Remote procedure calls, or RPC, let services request work from other services, here through a queue for high-volume internal processing.	Medium	Strong internal control; outside services need a gateway to connect	High dependence on the queue service and its message formats

For cost, we used an explicit engineering model rather than pretending the protocol itself has a licensing price. A2A itself was not our main expense. Our bill came from adapter work, security review, conformance automation, task persistence, and operations.

Our planning example assumed three days of one engineer's work for each custom connector between two platforms and four days for each vendor-facing A2A adapter. Connecting every pair of four agent platforms requires six custom integrations, totaling 18 engineer-days in this model. Four A2A adapters total 16 engineer-days.

That is only a break-even illustration, not a universal estimate. Under our illustrative assumptions, a custom connector costs less at two platforms, while A2A adapters cost less at four fully interconnected platforms. These crossover points depend on the assumed engineering effort and connection topology; they are not measured adoption thresholds.

Costs fall further when we run the compatibility tests in continuous integration, or CI, which checks every code change automatically. Each new agent implementation runs against the same discovery, auth, lifecycle, and artifact tests. We still write vendor-specific adapters, but we stop rewriting the acceptance criteria.

A2A did not eliminate integration engineering in our lab. It moved that work from ad hoc business code into a reusable protocol and compatibility layer. That is a meaningful improvement, but it is not zero-cost interoperability.

What This Audit Could Not Verify

This local audit did not establish production response times or processing capacity. Our sources also did not provide comparable performance measurements across products.

Our Final Verdict: When to Deploy, When to Skip

Our verdict is positive with conditions. A2A 1.0 is useful when we treat it as a tested network contract. It is risky when we treat the presence of an agent card as proof that two systems will cooperate.

Deploy this if:

We operate agents owned by different teams, vendors, or runtime stacks.
We need explicit task states rather than a single synchronous chat response.
We exchange files or structured artifacts that must remain associated with a task.
We need to learn what agents can do and agree with them on how to communicate.
We can run conformance checks against every agent release.
We are willing to keep authentication, authorization, and identity policy outside the conversational payload.
We expect enough integrations for shared A2A adapters to require less work than separate connectors between every pair of services.
We can persist task state and enforce valid transitions under retries and cancellation.

Hold off or avoid this if:

We have one caller and one stable internal service.
A normal function call, queue message, or typed REST endpoint already solves the problem.
We need every response to meet a strict deadline, and another HTTP request with validation checks is unacceptable.
Our agents cannot provide stable descriptions of their capabilities, accepted inputs, returned outputs, or generated files.
We expect A2A to coordinate work, choose agents based on request meaning, issue credentials, or enforce our application's access rules.
We cannot test old and new card formats during migration.
We need guaranteed compatibility with pre-1.0 clients but cannot operate a translation boundary.

For a system already using pre-1.0 agents, we would not upgrade every agent in place. We would first list how clients find agents, interpret their cards, send requests, group related exchanges, track tasks, handle generated files, and authenticate. We would then place a compatibility proxy in front of one non-critical agent. We would test saved requests through both the direct and proxy connections and reject any translation that loses information.

After that, we would update clients to agree directly with agents on a supported 1.0 interface and track use of the older connection method. We would name an adapter owner and define usage thresholds for retiring it, so temporary compatibility code did not become permanent by default.

The strongest outcome from our lab was not that two Python processes exchanged a message. Custom HTTP could have done that in an afternoon. The stronger result was that we could turn interoperability into a repeatable test matrix covering discovery, authentication failure, state transitions, cancellation, and artifacts.

The weakest point was backward compatibility. Pre-1.0 assumptions did not become safe merely because the current endpoint implemented A2A 1.0. We needed explicit translation, dual discovery support, and negative tests. We would test each legacy client directly before deciding whether it needs a translation layer; our fixture failures do not establish that every pre-1.0 migration requires one.

We would deploy A2A for a multi-vendor agent platform, but only behind a compatibility gateway and only with conformance tests in CI. For two closely connected internal services, we would keep a simpler application programming interface, or API, with defined request and response types. We would add A2A only when we had enough separate agents to justify the extra integration work.

If that boundary is already becoming expensive, contact our infrastructure team before adding another one-off connector. A week spent defining the compatibility contract is usually cheaper than discovering, during a vendor migration, that every agent interpreted the same task differently.

Langfuse vs Arize Phoenix Self-Hosted: What Our Traces, Evals, and Containers Exposed

Jangwook Kim — Wed, 23 Sep 2026 00:49:37 +0000

Why We Brought This Tool Into Our Lab

We did not need another dashboard counting tokens, the chunks of text a model processes. We needed to reconstruct failed agent runs and compare prompt revisions against fixed datasets. Prompts are inputs sent to a model; fixed datasets let us test each revision on the same examples. We also needed to attach human or automated scores and keep sensitive traces inside infrastructure we controlled.

Self-Hosted Observability Trade-Off

Because Langfuse required multiple stateful services while Phoenix scaled from a single container to PostgreSQL, choose Langfuse for product-facing prompt, feedback, and trace-review workflows, and Phoenix for OpenTelemetry-first Python evaluation workflows.

Our immediate production problem was scattered debugging information. A trace records the steps an application takes to handle a request. We collected application activity through OpenTelemetry, a standard set of tools for recording and sharing monitoring data. We kept model details in provider logs, test results in analysis notebooks, and prompt versions in deployment settings. To investigate a poor response, engineers had to match records across four systems manually. That worked for a prototype but wasted time once several engineers were changing the application at once.

We brought both Langfuse and Arize Phoenix into our lab because they overlap without being interchangeable.

Langfuse combined tools for operating applications built with large language models, or LLMs. It organized traces into spans, which record individual steps such as a model call. We could track model responses, group interactions into sessions, and connect them to users and prompt versions. We could also manage test datasets, experiment runs, scores, and annotation workflows in which people label or review results. This worked well for engineering and product teams investigating a customer interaction.

Phoenix helped us inspect application behavior and evaluate results using OpenTelemetry and OpenInference. OpenInference supplies shared names for trace details, such as the model used and the type of operation. We could reuse the code already collecting our traces. We could then inspect how the application fetched information and called tools, build datasets, and run Python experiments without adopting a product-specific trace format.

That distinction drove most of our final decision:

We preferred Langfuse when our priorities were managing prompts over time, collecting application feedback, grouping interactions into sessions, and reviewing traces as a team.
We preferred Phoenix when working with other OpenTelemetry tools, analyzing results in code notebooks, checking the quality of retrieved information, and defining experiments in code mattered more.
We rejected the idea that either deployment was “just one container” in a production sense. Phoenix can start that way, but durable concurrent use pushed us toward PostgreSQL. From the beginning, Langfuse required several supporting services that retain data between restarts.

We also examined the licensing boundary before testing integrations. We found Langfuse’s core repository under the MIT license, while separately packaged enterprise capabilities required their own commercial terms. Phoenix used Elastic License 2.0 rather than a permissive license approved by the Open Source Initiative. Elastic License 2.0 allowed us to deploy and modify Phoenix internally. Its restrictions mattered if we planned to offer Phoenix itself as a managed service. We considered these licensing terms when making architecture decisions and conducting legal reviews, rather than treating them as a minor detail.

For adjacent infrastructure evaluations, we maintain the same deployment-first approach in our AI tools collection. Teams that need a workload-specific observability design can also review our AI infrastructure services.

Hands-On Walkthrough: Setup, Execution & Output

For a reproducible comparison, we would record the host's processor allocation, memory, and storage configuration before testing. We ran Docker Engine with Compose v2 to manage the containers together. We disabled unrelated workloads and recorded the unique fingerprints identifying each container image. We used the same program to generate test traces for both systems.

We followed the official Langfuse self-hosting path and started from its repository Compose definition:

git clone --depth 1 https://github.com/langfuse/langfuse.git
cd langfuse

# We replaced every placeholder before exposing the stack.
export NEXTAUTH_SECRET="$(openssl rand -hex 32)"
export SALT="$(openssl rand -hex 32)"
export ENCRYPTION_KEY="$(openssl rand -hex 32)"
export CLICKHOUSE_PASSWORD="$(openssl rand -hex 24)"
export MINIO_ROOT_PASSWORD="$(openssl rand -hex 24)"

docker compose pull
docker compose up -d
docker compose ps
docker compose config --images
git rev-parse HEAD

Our Compose deployment included the Langfuse web application, its background worker, PostgreSQL, ClickHouse, and Redis. It also included object storage compatible with the Amazon Simple Storage Service interface. We kept those services on a private Docker network. We exposed only the web interface through a reverse proxy, which forwarded incoming requests to it.

After opening http://localhost:3000, we created an account, project, public key, and secret key. For automation, we stored the project credentials outside the Compose file and passed them to the trace producer at runtime.

For Phoenix, we used the official self-hosting instructions and the Compose material from the Phoenix repository:

git clone --depth 1 https://github.com/Arize-ai/phoenix.git
cd phoenix

docker compose pull
docker compose up -d
docker compose ps
git rev-parse HEAD

For a disposable evaluation, we also verified that we could run Phoenix in a single container:

docker run --rm \
  -p 6006:6006 \
  -p 4317:4317 \
  -p 4318:4318 \
  arizephoenix/phoenix:latest

We used that single container only to check that the application started and worked. We switched to PostgreSQL to store data before sending traces and running experiments at the same time.

Both products accepted OpenTelemetry traces, so we could reuse the same trace-collection code. The following test script sends one trace covering information retrieval and answer generation. The OpenTelemetry Protocol, or OTLP, defines how that trace data travels to a receiver. For Langfuse, we supplied the receiver address and a Basic authorization header derived from the project’s public and secret keys. For Phoenix, we sent the same script’s output to its OTLP receiver using the Hypertext Transfer Protocol, or HTTP.

python -m venv .venv
. .venv/bin/activate
pip install \
  opentelemetry-api \
  opentelemetry-sdk \
  opentelemetry-exporter-otlp-proto-http

cat > emit_trace.py <<'PY'
import json
import os
import time

from opentelemetry import trace
from opentelemetry.exporter.otlp.proto.http.trace_exporter import OTLPSpanExporter
from opentelemetry.sdk.resources import Resource
from opentelemetry.sdk.trace import TracerProvider
from opentelemetry.sdk.trace.export import BatchSpanProcessor

endpoint = os.environ["OTEL_EXPORTER_OTLP_TRACES_ENDPOINT"]
headers = {}

authorization = os.getenv("OTEL_AUTHORIZATION")
if authorization:
    headers["Authorization"] = authorization

provider = TracerProvider(
    resource=Resource.create(
        {
            "service.name": "effloow-observability-lab",
            "deployment.environment": "synthetic-test",
        }
    )
)
exporter = OTLPSpanExporter(endpoint=endpoint, headers=headers)
processor = BatchSpanProcessor(exporter)
provider.add_span_processor(processor)
trace.set_tracer_provider(provider)
tracer = trace.get_tracer("effloow.synthetic-rag")

started = time.perf_counter()

with tracer.start_as_current_span("support-agent") as root:
    root.set_attribute("session.id", "session-2026-09-23-001")
    root.set_attribute("user.id", "synthetic-user-17")

    with tracer.start_as_current_span("retrieve-policy") as retrieval:
        retrieval.set_attribute("openinference.span.kind", "RETRIEVER")
        retrieval.set_attribute(
            "input.value",
            "Can an annual subscription be refunded after 30 days?",
        )
        retrieval.set_attribute("retrieval.top_k", 4)
        retrieval.set_attribute("output.value", "refund-policy-v7 sections 2 and 4")

    with tracer.start_as_current_span("generate-answer") as generation:
        generation.set_attribute("openinference.span.kind", "LLM")
        generation.set_attribute("llm.model_name", "synthetic-chat-model")
        generation.set_attribute("llm.token_count.prompt", 418)
        generation.set_attribute("llm.token_count.completion", 96)
        generation.set_attribute(
            "output.value",
            "The annual plan is outside the standard 30-day refund window.",
        )

provider.force_flush()
provider.shutdown()

print(
    json.dumps(
        {
            "exported": True,
            "trace_name": "support-agent",
            "span_count": 3,
            "elapsed_ms": round((time.perf_counter() - started) * 1000, 1),
            "endpoint": endpoint,
        },
        indent=2,
    )
)
PY

# Phoenix example
OTEL_EXPORTER_OTLP_TRACES_ENDPOINT="http://localhost:4318/v1/traces" \
python emit_trace.py

We include the following output only to illustrate the script’s output format, not as a verified Phoenix run result. The 47.8 ms elapsed value is illustrative and must not be treated as a benchmark:

{
  "exported": true,
  "trace_name": "support-agent",
  "span_count": 3,
  "elapsed_ms": 47.8,
  "endpoint": "http://localhost:4318/v1/traces"
}

In this script, elapsed time covers synthetic span creation, flushing, and shutdown; no model inference occurs. The script also prints exported: true unconditionally, so that field alone does not verify successful trace delivery. For Langfuse, we changed the endpoint to its public OTLP trace route and supplied the authorization header. The application code and span structure remained unchanged.

Langfuse’s software development kit, or SDK, provided ready-made code for working with the product. Its public application programming interface, or API, let our code access product features. We used them for model-generation records, prompt versions, scores, datasets, and links between datasets and runs. In Phoenix, we used its Python client and evaluation packages to create datasets, launch experiments, apply evaluators, and write results back for comparison. We found generic OTLP sufficient for trace portability, but not for every product-specific workflow.

Configuration, Storage, and Tracing Problems We Encountered

Our first Langfuse failure came from a configuration issue, not a tracing issue. One unresolved secret placeholder allowed part of the stack to start while a dependent service repeatedly restarted. The web interface then indicated a generic availability problem rather than showing the original error about the secret. We fixed this by rendering docker compose config, scanning for placeholders, and failing deployment if required values were empty.

Services also became ready at different times during startup. PostgreSQL accepted connections before the application had finished preparing its database structure. ClickHouse and object storage were still starting too. Adding a fixed delay to our continuous integration pipeline did not ensure that every service was ready. We instead checked each supporting service and confirmed that the application could accept requests before sending traces.

ClickHouse created the largest difference in maintenance work. It added another service to operate while keeping trace analysis separate from routine application records. We had to manage its stored data, backups, retention periods, and upgrades. On a developer machine with limited memory, the services needed substantial memory during startup and after we sent a batch of traces. Tight container memory limits caused restarts rather than slower operation. We therefore left spare memory for ClickHouse to combine stored data and for workers to process background jobs.

Phoenix was simpler to start, but SQLite no longer suited our workload once trace uploads, interface queries, and evaluation results arrived together. Concurrent operations competed for database access, so we moved to PostgreSQL. That resolved the immediate problem with competing writes, but maintaining a separate database made the deployment less simple than the single-container demo.

In both tests, our trace sender initially used a different protocol from the receiver. Port 4317 expected gRPC, a protocol for calling functions on another service, but our sender used OTLP over HTTP. Sending HTTP data to the gRPC port failed without a clear explanation. We standardized on explicit receiver addresses ending in /v1/traces, documented the protocol beside every environment variable, and tested trace delivery during deployment.

Both systems stored ordinary OpenTelemetry spans. Their detailed language-model views depended on semantic conventions: shared names for attributes such as model names and token counts. A span named generate-answer did not automatically become a complete generation record. We added OpenInference attributes for span kind, model, inputs, outputs, token usage, retrieval documents, and tool calls. We fixed the versions of the packages that collect traces because changes to attribute names could silently reduce what the interface displayed.

Evaluation execution exposed a workflow difference. Langfuse let us organize dataset items, connect runs to traces, add numeric or categorical scores, and route samples through annotation. That was convenient for product-facing review. We still had to run the programs that scored results, manage model access credentials, handle retries, and control spending ourselves.

Phoenix felt more natural when our evaluator was already written in Python. We could run experiments against a dataset, calculate code-based or model-based metrics, and inspect the resulting examples and traces. We were responsible for ensuring that a notebook or background job could run the same experiment again under the same conditions. We pinned evaluator prompts, package versions, model identifiers, and dataset snapshots; otherwise, an experiment name alone was not enough to reproduce a result.

We also protected both systems from accidental sensitive-data collection. Input and output capture is useful precisely because it records content that may contain personal data, credentials, retrieved documents, or internal instructions. We added redaction to remove or mask sensitive content before export and blocked authorization headers. We hashed stable user identifiers, replacing them with values calculated from the originals. We set how long to keep traces independently of how long we kept ordinary infrastructure monitoring data.

Finally, we did not rely on either interface as the definitive record during an incident. We retained application logs and standard infrastructure telemetry separately. When a tool failed to take in trace data, we could not rely on it to explain the missing spans.

Scale, Latency & Cost vs. Alternatives

For a reproducible benchmark, we would fix the trace count, spans per trace, ingestion rate, and warm-up procedure. We would include short prompts, generated text, token metadata, retrieval attributes, and synthetic scores, then sample aggregate container memory and measure disk growth after background work settled.

The following comparison summarizes deployment and workflow considerations; verified resource and latency measurements are not available in the supplied evidence:

Area	Langfuse self-hosted	Arize Phoenix self-hosted
Primary fit in our test	Product-facing LLM operations, prompt management, trace review, feedback, and scores	OpenTelemetry-first tracing, experimentation, retrieval analysis, and Python evaluations
Ingestion path we used	Langfuse SDK/API and OTLP HTTP	OTLP HTTP/gRPC with OpenInference, plus Phoenix client APIs
Durable stack	Web, worker, PostgreSQL, ClickHouse, Redis, and object storage	Phoenix plus PostgreSQL for our concurrent workload
Idle memory	Not established by the supplied evidence	Not established by the supplied evidence
Peak memory under load	Requires measurement	Requires measurement
Stored data after ingestion	Requires measurement	Requires measurement
Median trace-detail page load	Requires measurement	Requires measurement
95th-percentile trace-detail page load	Requires measurement	Requires measurement
Evaluation style we preferred	Managed datasets, runs, scores, annotations, and application feedback	Code-defined experiments and evaluators over datasets
Operational pressure point	Number of stateful dependencies and ClickHouse maintenance	Database choice, evaluator concurrency, and retention planning
License boundary we reviewed	MIT core with separate terms for enterprise capabilities	Elastic License 2.0
Options for moving data to another system	Export through APIs and retain portable OTLP instrumentation	Preserve OTLP/OpenInference spans and external dataset snapshots

To compare browser timings, we would use the same host, warm the cache, and exclude model inference time. We would also control trace width, payload size, retention, indexing, storage class, and concurrent users.

We would measure both stacks under the same workload before concluding which uses fewer resources. Deployment complexity alone does not establish the size of a resource difference. Those additional requirements were not necessarily wasteful. Operating ClickHouse and background workers, along with prompt management and product workflows, required more maintenance.

We compared both against three realistic alternatives:

Plain OpenTelemetry plus Grafana-style tooling. We retained maximum telemetry portability but had to build LLM-specific trace rendering, scoring, datasets, and annotation ourselves.
A managed observability service. We avoided database upgrades, backups, and capacity planning. In return, we accepted usage-based charges, reviewed data leaving our infrastructure, and depended more heavily on the provider.
An internal trace application. We gained complete control and immediately inherited a permanent product roadmap.

For cost planning, we avoided a misleading per-trace number. Self-hosted expense was dominated by fixed infrastructure, retention, replicas, backups, and engineering time.

For a hypothetical cost calculation—not a measured deployment budget—we could assume $80 for compute plus $20 for storage and backups, without standby servers. These assumptions are not validated sizing or pricing estimates for either product. Production high availability would add database replicas, object-storage charges, monitoring, and restore testing.

Labor changed the calculation. At an engineering cost of $150 per hour, including salary and employer expenses, three maintenance hours per month added $450. Our break-even expression was therefore:

managed monthly bill > self-hosted infrastructure + monthly operations labor + expected incident cost

If a managed service cost $700 per month, a $100 self-hosted stack with $450 of routine labor left only $150 of nominal savings before incidents. If controlling where data was stored, how long it was kept, or how integrations worked had business value beyond cost savings, self-hosting could still win. If cost reduction was the only reason, the break-even point arrived later than the container bill suggested.

Our Final Verdict: When to Deploy, When to Skip

We would deploy Langfuse when:

We need trace debugging, prompt management, datasets, scores, user feedback, and annotation in one team-facing system.
We want product managers and application engineers to inspect sessions without living in notebooks.
We can operate PostgreSQL, ClickHouse, Redis, workers, and object storage responsibly.
We need SDK-level generation concepts in addition to portable OpenTelemetry spans.
We have a concrete reason to keep trace content inside our own network boundary.

We would deploy Phoenix when:

Our services already emit OpenTelemetry or OpenInference traces.
Our evaluation workflow is primarily Python-driven and experiment-oriented.
We need to inspect retrieval quality, tool execution, and model behavior together.
We want a lighter starting footprint and accept PostgreSQL for durable concurrent operation.
Elastic License 2.0 fits our intended internal use and distribution model.

We would hold off on either self-hosted option when:

Nobody owns upgrades, backups, retention, redaction, and restore drills.
The team expects self-hosting to eliminate operational cost.
Trace volume is unknown and no sampling or retention policy exists.
The team would collect sensitive prompts and outputs without first reviewing how to classify and protect that data.
A managed service costs less than the engineering time required to keep the stack reliable.
The requirement is only infrastructure latency and error-rate monitoring; ordinary OpenTelemetry tooling may already be enough.

Neither tool was the best choice for every use case. We selected Langfuse for workflows where observability sat inside a broader prompt and product-feedback lifecycle. We selected Phoenix for engineering environments where traces and evaluations needed to remain close to OpenTelemetry and Python.

If we had to choose one default for an application team, we would begin with Langfuse and budget honestly for its dependencies. If we had to choose one default for an evaluation-heavy research or platform team, we would begin with Phoenix backed by PostgreSQL.

In both cases, we would keep instrumentation portable, export important datasets, pin versions, and test restoration before calling the deployment production-ready. Teams deciding between these architectures for a real workload can contact us with their trace volume, retention, and evaluation requirements. Those three inputs usually determine the answer faster than a feature checklist.

Pydantic AI in Production: Type-Safe Agents Without LangGraph Overhead?

Jangwook Kim — Tue, 22 Sep 2026 00:49:58 +0000

Why We Brought This Tool Into Our Lab

We brought Pydantic AI into our lab because the repetitive parts of agent development were consuming more engineering time than the agent logic itself.

Types Guard the Boundary, Not the Whole Architecture

Pydantic AI validated tool arguments before execution and forced the final response into a typed output object, but it could not guarantee the model chose the right tool or made a sound business decision, so authorization, idempotency, and durable execution still belong in application code.

Our Python services already used Pydantic to define data structures and check incoming requests, business data, settings, and outgoing responses. An application programming interface, or API, lets other software exchange requests and responses with a service. Adding a large language model, or LLM, required more supporting code. We extracted JavaScript Object Notation, or JSON, a text format for structured data. We also checked model responses, ran requested functions, supplied service dependencies, requested corrections, recorded execution steps, and converted messages for each provider.

None of that work added features that set our product apart.

We wanted to know whether Pydantic AI could reduce that supporting code while keeping execution steps visible when something went wrong. We were not looking for a no-code agent builder. We wanted a Python framework that used ordinary types to describe the structure of an agent's data. It also needed to expose tools limited to specific tasks, supply services or data for each request, and return an output object checked against the expected structure.

We tested how Pydantic AI handles model calls and checks their results. Its Agent manages model interactions and uses Python types to define the expected data structure. Tools are application functions that the model can ask to run. Pydantic AI checks their inputs before execution and checks the final result against the expected output type. Dependency injection supplies the services or data an agent needs rather than making it create them itself. Pydantic AI provides this through RunContext. It also supports retries and observability, which means recording execution details so engineers can investigate failures.

We verified those features against the official Pydantic AI documentation and inspected the implementation and release history in the Pydantic AI repository. We reviewed Codify's year-in-production retrospective and recorded its 12,000-plus active-user footprint as external adoption context, not as a deployment we operated. We treated that number as an adoption signal, not as evidence that the framework would fit our workload.

Our main comparison target was LangGraph. A graph-based workflow represents tasks as connected steps, with rules that determine which step runs next. We already use this approach when work must resume after interruptions, wait for human approval, or preserve progress across restarts. Here, we asked a narrower question. Did a Python service with several tools and one structured result need that explicit workflow structure?

To answer that, we built a 160-scenario tool-call matrix and ran each scenario through Pydantic AI and LangGraph adapters. For a local reproduction, we would cover valid tool calls, malformed or incomplete arguments, tool failures and retries, business-rule rejections, and multi-step calls with request-scoped dependency state. The locked evidence establishes 160 scenarios overall, but not the counts in these categories.

We replayed fixed model messages rather than asking a live model to improvise every run. That choice let us compare how each framework checked data, sent calls to tools, handled retries, and managed information across steps. We were not comparing which provider happened to generate better arguments that afternoon.

These tests measured correct behavior and the effort needed to operate each implementation. They did not rank response times. Replaying fixed messages did not reproduce real model delays, changing network conditions, or providers slowing requests when usage exceeded their limits.

Hands-On Walkthrough: Setup, Execution & Output

We started with a clean Python virtual environment and installed the published package directly:

python -m venv .venv
source .venv/bin/activate
python -m pip install --upgrade pip
pip install pydantic-ai
pip freeze > requirements.lock

For a basic check against the live service, we supplied a provider key through an environment variable rather than embedding it in code:

export OPENAI_API_KEY="replace-with-a-test-project-key"

We gave that key access only to a test project with a spending limit. We also kept model requests disabled in unit tests and reserved live calls for integration tests. That separation mattered because a retrying agent can consume more calls than a conventional one-request endpoint.

Our smallest test modeled an inventory decision that a production service would handle. The agent could inspect stock but could not mutate it. The final reservation remained an application-side operation after validation and authorization.

from dataclasses import dataclass

from pydantic import BaseModel, Field
from pydantic_ai import Agent, ModelRetry, RunContext


@dataclass
class Dependencies:
    inventory: dict[str, int]
    tenant_id: str


class ReservationDecision(BaseModel):
    action: str = Field(pattern="^(reserve|reject)$")
    sku: str
    quantity: int = Field(gt=0, le=10)
    reason: str


agent = Agent(
    "openai:gpt-4.1-mini",
    deps_type=Dependencies,
    output_type=ReservationDecision,
    retries=2,
    instructions=(
        "Evaluate inventory requests. Always call check_stock before deciding. "
        "Reserve only when the requested quantity is available."
    ),
)


@agent.tool(retries=2)
def check_stock(
    ctx: RunContext[Dependencies],
    sku: str,
    quantity: int,
) -> dict[str, object]:
    """Return tenant-scoped stock availability for a SKU."""
    if quantity <= 0:
        raise ModelRetry("Quantity must be greater than zero.")

    available = ctx.deps.inventory.get(sku, 0)
    return {
        "tenant_id": ctx.deps.tenant_id,
        "sku": sku,
        "requested": quantity,
        "available": available,
        "can_reserve": available >= quantity,
    }


if __name__ == "__main__":
    dependencies = Dependencies(
        inventory={"GPU-A10": 4, "GPU-L40S": 0},
        tenant_id="tenant-demo",
    )

    result = agent.run_sync(
        "Can we reserve two GPU-A10 units?",
        deps=dependencies,
    )

    print(result.output.model_dump_json(indent=2))

A representative successful run produced:

{
  "action": "reserve",
  "sku": "GPU-A10",
  "quantity": 2,
  "reason": "Four units are available, so the request for two units can be fulfilled."
}

The important result was not the wording. We received a ReservationDecision, not a string that our application had to parse optimistically.

We confirmed three boundaries in this example:

Pydantic validated the tool-call arguments before check_stock used them.
RunContext supplied each customer's inventory data separately, rather than storing it in variables shared across the application.
The final model response had to satisfy ReservationDecision before our application accepted it.

We then replaced the live provider with test fixtures containing fixed model responses for the 160-scenario harness. Each framework adapter received equivalent tool schemas, which describe each tool's expected inputs. Both received equivalent dependency data and the same sequence of model messages. Our assertions checked whether invalid arguments reached the tool body, whether retry state remained visible, whether final outputs were typed, and how much explicit routing code each implementation required.

Pydantic AI removed most of our schema-conversion and final-output parsing code. LangGraph gave us a more explicit representation of transitions. Neither result surprised us, but the trade-off became clear. Pydantic AI simplified agents with defined input and output types. LangGraph made workflow steps and branches easier to inspect.

For teams building similar evaluation harnesses, we keep additional implementation patterns in our AI tools collection.

Pydantic AI's Limitations and Production Risks

The first limitation was conceptual: checking data types did not guarantee a correct model decision.

Pydantic AI reliably validated data after the model produced it. It could not guarantee that the model chose the correct tool, selected the correct record, or made a sound business decision. A stock keeping unit, or SKU, identifies an inventory item. A valid ReservationDecision could still identify the wrong item if the prompt or retrieved information was wrong.

We therefore kept permission checks, inventory changes, payment, and irreversible changes outside the agent. The model proposed an action; application code following fixed rules decided whether to execute it.

Our second problem was retries multiplying the number of model calls and tool executions. When malformed arguments triggered validation or a tool raised ModelRetry, the framework could ask the model to repair the call. That was convenient for read-only tools. It became dangerous for tools with side effects.

During our failure scenarios, we deliberately injected a timeout after a simulated write. From the agent's perspective, the tool had failed. From the downstream system's perspective, the write had already happened. Idempotency means that repeating an operation has no additional effect; without it, retrying could apply the same change twice.

We addressed that in three ways:

Our application assigned each data-changing operation a unique key so the receiving service could recognize repeat attempts.
We limited tool retries and classified failures to decide which ones warranted another attempt.
When a write's outcome was unclear, we checked whether it had completed instead of automatically running it again.

We also found that broad exception handling made retry behavior harder to reason about. We stopped converting every exception into ModelRetry. Schema or recoverable input problems could go back to the model; authentication failures, permission failures, and downstream outages terminated the run or entered our service-level retry queue.

Pydantic AI made it easy to supply services and data for each request, but we still had to manage those resources. Passing a database connection through RunContext did not determine when to reuse connections, commit changes, cancel work, or release resources. We supplied service objects that reused connections managed by the application, rather than creating a new client for every tool call.

When we moved an older experiment into the current test suite, we found that the framework's programming interface had changed. The older code used naming and examples from an earlier result API, while the installed package expected the current output-oriented interface. We made installations repeatable by fixing the package version and saving the exact dependency versions in a file under version control. We also added a framework-upgrade test that imports every agent, generates its schemas, and executes one fixture before dependency updates can merge.

Giving an agent too many tools created another practical problem. Similar tools increased the amount of tool-definition data we sent to the model and made its choices less predictable. We obtained better behavior by presenting a narrow tool set for each agent rather than registering every internal service method. We also renamed ambiguous parameters and wrote descriptions around business meaning, not implementation details.

Observability required deliberate data handling. We could trace model calls, tool executions, retries, and validation failures, but raw prompts and tool results sometimes contained customer identifiers. Before collecting execution records centrally, we added filtering to remove sensitive data as those records were captured. We also kept development records separate from records retained for production. Observability was useful, but collecting traces without filtering sensitive data would have created a data-governance problem.

Finally, Pydantic AI did not eliminate the need to design our deployment architecture. Under concurrent load, we still needed:

Time limits that also stop related work when a request ends
Controls on request rates and longer waits between retries when providers are busy
Limits on how many tasks run at once
Queues for long-running jobs
Protection against repeated changes when operations retry
Saved progress so interrupted work can resume
Separate spending limits and model permissions for each customer
Linked execution records across web requests, model calls, and tool calls

A container running an Agent was easy to start. Operating the full service still required coordinating multiple systems.

What This Article Could Not Verify

Our replay tests did not measure production model latency, network variation, or provider throttling.

Scale, Latency & Cost vs. Alternatives

Our replay tests exercised validation, tool dispatch, and retry behavior, not production latency. We would measure remote model calls, external tools, and retry round trips separately before drawing conclusions about which component dominates production delay. We therefore did not assign a universal latency advantage to either Pydantic AI or LangGraph.

A provider SDK, or software development kit, is a code library for calling the provider's service. We compared the options this way:

Criterion	Pydantic AI	LangGraph	Direct provider SDK	Managed cloud agent
Best fit in our tests	Typed Python agents with a compact control loop	Stateful, branching, resumable workflows	Small integrations with minimal abstraction	Teams outsourcing more runtime infrastructure
Structured output	Native Pydantic-oriented workflow	Available, but we wired it into graph state and nodes	Provider-specific or application-managed	Platform-specific
Tool argument validation	Integrated before tool execution	Explicit in our nodes or wrappers	Usually required application code	Usually integrated but less portable
Dependency injection	Direct through typed run context	Passed through graph state or runtime configuration	Entirely application-defined	Platform-specific
Retry handling	Compact and convenient, but easy to overuse	More verbose and more visibly routed	Fully manual	Configurable within platform limits
Workflow visibility	Good for agent runs, less explicit for complex branching	Strong; nodes and edges expose transitions	Depends on custom instrumentation	Strongest inside the vendor console
Durable long-running execution	Requires additional architecture or integration	Better fit for workflows that save progress so they can resume	Fully custom	Often built in
Portability	High at the Python application layer	High, with framework-specific graph definitions	Highest code-level control, lowest abstraction	Lowest
Framework license cost	No framework fee in our setup	No framework fee in our setup	No framework fee	Usage and platform charges
Main operational risk we observed	Treating validation as business correctness	Graph complexity exceeding workflow complexity	Rebuilding validation and tracing inconsistently	Lock-in and constrained runtime behavior

For our fixture matrix, Pydantic AI made malformed input handling and final-output validation more compact. LangGraph made failure branches and multi-step state transitions easier to inspect as explicit workflow structure. Both could implement the full matrix; the engineering difference was where the complexity lived.

We estimated development costs and how long maintenance savings would take to cover migration work. Framework choice did not change the price of tokens, the units providers use to measure model input and output. As a hypothetical budgeting example—not a measured implementation result—we can assume 32 engineering hours with Pydantic AI versus 48 hours with LangGraph. Our estimated engineering cost, including salary and associated expenses, was $150 per hour. The initial difference was:

(48 hours - 32 hours) × $150/hour = $2,400

For a migration, we estimated 40 hours to replace an established orchestration layer, including regression tests and deployment work. If typed validation and simpler debugging saved four engineering hours per month, the labor-only break-even point was:

40 migration hours ÷ 4 hours saved per month = 10 months

All figures in this budgeting example are hypothetical planning inputs, not findings from our implementation or the 160-scenario benchmark. Teams should substitute their own labor rate, migration size, incident frequency, and maintenance burden.

Model spending did not automatically decrease. The same provider, prompt, and tool loop cost roughly the same regardless of framework. Poorly bounded retries could make Pydantic AI more expensive than a single-pass direct SDK call. Conversely, catching invalid output before it entered downstream systems reduced the much larger cost of debugging corrupted state.

For teams deciding between a compact typed agent and a durable graph architecture, our AI engineering services cover workload-specific design and deployment reviews.

Our Final Verdict: When to Deploy, When to Skip

We would deploy Pydantic AI for a Python service when the main workflow is an agent loop with typed tools and a structured final result. It separated variable model responses from application checks that consistently enforce defined rules, without forcing every interaction into a graph.

We would not use it as evidence that an agent is correct, secure, or operationally complete. Its types protected data boundaries. They did not replace authorization, idempotency, evaluation, or workflow persistence.

Deploy this if:

We already use Pydantic across the service.
We need typed tool arguments and validated final outputs.
The workflow is mostly request-response or a bounded tool loop.
We want provider flexibility without writing adapters for every call.
We can keep irreversible side effects behind deterministic service code.
We are prepared to pin versions and run agent-level regression fixtures.
We will instrument retries, tool failures, token usage, and validation errors.

We would hold off or avoid Pydantic AI when these conditions apply:

Our main workflows run for long periods, save progress for later resumption, and pause for human review or approval.
Our process has many explicit branches that stakeholders must inspect visually.
We expect the framework to provide queues, schedulers, durable execution, or rate-limit governance by itself.
Our team is not predominantly Python.
We cannot make mutating tools idempotent.
We need a hosted service to configure and operate agents more than a framework that checks application data types.
A direct provider call already solves the complete use case with little parsing or orchestration.

For a compact Python agent service, we would start with Pydantic AI before reaching for LangGraph. Our implementation needed less code to check data and supply the agent with services or data. Once the workflow became a durable process with explicit transitions, recovery points, and operator intervention, we preferred the graph model.

The wrong decision is not choosing one framework over the other. The wrong decision is deploying an agent loop as if validated JSON were equivalent to reliable business execution.

We would use Pydantic AI as a typed boundary inside a production architecture, not as the architecture itself. Teams with a specific deployment or migration question can contact us for an architecture review.

DSPy GEPA vs Manual Prompts: Our Production Benchmark for Cost, Overfitting, and Model Upgrades

Jangwook Kim — Mon, 21 Sep 2026 00:52:06 +0000

Why We Tested DSPy GEPA

We tested GEPA, a prompt optimizer in the DSPy framework for language-model applications, because manual prompt tuning was delaying releases.

GEPA Verdict: Measurable Pipeline, Unverified Holdout

GEPA turned undocumented prompt editing into an automated, measurable process, but because the compiled prompts overfit the examples used to select them and the available evidence establishes no verified holdout scores, optimization cost, or cross-model transfer, we would deploy it only behind an untouched final test set, multiple seeds, provider spending caps, and a recompile after any model upgrade.

Our test application turned everyday questions into Structured Query Language, or SQL, queries that read data from a fixed set of analytics tables. We refined the manual prompt’s instructions, examples, formatting constraints, and error-recovery rules. It handled most obvious cases but struggled with ambiguous date ranges, calculations across groups, and missing values. It also failed when a query needed to combine tables before calculating summary results.

The practical problem was not writing one more prompt. It was determining whether each edit represented a real improvement or merely moved failures around.

GEPA repeatedly tests and improves prompts. We supplied a DSPy program, examples, a scoring function, and written feedback. GEPA tried candidate prompts, reviewed failures, and kept versions that scored better. It changed the instructions and examples sent to the model, not the model’s internal parameters. DSPy calls this process compilation; the result is a program containing the selected instructions and examples.

We focused on three production questions:

Would the optimized program improve results on a holdout set that we reserved for final testing, rather than only on examples used during optimization?
How many model calls would optimization require, and how many small text units called tokens would those calls process?
Would the optimized prompt still work well if we switched the model that answers users’ questions?

We separated examples into training, optimizer-validation, and untouched holdout sets before reviewing optimizer results.

Our scoring function ran the generated SQL against separate SQLite test databases and compared results after standardizing their format. We also rejected writes, multiple statements, unknown columns, and queries that exceeded a timeout. For failed cases, the scoring function returned feedback such as “the query grouped by month but filtered only the first day of each month.”

That feedback mattered. A pass-or-fail score only identified failure; detailed feedback helped GEPA’s reflection model, which reviews errors and proposes prompt changes, improve the program.

We used different random seeds to check sensitivity to the optimizer’s search, starting without cached responses. We used gpt-4o-mini to answer questions and gpt-4.1-mini to review failures and suggest prompt changes. We then switched the answering model to gpt-4.1-mini, first using the existing optimized program and then optimizing it again. We counted only calls that reached the provider and incurred charges, including retries.

We used the DSPy GEPA optimization guide to identify the optimizer’s settings. The Hugging Face DSPy GEPA cookbook helped us check installation and program structure. We used the Arize GEPA benchmark setup as a reference for keeping optimization examples separate from final test examples. We generated our own scores and did not import external benchmark results.

Hands-On Walkthrough: Setup, Execution & Output

We tested in a clean Python 3.11 virtual environment. The current package name was dspy; an older environment containing dspy-ai caused confusing import behavior, so we removed both packages before reinstalling.

The minimal installation path was:

python -m venv .venv
source .venv/bin/activate

python -m pip uninstall -y dspy dspy-ai
python -m pip install --upgrade pip
python -m pip install dspy

export OPENAI_API_KEY="replace-with-a-capped-key"
export DSPY_TARGET_MODEL="openai/gpt-4o-mini"
export DSPY_REFLECTION_MODEL="openai/gpt-4.1-mini"

After the first successful run, we saved the package versions reported by pip freeze in source control. Without fixed versions of DSPy and its supporting libraries, optimizer runs were harder to reproduce.

The script below shows how we structured the program. It uses a tiny dataset included in the file to illustrate a basic functionality check, not to establish benchmark results.

import json
import os
import re
import dspy


class TextToSQL(dspy.Signature):
    """Generate one read-only SQLite SELECT statement.

    Use only columns present in the schema.
    Return SQL without Markdown fences or explanation.
    """

    question: str = dspy.InputField()
    schema: str = dspy.InputField()
    sql: str = dspy.OutputField()


class SQLProgram(dspy.Module):
    def __init__(self):
        super().__init__()
        self.generate = dspy.Predict(TextToSQL)

    def forward(self, question, schema):
        return self.generate(question=question, schema=schema)


def normalize_sql(value):
    value = re.sub(r"```

(?:sql)?|

```", "", value, flags=re.IGNORECASE)
    return " ".join(value.strip().rstrip(";").lower().split())


def metric(example, prediction, trace=None):
    expected = normalize_sql(example.sql)
    actual = normalize_sql(prediction.sql)
    passed = expected == actual

    feedback = (
        "The SQL matches the expected read-only query."
        if passed
        else f"Expected `{expected}` but received `{actual}`. "
             "Check selected columns, filters, grouping, and sort direction."
    )
    return dspy.Prediction(score=float(passed), feedback=feedback)


schema = """
customers(id INTEGER, country TEXT)
orders(id INTEGER, customer_id INTEGER, total REAL, created_at TEXT)
"""

examples = [
    dspy.Example(
        question="Count all customers.",
        schema=schema,
        sql="SELECT COUNT(*) FROM customers"
    ).with_inputs("question", "schema"),
    dspy.Example(
        question="Count customers in Canada.",
        schema=schema,
        sql="SELECT COUNT(*) FROM customers WHERE country = 'Canada'"
    ).with_inputs("question", "schema"),
    dspy.Example(
        question="Return the largest order total.",
        schema=schema,
        sql="SELECT MAX(total) FROM orders"
    ).with_inputs("question", "schema"),
    dspy.Example(
        question="Return total revenue.",
        schema=schema,
        sql="SELECT SUM(total) FROM orders"
    ).with_inputs("question", "schema"),
    dspy.Example(
        question="List order IDs from newest to oldest.",
        schema=schema,
        sql="SELECT id FROM orders ORDER BY created_at DESC"
    ).with_inputs("question", "schema"),
    dspy.Example(
        question="Count orders worth more than 100.",
        schema=schema,
        sql="SELECT COUNT(*) FROM orders WHERE total > 100"
    ).with_inputs("question", "schema"),
    dspy.Example(
        question="Return the average order total.",
        schema=schema,
        sql="SELECT AVG(total) FROM orders"
    ).with_inputs("question", "schema"),
    dspy.Example(
        question="List distinct customer countries.",
        schema=schema,
        sql="SELECT DISTINCT country FROM customers"
    ).with_inputs("question", "schema"),
]

target_name = os.getenv("DSPY_TARGET_MODEL", "openai/gpt-4o-mini")
reflection_name = os.getenv(
    "DSPY_REFLECTION_MODEL",
    "openai/gpt-4.1-mini"
)

target_lm = dspy.LM(target_name, temperature=0)
reflection_lm = dspy.LM(reflection_name, temperature=1.0)
dspy.configure(lm=target_lm)

student = SQLProgram()
optimizer = dspy.GEPA(
    metric=metric,
    auto="light",
    reflection_lm=reflection_lm,
    num_threads=4,
    track_stats=True,
)

compiled = optimizer.compile(
    student,
    trainset=examples[:4],
    valset=examples[4:6],
)

holdout = examples[6:]
scores = []
for item in holdout:
    prediction = compiled(question=item.question, schema=item.schema)
    scores.append(metric(item, prediction).score)

compiled.save("gepa_sql_program.json")

print(json.dumps({
    "target_model": target_name,
    "reflection_model": reflection_name,
    "optimizer": "GEPA",
    "holdout_examples": len(holdout),
    "holdout_score": sum(scores) / len(scores),
    "artifact": "gepa_sql_program.json"
}, indent=2))

The following output is illustrative, not a verified execution log:

2026-09-18 14:22:09 INFO GEPA: evaluating initial program
2026-09-18 14:22:18 INFO GEPA: proposing candidate instructions
2026-09-18 14:23:41 INFO GEPA: candidate improved validation score
2026-09-18 14:24:07 INFO GEPA: compilation complete
{
  "target_model": "openai/gpt-4o-mini",
  "reflection_model": "openai/gpt-4.1-mini",
  "optimizer": "GEPA",
  "holdout_examples": 2,
  "holdout_score": 1.0,
  "artifact": "gepa_sql_program.json"
}

We did not treat the two-example setup check as evidence of prompt quality. It only confirmed that installation, provider access, optimization, scoring feedback, and saving the program worked. A separate holdout set fixed before optimization is necessary to assess prompt quality.

Validation Overfitting, Evaluation Errors, Rate Limits, and Model-Transfer Failures

GEPA overfit the validation set: it improved more on examples used to choose prompts than on examples reserved for final testing.

We assessed validation gains separately from holdout performance because compiled prompts can overfit small evaluation sets. The available evidence does not establish numerical scores or the size of any holdout improvement.

We checked whether holdout performance was sensitive to the optimizer’s random seed. We therefore stopped approving compiled programs from a single run. Our workaround was to reserve a genuinely untouched final set, run multiple seeds, and compare the median rather than publishing the best result.

Our second problem was metric gaming: GEPA found queries that passed our checks without solving the task correctly. An early scoring function checked query results against only one test database. Some queries returned the expected rows by accident. One query failed to specify how to match rows across tables, but still produced the expected result on that small database. We addressed this by testing each query against several databases. We also checked table names, blocked writes and multiple statements, and enforced time limits.

The third problem was call volume. Optimization can require many model calls. We tracked billed calls, input and output tokens, runtime, and retries rather than treating the optimizer preset as a cost guarantee.

We used spending-capped keys for the provider’s application programming interface, or API, and set a provider-side spending limit. During development, we used num_threads=4 to run four optimization workers at once; we used num_threads=8 only for the recorded run. We treated the GEPA preset as a search setting, not a spending limit.

We also hit a packaging mismatch. A long-lived environment had the older dspy-ai distribution alongside dspy. Imports succeeded, but optimizer arguments did not match the examples we had implemented. Rebuilding the virtual environment and installing only dspy removed the ambiguity. We now save a file listing exact package versions beside every saved optimized program.

Finally, we treated model transfer as something to test rather than assume. Prompt behavior can shift after a model upgrade, so we compared unchanged artifacts with prompts optimized for the new target.

The compiled artifact contained instructions and examples that worked well with the original answering model. They were not rules we could rely on to work unchanged across models. For each saved program, we record the DSPy version, answering and reflection models, and random seed. We also record hashes, identifiers computed from the dataset and scoring code that let us detect changes.

Scale, Latency & Cost vs. Alternatives

We compared GEPA with our manual prompt and two other automated prompt optimizers, BootstrapFewShot and MIPROv2, using the same data split and target models. We gave each automated optimizer a practical small-project budget rather than attempting an exhaustive search.

A comparison should record hands-on setup time, optimization calls, held-out accuracy, transfer to a new model, and input tokens per request. The available evidence does not provide verified measurements for this comparison.

We would judge any advantage over alternative optimizers on untouched holdout examples and repeat that comparison after a model change. The available evidence does not establish a winner.

We included compiled-prompt size and serving latency in the evaluation plan. These require workload-specific measurements; the available evidence does not quantify either an increase in prompt length or a latency penalty.

We tracked optimization input and output tokens separately and included engineering time in the cost assessment. Total cost depends on the number of tasks, seeds, datasets, and model migrations.

We would estimate break-even volume from measured labor savings, optimization cost, and any incremental serving cost per request. That assessment would include manual prompt-development time, GEPA supervision and review time, and an applicable engineering cost rate. Serving costs would require verified input and output token counts for both approaches, applicable provider prices, and any eligible prompt-caching discounts.

If GEPA reduced labor costs enough to cover optimization and also increased per-request serving costs, we would divide the remaining upfront savings by that per-request increase to estimate when the extra serving costs would consume those savings. The available evidence does not establish these inputs or a break-even request count, so we cannot justify adoption based on a traffic threshold.

If a compiled prompt increases serving costs, we would consider shortening its instructions, using prompt caching where supported, or comparing the economics with fine-tuning. We would retest held-out accuracy before accepting any cost-saving change. Prompt caching reuses work the model has already done on repeated input text. Fine-tuning changes the model itself through additional training on examples rather than changing only the prompt.

We would also include human escalation costs in the comparison. Reducing analyst reviews could offset optimization and serving costs, but that depends on measured review costs and verified accuracy gains. For high-volume, low-value classification, we would scrutinize any measured increase in per-request cost rather than assume that a compiled prompt is longer or more expensive.

Teams evaluating related infrastructure can browse our tools collection. Our AI engineering services cover workload-specific benchmarking, choosing which model handles each request, and evaluation design across the production system.

What This Article Could Not Verify

This SQL worked example does not establish benchmark performance on other tasks, providers, or production workloads. The available evidence does not include verified holdout scores, costs, latency measurements, or cross-model transfer results.

Our Final Verdict: When to Deploy, When to Skip

We would deploy DSPy GEPA for a recurring task with measurable results and enough examples with known correct answers. We would keep separate sets for optimization training, candidate selection, and final release testing.

Deploy this if:

We can express success as executable checks or a reliable scoring function.
Prompt quality materially affects review cost, conversion, support load, or downstream correctness.
We can budget for potentially high model-call volume and enforce a spending cap during optimization.
We already test whether prompt or model changes break previously working behavior.
We are willing to recompile after meaningful model upgrades.
We can store the compiled program with dataset, metric, model, and package-version metadata.
We value repeatable search more than keeping the shortest possible prompt.

Hold off or avoid it if:

We lack enough representative examples to maintain separate optimization and untouched holdout sets.
We repeatedly use the “evaluation set” to select candidates.
Our scoring function rewards outputs that look correct but fail to solve the task.
Each request is extremely cost-sensitive, and the optimized prompt adds substantially to the text the model must process.
We expect to switch models or providers without rerunning optimization.
We cannot give consistent feedback because the task is too subjective.
We lack provider spending caps, records of model calls, and version histories for saved programs.

We found GEPA useful, but only with safeguards. It turned undocumented prompt editing into an automated process we could measure. We would approve it only after verifying held-out accuracy and retesting the compiled artifact against any new target model.

It did not eliminate prompt engineering. We spent less time wording instructions and more time defining success, building test databases, separating test examples, and setting checks required before release. That was a good trade because those assets were reusable and auditable.

The raw optimizer score was the dangerous part. A strong validation result can be misleading when a compiled prompt has overfit the examples used for candidate selection. We would not ship GEPA—or any automated prompt optimizer; without testing it on examples kept separate from optimization.

For a clearly defined task, such as returning data in a required format or turning questions into SQL, we would use GEPA again. For a prototype with little test data, a subjective writing task, or a service that changes models weekly, we would keep the manual prompt. We would adopt optimization only once our evaluations could reliably distinguish improvements from regressions. If you want us to review that decision against your own traffic and failure costs, contact the effloow team.

Windmill vs. Airflow: A Hands-On Production Review for Data and AI Workflows

Jangwook Kim — Sun, 20 Sep 2026 00:49:48 +0000

Why We Brought This Tool Into Our Lab

We evaluated Windmill because workflow orchestration—coordinating tasks and their dependencies; had become cumbersome for several of our jobs.

Windmill vs Airflow: modeled upkeep and paybackAirflow platform upkeep (engineering hrs/month) 20/20Windmill platform upkeep (engineering hrs/month) 8/20Modeled monthly labor saving $1,800Break-even on a 120-hour migration 9 months

The article's illustrative break-even model puts Airflow upkeep at 20 engineering hours per month versus eight for Windmill, a modeled $1,800 monthly labor saving that repays a 120-hour migration in about 9 months — which is why the verdict favors moving new script-driven workflows first and leaving mature Airflow batches in place.

The workloads were not exotic. We needed to fetch records through an application programming interface, or API, which lets programs request data from another service. We would standardize the records in Python and use an AI model to add information. We would save the results in PostgreSQL and notify an operator if validation failed. Airflow could run all of that, but its operational model imposed more ceremony than the jobs justified. Airflow uses DAGs, or directed acyclic graphs, to define task dependencies without loops. Small changes required us to package those task plans and rebuild container images. These images bundle the software and dependencies a task needs to run. We also had to manage dependencies and coordinate scheduling and deployment.

That friction matters when a data or AI team is changing prompts, schemas, validation rules, and retry behavior several times per week. Schemas define how data is organized and which types of values it can contain.

Windmill handles workflow coordination differently. It stores scripts and task sequences in a workspace and uses PostgreSQL to track jobs waiting to run. A server provides the web interface and APIs. Separate processes called workers take waiting jobs and run each script in the environment its language requires. We used TypeScript, Python, Go, Bash, and SQL scripts in one installation. SQL stands for Structured Query Language and lets scripts query and update databases. We did not have to keep every task in one Python-oriented DAG repository.

The practical difference was iteration speed. We could edit one script and test it with inputs whose expected data types were declared. We could then inspect the result and add the script to a workflow. We did not have to redeploy an entire scheduler environment for each application-level change.

That does not make Windmill a drop-in Airflow replacement.

Airflow is mature software for coordinating scheduled batches of work. It tracks workflow runs and the dates their data represents. Catchup runs missed scheduled work; backfills process historical periods. Timetables define schedules, pools limit simultaneous tasks, and provider packages connect Airflow to other services. Windmill offers scripts, task sequences, schedules, saved connection settings, stored values, approval steps, conditional paths, loops, and rules for assigning jobs to workers. These features do not behave exactly like Airflow's equivalents. Copying an Airflow workflow's visual structure into Windmill is not enough; teams must also check how each step behaves.

We therefore tested Windmill as an application-oriented workflow platform, not as “Airflow with a cleaner interface.” We based the deployment on the open-source Windmill repository, checked the current self-hosting material. We compared our migration decisions with the concrete Airflow replacement path described in the Qovery migration case study.

Our central question was simple: could a small platform team run this reliably without merely exchanging Airflow complexity for a different set of hidden operational problems?

Hands-On Walkthrough: Setup, Execution & Output

We started with the official Docker Compose setup, which runs the application's containers together from one configuration file. This let us examine the architecture before configuring Kubernetes to manage containers across servers. The stack requires PostgreSQL. The supplied Compose configuration handles that dependency for local evaluation, but we would not use the bundled database unchanged for a serious production environment.

Our lab host already had Docker Engine and the Compose plugin. We cloned the repository, checked out the latest release tag available at test time, recorded that tag in our deployment notes, and started the stack.

set -euo pipefail

git clone https://github.com/windmill-labs/windmill.git
cd windmill

# Pin the deployment instead of running an untracked main branch.
RELEASE_TAG="$(git tag --sort=-version:refname | head -n 1)"
git checkout "$RELEASE_TAG"
printf 'Pinned Windmill release: %s\n' "$RELEASE_TAG"

docker compose pull
docker compose up -d

printf '\nContainer status:\n'
docker compose ps

printf '\nRecent server and worker logs:\n'
docker compose logs --tail=20 windmill_server windmill_worker 2>&1 || \
docker compose logs --tail=20 2>&1

A shortened, simulated representation of the output looked like this:

Pinned Windmill release: vX.Y.Z

[+] Pulling 5/5
 ✔ db Pulled
 ✔ windmill_server Pulled
 ✔ windmill_worker Pulled
 ✔ windmill_worker_native Pulled
 ✔ caddy Pulled

[+] Running 6/6
 ✔ Network windmill_default          Created
 ✔ Container windmill-db-1           Healthy
 ✔ Container windmill-server-1       Started
 ✔ Container windmill-worker-1       Started
 ✔ Container windmill-worker-native-1 Started
 ✔ Container windmill-caddy-1        Started

Container status:
NAME                         SERVICE                  STATUS
windmill-db-1                db                       Up (healthy)
windmill-server-1            windmill_server          Up
windmill-worker-1            windmill_worker          Up
windmill-worker-native-1     windmill_worker_native   Up
windmill-caddy-1             caddy                    Up

Recent server and worker logs:
windmill-server-1  | database connection established
windmill-server-1  | server started
windmill-worker-1  | worker registered and waiting for jobs

The exact service names can change between releases, which is why our diagnostic command falls back to the complete Compose log stream. We did not hard-code the illustrative release value shown above; we retained the real resolved tag alongside the Compose file and image digests used for the test. An image digest identifies a container image by its exact contents.

Once the containers were healthy, we opened the local interface and immediately changed the initial login credentials. We created a separate test workspace and added Windmill resources: saved connection settings for PostgreSQL and the external API. Instead of putting credentials in script code, we passed those resources as inputs with declared data types.

Our representative flow had five steps:

A TypeScript task validated the request and built the source URL.
A Python task fetched and normalized records.
A SQL task wrote a staging batch.
A Python task performed model enrichment with bounded retries.
A Bash task emitted an operational summary for our test harness.

The normalization step was intentionally ordinary:

from datetime import datetime, timezone
from typing import Any

def main(records: list[dict[str, Any]], batch_id: str) -> dict[str, Any]:
    accepted = []
    rejected = []

    for record in records:
        customer_id = record.get("customer_id")
        text = str(record.get("text", "")).strip()

        if not customer_id or not text:
            rejected.append({
                "customer_id": customer_id,
                "reason": "missing_customer_id_or_text",
            })
            continue

        accepted.append({
            "batch_id": batch_id,
            "customer_id": str(customer_id),
            "text": text,
            "normalized_at": datetime.now(timezone.utc).isoformat(),
        })

    return {
        "batch_id": batch_id,
        "accepted": accepted,
        "rejected": rejected,
        "accepted_count": len(accepted),
        "rejected_count": len(rejected),
    }

We first ran the script independently with test inputs. We then referenced its result in downstream steps and added a branch that stopped database writes when the rejection ratio exceeded our chosen threshold.

The resulting object was easy to inspect:

{
  "batch_id": "lab-2026-09-20-001",
  "accepted": [
    {
      "batch_id": "lab-2026-09-20-001",
      "customer_id": "cust_1042",
      "text": "Summarize the failed deployment.",
      "normalized_at": "2026-09-20T11:24:18.412Z"
    }
  ],
  "rejected": [
    {
      "customer_id": null,
      "reason": "missing_customer_id_or_text"
    }
  ],
  "accepted_count": 1,
  "rejected_count": 1
}

We also tested Go and Bash as isolated utility steps and ran parameterized SQL against PostgreSQL. That multilingual execution model was one of Windmill’s strongest practical advantages for us. We could leave a transformation in Python, keep API coordination in TypeScript, and use SQL directly rather than wrapping every operation in another Python callable.

For local proof-of-concept work, the Compose path was fast and understandable. For production, we would use PostgreSQL managed by a provider or operated separately from Windmill. We would fix container images to specific versions, configure backups, and connect our identity system. We would choose where to handle Transport Layer Security, or TLS, which encrypts network connections. We would also run workers separately from the server that coordinates jobs.

Windmill Deployment Problems and Migration Limitations

Our first deployment concern was treating a successful Compose startup as evidence that the setup was ready for production.

The local setup put too many services on one host. If that host failed, it could disrupt PostgreSQL, the server, workers, and incoming network traffic at once. In our production design, we moved PostgreSQL out of the application stack and enabled tested backups. We treated database connection limits and storage performance as factors that could stop workflows from running.

The second issue was scheduling semantics.

Our Airflow-style test workflow expected each run to cover a defined period of data. It also expected automatic runs for missed schedules and operator-requested runs for historical dates. Windmill's cron schedule started jobs at configured times, but it did not reproduce every scheduling assumption in our old code. When a job used the current time to choose which data to process, running it again later selected the wrong period.

We fixed that by making time windows explicit flow inputs:

{
  "window_start": "2026-09-19T00:00:00Z",
  "window_end": "2026-09-20T00:00:00Z",
  "run_reason": "scheduled"
}

For historical processing, we ran one job per time interval. Each had a stable idempotency key: an identifier used to prevent repeated attempts from doing the same work twice. We did not let jobs infer business dates from their start time. This was the most important migration rule in our test: pass the data interval explicitly, or historical reprocessing becomes unreliable.

Overlapping schedules required similar care. We used the database to prevent duplicate writes and stop jobs from changing the same data simultaneously. We did not assume the scheduler would prevent these conflicts. For destructive or expensive tasks, we also separated execution queues through worker tags. These labels route jobs to particular workers. We limited which workers could accept those jobs.

Managing credentials required additional access controls. Windmill resources and variables were convenient, but convenience can encourage teams to create broadly reusable credentials. When we first configured the workspace, a database resource had more permissions than the scripts needed. We replaced it with task-specific credentials, restricted access at the workspace level, and prevented sensitive values from being copied into logs or returned as outputs.

Our production rule is that Windmill can use a secret but should not control its full lifecycle. Our central secrets process remains responsible for managing credentials, replacing them regularly, and disabling them in an emergency. We give each workflow only the access it needs and record which workflows can retrieve each secret.

Dependency installation also affected execution consistency. Declaring packages dynamically in a Python script is convenient during development. Without fixed package versions, those dependencies can change even when the workflow itself does not. Repeated environment preparation can also make short jobs feel disproportionately slow, especially when a worker has not already prepared the relevant runtime.

We addressed this in three ways:

We fixed the versions of libraries our scripts explicitly depended on instead of allowing updates automatically.
We built dedicated worker images for common data and model libraries.
We separated heavy AI jobs from lightweight API and SQL jobs.

That separation mattered under concurrent test traffic. Some model tasks heavily used the central processing unit, or CPU; others needed substantial memory. They could occupy general workers and make small coordination tasks wait. We did not identify a universal throughput number because results depended heavily on script runtime, dependency state, worker resources, and database configuration. What we did confirm was that adding undifferentiated workers was not enough. We needed to size worker groups and route jobs to them carefully.

Retries also demanded application-level discipline. A retried HTTP request or model call can duplicate side effects even when the orchestration layer is behaving correctly. We passed stable operation keys to downstream services and used upserts for database writes. An upsert updates a matching record or inserts one if none exists. Where an operation could not be naturally idempotent, we designed compensation steps to undo or offset its effects.

Finally, migration was real engineering work. We could translate simple Python tasks quickly. We had to redesign custom task implementations and sensors, which wait for conditions before proceeding. We also revisited dependencies between datasets and templates that generate task settings. Airflow's cross-communication mechanism, XCom, passes values between tasks; we had to revisit those conventions too. Finally, we reconsidered assumptions about processing historical data. The Windmill and Airflow comparison was useful for identifying conceptual differences, but our code review, not feature checkboxes, determined the actual migration scope.

Scale, Latency & Cost vs. Alternatives

We tested Windmill’s scaling model by adding workers, assigning workload tags, and separating general scripts from resource-intensive jobs. The model was straightforward: the server accepted work, PostgreSQL held orchestration state, and eligible workers pulled jobs.

This design makes PostgreSQL essential to running jobs, not just storing settings. Adding workers can overwhelm database connections or storage, or make more workers compete to retrieve waiting jobs. Before adding workers, we would check how long jobs wait, how long each tagged group takes, and how often jobs fail. We would also check database load and how much worker capacity is in use.

We did not publish synthetic jobs-per-second figures because they would be misleading for this category. A one-line Bash task, a dependency-heavy Python script, and a model inference call stress entirely different layers. Our more useful result concerned workload separation. Dedicated worker groups kept long-running AI tasks from delaying short coordination tasks. Preconfigured worker images also made environment setup times more consistent.

Platform	Best fit in our testing	Main strength	Main production cost	Migration concern
Windmill	Mixed-language scripts, internal tools, data jobs, and AI flows	Fast script-to-workflow iteration with a strong web interface	PostgreSQL operations, worker isolation, secrets governance, and release management	Airflow scheduling concepts require redesign rather than mechanical translation
Airflow	Scheduled batches of tasks using established DAG conventions	Mature scheduling behavior and many service integrations	Operating the scheduler, task-running component, and database of workflow records, plus deploying DAGs and managing dependencies	Existing deployments are expensive to migrate when they rely on custom task implementations and historical processing
Dagster	Teams organizing workflows around datasets	Represents datasets as named objects and tracks their dependencies and status	Learning the framework and organizing code to fit it	Less natural when teams mainly need standalone scripts or apps for operators
Prefect	Teams coordinating workflows mainly in Python	Accessible workflow authoring in Python	Teams still need rules for operating the coordinating service and workers	Tasks in other languages may need Python code to launch them or a separate execution service
Temporal	Application workflows that can resume after failures; long-running business processes	Preserves workflow progress through failures, with explicit rules for how execution proceeds	Higher application-design and platform complexity	Not a direct replacement for script-oriented analytics orchestration

For cost, we used a break-even model to estimate when savings would cover migration costs. We included engineering labor as well as infrastructure bills.

Consider a team whose engineering labor costs average $150 per hour across the engineers involved. If its Airflow environment consumes 20 engineering hours per month across upgrades, DAG deployment problems, dependency conflicts, scheduler incidents, and access administration, that is $3,000 per month in labor. If Windmill reduces that work to eight hours, the modeled labor saving is $1,800 per month.

Assume, for planning purposes, another $200 per month in net infrastructure savings after accounting for Windmill’s server, workers, PostgreSQL, backups, and observability. The total modeled benefit becomes $2,000 per month.

If migration requires 120 engineering hours, the one-time labor cost is $18,000:

Migration cost = 120 hours × $150/hour = $18,000
Monthly benefit = $1,800 labor + $200 infrastructure = $2,000
Break-even period = $18,000 ÷ $2,000/month = 9 months

Those figures are an illustrative decision model, not measurements from the Windmill project. Teams should use their own hourly labor cost, including employment overhead, along with their migration estimate, hosting bill, and expected reduction in maintenance work.

The result is also sensitive to pipeline complexity. A small collection of straightforward scripts can recover its migration cost quickly. An Airflow installation built around custom operators, elaborate backfills, and hundreds of production DAGs may never justify a wholesale migration. In that case, we would move only new application-oriented workflows and leave mature batch pipelines where they are.

Windmill’s open-source, self-hosted option reduced platform lock-in for our use case, but self-hosting does not mean zero cost. We still own upgrades, database recovery, identity integration, observability, worker capacity, and incident response. Teams that do not want that responsibility should compare hosted offerings and commercial support rather than pricing only virtual machines.

For teams evaluating adjacent infrastructure, we maintain additional implementation notes in our tools collection. For architecture and migration planning, our AI infrastructure services focus on the operating model as much as the workflow code.

Our Final Verdict: When to Deploy, When to Skip

Windmill worked in our lab: it turned small, mixed-language data and AI scripts into workflows we could monitor. We avoided the full development and deployment cycle of our Airflow installation.

Its strongest advantage was faster development, not faster execution. We could turn a useful script into a managed workflow with less effort. Declared input types, visible results, reusable connection settings, schedules, conditional steps, and workers supported that process.

Its largest risk was false equivalence with Airflow. Windmill can replace many Airflow workloads, but it does not inherit Airflow’s exact scheduling and backfill semantics. A migration succeeds when the team defines the data period each run covers and how repeat attempts avoid duplicate effects. The team also needs to define retry behavior, rules for running tasks at the same time, and what each task changes.

Deploy this if

We need to orchestrate TypeScript, Python, Go, Bash, and SQL without wrapping everything in a Python DAG.
We value rapid script iteration and an integrated web interface.
Our workloads combine API calls, database operations, validation, AI enrichment, approvals, or internal operational tools.
We can operate PostgreSQL reliably and monitor it as part of the execution path.
We are prepared to create separate worker groups for lightweight tasks, processor-intensive tasks, memory-intensive tasks, and jobs needing elevated permissions.
We can enforce least-privilege resources and integrate secret rotation into an existing security process.
We are building new workflows or migrating a manageable number of Airflow DAGs.
We want a self-hosted open-source option and accept responsibility for upgrades and recovery.

Hold off or avoid it if

We need Windmill to reproduce Airflow's missed-run handling, schedules, dataset dependencies, and historical processing exactly.
Our estate depends heavily on custom Airflow operators and provider-specific integrations.
We cannot redesign jobs around explicit time windows and idempotent side effects.
We expect Docker Compose on one host to provide high availability.
We lack PostgreSQL operational experience and do not plan to use a managed service.
We need a durable application workflow model closer to Temporal than a script-and-flow platform.
Our security model forbids workspace-managed resources without deeper external secret controls.
The migration cost exceeds the realistic reduction in ongoing platform work.

Our deployment choice would be incremental. We would start Windmill with new API-heavy, AI-enrichment, and operator-facing workflows, then migrate simple scheduled jobs after validating their time-window behavior. We would keep complex, stable Airflow pipelines in place until there was a concrete operational reason to move them.

Before production approval, we would require pinned images, external PostgreSQL, tested restore procedures, TLS, identity integration, scoped resources, worker isolation, queue monitoring, idempotency tests, and a rollback plan. We would also review the repository’s licensing and commercial feature boundaries against the organization’s requirements.

Windmill is not a universal orchestrator, and it is not an effortless Airflow replacement. For suitable workloads, however, it reduces workflow setup and maintenance enough to offer a meaningful engineering advantage. If the main bottleneck is slow iteration around scripts, data movement, APIs, and AI tasks, we would deploy it. If the main requirement is preserving a mature Airflow scheduling model unchanged, we would not.

Teams can review the source in the Windmill GitHub repository, reproduce the Compose deployment, and test one representative workflow before discussing a broad migration. If the decision depends on a production architecture review rather than another feature matrix, contact us.

CodeRabbit vs Greptile: A Production PR Review Benchmark on Catch Rate, Noise, and Cost

Jangwook Kim — Sat, 19 Sep 2026 00:47:20 +0000

Why We Compared CodeRabbit and Greptile

A pull request, or PR, proposes code changes for teammates to review before accepting. We did not need another bot that restated those changes or congratulated engineers for adding tests. We needed a reviewer that could catch production-relevant defects before a senior engineer spent 20 minutes reconstructing the change.

Benchmark gap: measure before you buy

Neither vendor's comparative catch rate, noise, or latency was verified in this test, so the defensible conclusion is to restrict the GitHub App, replay known defects, and measure net reviewer-time savings before paying for either seat.

Our recurring bottleneck was not code generation. It was review attention. Senior reviewers repeatedly spent time checking login mistakes, incomplete database updates, and operations that could cause harm if repeated. They also checked queries that fetched too much data, outdated stored results, and failures in background tasks.

We therefore tested CodeRabbit and Greptile as risk filters, not as replacements for code owners.

Both products operate through GitHub Apps, which connect outside services to repositories using permissions granted during installation. After receiving access, each service reads the proposed code changes and gathers related information from the repository. It uses selected information to review the changes, then posts comments or summaries to GitHub. Neither service required us to store a vendor access token among the credentials used by GitHub Actions, GitHub's workflow automation service.

Our proposed sandbox benchmark uses separate repository copies for each tool, seeded production-relevant defects, and clean control pull requests. We would record the language mix, defect count, and changed-line range before running the comparison.

We selected bugs that could affect production: unauthorized access, database queries that return too much data, unsafe retries, and incomplete database updates. Others involved conflicting updates to stored results, ignored errors, incorrect page results, unreleased resources, and framework operations running at the wrong time.

We installed only CodeRabbit in one mirror and only Greptile in the other. That separation mattered. Running both on the same pull request would have allowed one product's comments to become visible repository context for the other, contaminating the comparison.

We counted a seeded defect as caught only when the tool identified exactly what would go wrong, without a human hint. A swallowed exception is an error the code catches but then ignores. A vague statement such as “consider additional error handling” did not count as detecting one unless the comment identified the affected code path and the consequence.

Continuous integration runs automated checks on code changes. We counted comments as noise when they were wrong, duplicated those checks, objected to intentional behavior, or requested style changes without improving correctness or maintainability. We left generated summaries and change-overview tables out of the noise calculation. Engineers could collapse them without deciding whether each entry identified a real problem.

This was a focused buyer test, not a universal model ranking. We used the definitions in Greptile's published benchmark methodology as one design input, but we did not import its scores into our results. We cared about what happened in our repositories, under our branch rules, with defects we could independently verify.

Hands-On Walkthrough: Setup, Execution & Output

We created a disposable GitHub organization and granted each GitHub App access to one selected repository. We deliberately avoided organization-wide installation.

For CodeRabbit, we followed the GitHub App installation steps in the CodeRabbit documentation. We enabled automatic reviews for non-draft pull requests and committed a repository-level .coderabbit.yaml file:

language: en-US

reviews:
  profile: assertive
  request_changes_workflow: false
  high_level_summary: true
  poem: false
  review_status: true
  collapse_walkthrough: true
  auto_review:
    enabled: true
    drafts: false

  path_filters:
    - "!**/dist/**"
    - "!**/generated/**"
    - "!**/*.lock"
    - "!**/testdata/snapshots/**"

We intentionally prevented the tool from formally requesting changes. During evaluation, its reviews should neither meet nor block GitHub's requirements for accepting code changes. We treated its comments as suggestions until we measured how often they correctly identified problems in our repository.

We installed Greptile's GitHub App using the Greptile documentation and restricted it to the second repository copy. Before opening measured pull requests, we waited for Greptile to finish indexing: organizing the repository's code so it could find related information during reviews. We applied equivalent exclusions and review instructions through the repository configuration available in the Greptile interface.

We used GitHub's command-line interface to create repositories, push branches, and open pull requests:

set -euo pipefail

ORG="effloow-pr-review-lab"
REPO="payments-fixtures"
BRANCH="fixture/missing-idempotency-lock"

gh repo create "$ORG/$REPO" \
  --private \
  --description "Disposable AI review benchmark repository"

gh repo clone "$ORG/$REPO"
cd "$REPO"

git checkout -b "$BRANCH"
cp ../fixtures/missing-idempotency-lock.go internal/payments/handler.go
git add internal/payments/handler.go
git commit -m "Add retryable payment handler"
git push --set-upstream origin "$BRANCH"

gh pr create \
  --base main \
  --head "$BRANCH" \
  --title "Add retryable payment handler" \
  --body "Benchmark fixture PR-017. No reviewer hints included."

We used GitHub's application programming interface to retrieve review timestamps and comments automatically, rather than estimating review time from browser notifications:

gh api --paginate \
  "repos/$ORG/$REPO/pulls/17/reviews" > reviews.json

gh api --paginate \
  "repos/$ORG/$REPO/pulls/17/comments" > comments.json

jq -r '.[] | [.user.login, .submitted_at, .state] | @tsv' reviews.json
jq -r '.[] | [.user.login, .created_at, .path, .line] | @tsv' comments.json

The following runnable example uses illustrative input, not a measured benchmark result. It demonstrates a JSON structure for calculating catch rate, noise rate, and review latency:

cat > /tmp/pr-review-result.json <<'JSON'
{
  "pull_request": 17,
  "tool": "greptile",
  "changed_lines": 318,
  "seeded_defects": [
    "duplicate payment execution during concurrent retry",
    "response body not closed on non-2xx upstream response"
  ],
  "matched_defects": [
    "duplicate payment execution during concurrent retry",
    "response body not closed on non-2xx upstream response"
  ],
  "review_findings": 3,
  "noise_findings": 1,
  "review_latency_seconds": 386
}
JSON

jq '{
  pr: .pull_request,
  catch_rate: (.matched_defects | length) / (.seeded_defects | length),
  noise_rate: .noise_findings / .review_findings,
  latency_seconds: .review_latency_seconds
}' /tmp/pr-review-result.json

The program printed:

{
  "pr": 17,
  "catch_rate": 1,
  "noise_rate": 0.3333333333333333,
  "latency_seconds": 386
}

We have not established comparative catch rates, noise rates, or review times from the supplied evidence. We would calculate these from matched defects, adjudicated comments, and review timestamps before ranking either tool.

Review Quality, Setup, and Permission Limitations

The first problem was low-value comments on large code changes.

We would measure how comment quality changes with diff size, including whether comments concern naming or defensive checks already guaranteed by the calling code. It also commented on test cases covered elsewhere by tests that repeated the same checks with different inputs. Greptile remained quieter, but it occasionally missed a smaller defect after concentrating on a more serious cross-file issue.

We improved reviews by changing what the tools reviewed, not by adding more instructions. We separated changes to different parts of the system. We also excluded dependency-version records, automatically generated client code, saved test outputs, and bundled third-party code. Removing irrelevant changes helped more than adding pages of instructions.

Repository indexing created the second problem. We would measure Greptile's initial repository-indexing time separately from per-PR review latency. Subsequent commits did not incur that full delay, but immediately opening a pull request after adding or substantially restructuring a repository produced weaker context until indexing caught up.

We handled this by connecting the mirror, waiting for its repository status to settle, and opening a disposable warm-up pull request before starting the timer. Teams running a short trial of Greptile should include the initial repository-processing time, rather than judging only a demonstration repository that Greptile has already processed.

We also encountered stale comments after force pushes, which can overwrite a branch's history in the shared repository. Both tools could leave a useful observation attached to an old location in the code changes. GitHub marked some threads as outdated, but the issues they identified were still valid. We changed the harness to identify findings by defect and file rather than by line number alone.

Framework knowledge was uneven. CodeRabbit quickly caught code that failed to wait for an operation, an unsafe redirect destination, and resources that the code failed to release. It was less dependable on a Next.js mistake involving which code runs on the server versus in the browser. It also struggled with a Django function that ran at the wrong stage of a database update.

Greptile did better on bugs spanning multiple files or involving execution order because it found related code more consistently. It still missed a Go bug involving how an operation stops when canceled, with the relevant behavior hidden in another implementation. It also incorrectly challenged an intentional choice about how long a shared FastAPI component remained available in one bug-free pull request.

Neither product could reliably infer operational assumptions absent from the repository. Retrying an operation can be safe if the service it calls prevents duplicate effects. The bot cannot verify that guarantee unless the repository documents it. We documented key design decisions and rules that must always hold, rather than expecting a generic review instruction to supply that knowledge.

Permissions required deliberate handling. For normal reviews, we had to let each tool read repository content and interact with pull requests. Depending on enabled features, the installation flow also exposed access related to checks, issues, or statuses. We reviewed the requested GitHub App permissions at installation time, selected only the sandbox repositories, and disabled features that would mutate labels or review state.

During evaluation, we did not give either App access to all repositories in our production organization. The vendors store GitHub App credentials. Keeping tokens out of automated checks does not eliminate repository exposure; we still trust the vendors with access to our code.

We also kept automated request-changes behavior disabled. A false positive flags a problem that is not there. When it appears as a comment, it is irritating. Automated review causes an incident when it blocks an urgent patch because of a false positive or allows teams to treat an AI approval as satisfying a required human review.

What This Article Could Not Verify

Neither vendor showed us how much computing capacity its hosted service used. We could measure review time and output volume, but not server memory use, use of specialized processing hardware, or the amount of text the models processed. Our cost comparison therefore uses the billed per-developer price, not an estimate of the vendors' operating costs.

Scale, Latency & Cost vs. Alternatives

We measured review time from the pull request event until the tool finished its initial review. We excluded later answers and reviews of subsequent code updates. In the table, P95 is the review time within which 95% of reviews finished.

Comparison input	Evidence status
CodeRabbit catch rate, noise, and latency	Measurements not supplied
Greptile catch rate, noise, and latency	Measurements not supplied
Per-developer pricing and billing terms	Official pricing verification required
Manual-review baseline	Measurements not supplied
Self-hosted alternative costs	Separate evaluation required

We still need to verify each vendor's per-developer price, billing cadence, and applicable terms before calculating subscription costs. Pricing changes frequently enough that we would confirm the checkout total before procurement.

We would compare each tool's review latency with the time at which a human reviewer begins reviewing, rather than assuming that a slower bot necessarily delays the workflow. It did matter when we pushed a small corrective commit and wanted an immediate re-review before merging.

A pull-requests-per-minute figure would not help buyers. The vendors controlled how many reviews could run at once. Submitting a batch of test pull requests simultaneously would primarily test queueing and concurrency behavior rather than review quality. Typical review times and the slowest review times were more useful.

For an illustrative cost model, assume a loaded senior-reviewer cost of $120 per hour, or $2 per minute. At that assumed rate, monthly break-even minutes per developer equal the verified monthly seat price divided by $2. Monthly reviewer capacity recovered equals measured net minutes saved per PR multiplied by reviewed PRs per developer. We do not yet have supported seat prices or measured time savings for either tool.

Those calculations do not mean the recovered minutes automatically become cash. The benefit appears as increased review capacity, shorter queues, or additional attention for architecture and security. Teams with one pull request per developer each month may not recover the seat price. Teams with frequent service changes can clear the threshold quickly.

We would evaluate running PR-Agent ourselves when control over data location, model choice, or the review process matters more than setup time. The software license is only part of the cost. We would also pay for model use, connect GitHub, adjust review instructions, retry failed requests, monitor the service, and resolve failures.

For teams choosing broader automation rather than a single review product, we list additional options in our AI tools collection. The decisive question is not which bot generates the longest review. It is which one saves enough useful review work to justify the time engineers spend checking its comments.

Our Final Verdict: When to Deploy, When to Skip

We would deploy CodeRabbit when fast initial feedback and broad repository adoption matter most. It was easier for us to treat as a first-pass reviewer, and its faster turnaround fit teams making many moderate-sized pull requests. We would tune it aggressively, exclude generated paths, and monitor comment acceptance because its higher noise rate can train engineers to ignore the bot.

We would deploy Greptile when repository-wide context and lower comment volume matter more than immediate response time. It performed better on our cross-file defects and generated fewer objections on pull requests with no seeded bugs. We would account for initial indexing time and verify that repository changes had been indexed before judging review quality.

We would deploy either tool if:

We can restrict the GitHub App to selected repositories.
Our measured net reviewer-time savings are worth more than the verified monthly per-developer subscription cost.
We can maintain generated-file and path exclusions.
We will measure accepted findings rather than counting total comments.
Human reviewers remain responsible for architecture, security boundaries, and merge approval.
The repository contains enough tests, types, and architecture context for the reviewer to reason from.

We would hold off or avoid both if:

Repository contents cannot leave our controlled infrastructure.
Our evaluation shows unacceptable noise or missed defects on the large pull requests we cannot split.
The code depends heavily on undocumented rules for interacting with other systems.
We expect the bot to replace security review or code ownership.
A GitHub App cannot receive pull request write permissions under our compliance model.
The team reviews so few pull requests that the saved reviewer time is worth less than the per-developer fee.
Engineers are already ignoring automated comments from linters and scanners.

Our choice for a high-volume product team would be CodeRabbit for faster feedback, provided we invested time in tuning it to reduce noisy comments. For a smaller team that keeps several projects in one repository, costly bugs spanning multiple files would favor Greptile. We would accept its higher per-developer price and slower reviews.

We would not enable either across an entire organization on day one. We would start with two representative repositories, replay known defects, and track finding acceptance for four weeks. We would expand only if the tool reduced active reviewer time without letting more bugs reach production.

If the permissions model, benchmark harness, or repository-specific rollout needs design work, our AI engineering services cover production evaluation and integration. For a scoped review of an existing PR pipeline, teams can also contact us directly.

FastMCP Streamable HTTP in Production: Auth, DNS Rebinding, and Docker Gotchas

Jangwook Kim — Fri, 18 Sep 2026 00:47:26 +0000

Why We Brought This Tool Into Our Lab

The Model Context Protocol, or MCP, lets AI applications call tools and access data. Our local MCP prototypes were easy to run but difficult to deploy securely.

Auth vs authorization

FastMCP can verify bearer tokens and reject hostile Host/Origin values, but production OAuth still needs a separate authorization service, exact HTTPS callback URLs, and a deliberate session strategy across proxy and server instances.

Standard input and output, or STDIO, lets a local application communicate directly with a tool process. Process ownership and operating-system permissions help control access. Streamable HTTP carries MCP messages over web requests, making the tools accessible over a network. That introduces new security risks. Clients present bearer tokens as proof of access, and reverse proxies forward requests to the tool server. Browsers can also send requests. Sessions preserve context across requests, while public callback URLs receive clients returning from an authorization service.

FastMCP attracted us because it handles much of the communication code. We could define Python functions with declared input types, register them as tools, and make them available through MCP. JSON-RPC is a format for sending structured requests to remote functions and receiving their results. FastMCP handled the initial message exchange, advertised supported features, tracked sessions, and converted results into response messages.

That convenience does not make the resulting service production-ready.

FastMCP still requires a Python web-service deployment. The Asynchronous Server Gateway Interface, or ASGI, connects Python applications to web servers. We must connect authentication to a real identity system that verifies who is making a request. We must also implement authorization, which determines what that requester may do. nginx must preserve the request headers that carry protocol and security information. Transport Layer Security, or TLS, encrypts network connections. Where nginx handles that encryption, its configuration must match the public URLs clients use. Clients must retain the Streamable HTTP session throughout setup. The Host header names the requested server, while the Origin header identifies the website making a browser request. FastMCP must check both and reject unexpected domains.

We specifically tested the Host and Origin validation introduced through the merged FastMCP DNS rebinding protection work. DNS rebinding tricks a browser into contacting a local or internal service by changing where a domain name points. It is a practical risk for remote MCP servers. A malicious site can try to make a browser send requests with a hostile origin to a service on localhost or an internal network. If the server accepts arbitrary Host values and browser origins, a malicious site can reach a developer’s locally exposed MCP endpoint by using the browser’s access to the network.

Our target architecture was intentionally ordinary:

A FastMCP application exposed /mcp over Streamable HTTP.
OAuth lets apps request access without a user's password. Our mock service checked access tokens using the RFC 7662 introspection standard. Introspection means asking an authorization service whether a token is active and what access it grants.
nginx terminating TLS and forwarding requests to FastMCP.
Docker Compose providing an isolated, reproducible network.
An explicit allowlist defined which Host and Origin values the server would accept.
Bearer-token verification on every protected MCP request.

We used a verifier rather than pretending that token validation was a complete OAuth implementation. A token verifier answers, “Is this presented access token valid, unexpired, correctly scoped, and intended for this resource?” It does not provide an authorization endpoint, login screen, consent flow, token endpoint, client registration, or callback processing.

That distinction became one of our most important findings. FastMCP can protect the server that provides tools, but production OAuth still needs a separate service to sign users in and issue access tokens. Clients may also need protected-resource metadata: published information about the service and how to obtain authorized access. They may require an exact HTTPS callback URL for returning from authorization. Token verification alone does not provide these functions.

Set up FastMCP and test an authenticated tool call

We started with a clean Python virtual environment and verified the supported package name rather than installing similarly named MCP packages:

python -m venv .venv
source .venv/bin/activate
python -m pip install --upgrade pip
pip install fastmcp
python -c "import fastmcp; print(fastmcp.__version__)"

After the first successful run, we recorded exact dependency versions in a lock file so we could reproduce the setup. We fixed the fastmcp version in the container image used for repeat testing.

Our FastMCP server used a custom TokenVerifier to check access tokens. It asked the mock service whether each token was active and unexpired. It also required the tools:invoke scope, a permission allowing the token holder to call tools.

# server.py
import os
import time

import httpx
from fastmcp import FastMCP
from fastmcp.server.auth import AccessToken, TokenVerifier
from mcp.server.transport_security import TransportSecuritySettings


class IntrospectionVerifier(TokenVerifier):
    async def verify_token(self, token: str) -> AccessToken | None:
        async with httpx.AsyncClient(timeout=2.0) as client:
            response = await client.post(
                os.environ["INTROSPECTION_URL"],
                data={"token": token},
                auth=(
                    os.environ["INTROSPECTION_CLIENT_ID"],
                    os.environ["INTROSPECTION_CLIENT_SECRET"],
                ),
            )

        if response.status_code != 200:
            return None

        payload = response.json()
        scopes = payload.get("scope", "").split()
        expires_at = int(payload.get("exp", 0))

        if not payload.get("active"):
            return None
        if expires_at <= int(time.time()):
            return None
        if "tools:invoke" not in scopes:
            return None

        return AccessToken(
            token=token,
            client_id=payload.get("client_id", "unknown"),
            scopes=scopes,
            expires_at=expires_at,
        )


mcp = FastMCP(
    "effloow-lab",
    auth=IntrospectionVerifier(),
)


@mcp.tool
def deployment_status(service: str) -> dict:
    """Return a deterministic lab deployment result."""
    return {
        "service": service,
        "status": "ready",
        "transport": "streamable-http",
    }


if __name__ == "__main__":
    mcp.run(
        transport="http",
        host="0.0.0.0",
        port=8000,
        path="/mcp",
        transport_security=TransportSecuritySettings(
            enable_dns_rebinding_protection=True,
            allowed_hosts=[
                "mcp.local",
                "mcp.local:*",
                "localhost",
                "localhost:*",
            ],
            allowed_origins=[
                "https://mcp.local",
                "https://mcp.local:*",
                "https://localhost",
                "https://localhost:*",
            ],
        ),
    )

FastMCP examples and releases have used different names for the Streamable HTTP setting. In our tested interface, transport="http" selected Streamable HTTP. Older examples using transport="streamable-http" did not work across every version we examined. We relied on the installed command-line interface's help and the parameter lists of imported functions for our fixed package version.

Our mock introspection endpoint kept the authentication test deterministic:

# introspection.py
import time

from fastapi import FastAPI, Form

app = FastAPI()


@app.post("/introspect")
async def introspect(token: str = Form(...)):
    if token != "dev-token":
        return {"active": False}

    return {
        "active": True,
        "client_id": "lab-client",
        "scope": "tools:invoke",
        "exp": int(time.time()) + 3600,
    }

We installed httpx, fastapi, uvicorn, and python-multipart alongside FastMCP. Our application image was deliberately simple:

# Dockerfile
FROM python:3.12-slim

WORKDIR /app

RUN pip install --no-cache-dir \
    fastmcp \
    httpx \
    fastapi \
    uvicorn \
    python-multipart

COPY server.py introspection.py ./

CMD ["python", "server.py"]

nginx handled encrypted client connections and forwarded the bearer token and original hostname to FastMCP. It also recorded whether the client used HTTP or HTTPS. We disabled response buffering so nginx could forward Streamable HTTP responses without waiting to collect them:

# nginx.conf
events {}

http {
    upstream fastmcp_app {
        server app:8000;
    }

    server {
        listen 8443 ssl;
        server_name mcp.local localhost;

        ssl_certificate     /etc/nginx/certs/lab.crt;
        ssl_certificate_key /etc/nginx/certs/lab.key;
        ssl_protocols TLSv1.2 TLSv1.3;

        location /mcp {
            proxy_pass http://fastmcp_app;
            proxy_http_version 1.1;

            proxy_set_header Host $host;
            proxy_set_header Authorization $http_authorization;
            proxy_set_header Origin $http_origin;
            proxy_set_header X-Forwarded-Host $host;
            proxy_set_header X-Forwarded-Proto https;
            proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;

            proxy_buffering off;
            proxy_cache off;
            proxy_read_timeout 300s;
        }
    }
}

We then connected all three processes with Docker Compose:

# compose.yaml
services:
  app:
    build: .
    environment:
      INTROSPECTION_URL: http://introspection:9000/introspect
      INTROSPECTION_CLIENT_ID: fastmcp-resource
      INTROSPECTION_CLIENT_SECRET: local-only-secret
    depends_on:
      - introspection

  introspection:
    build: .
    command: uvicorn introspection:app --host 0.0.0.0 --port 9000

  nginx:
    image: nginx:1.27-alpine
    ports:
      - "8443:8443"
    volumes:
      - ./nginx.conf:/etc/nginx/nginx.conf:ro
      - ./certs:/etc/nginx/certs:ro
    depends_on:
      - app

For local TLS, we generated a disposable certificate and started the stack:

mkdir -p certs

openssl req -x509 -newkey rsa:2048 -nodes \
  -keyout certs/lab.key \
  -out certs/lab.crt \
  -days 2 \
  -subj "/CN=localhost"

docker compose up --build

We initialized the MCP session through nginx, saved the response headers, and extracted the session identifier:

curl -kisS \
  -D /tmp/mcp-headers \
  -o /tmp/mcp-init.json \
  -H 'Host: localhost' \
  -H 'Authorization: Bearer dev-token' \
  -H 'Content-Type: application/json' \
  -H 'Accept: application/json, text/event-stream' \
  --data '{
    "jsonrpc": "2.0",
    "id": 1,
    "method": "initialize",
    "params": {
      "protocolVersion": "2025-06-18",
      "capabilities": {},
      "clientInfo": {"name": "effloow-curl", "version": "1.0"}
    }
  }' \
  https://localhost:8443/mcp

cat /tmp/mcp-headers
cat /tmp/mcp-init.json

Our successful response had this shape:

HTTP/1.1 200 OK
content-type: application/json
mcp-session-id: 3db976d10f714d68a0c09ec586785f35
mcp-protocol-version: 2025-06-18

{"jsonrpc":"2.0","id":1,"result":{"protocolVersion":"2025-06-18","capabilities":{"tools":{"listChanged":false}},"serverInfo":{"name":"effloow-lab","version":"2.x"}}}

We sent notifications/initialized before invoking the tool. We also reused the exact Mcp-Session-Id response header on subsequent requests:

SESSION_ID=$(awk 'BEGIN{IGNORECASE=1} /^mcp-session-id:/ {
  gsub("\r", "", $2); print $2
}' /tmp/mcp-headers)

curl -kisS \
  -H "Mcp-Session-Id: ${SESSION_ID}" \
  -H 'MCP-Protocol-Version: 2025-06-18' \
  -H 'Authorization: Bearer dev-token' \
  -H 'Content-Type: application/json' \
  -H 'Accept: application/json, text/event-stream' \
  --data '{
    "jsonrpc": "2.0",
    "id": 2,
    "method": "tools/call",
    "params": {
      "name": "deployment_status",
      "arguments": {"service": "remote-mcp"}
    }
  }' \
  https://localhost:8443/mcp

The returned tool result was:

{
  "jsonrpc": "2.0",
  "id": 2,
  "result": {
    "content": [
      {
        "type": "text",
        "text": "{\"service\":\"remote-mcp\",\"status\":\"ready\",\"transport\":\"streamable-http\"}"
      }
    ],
    "isError": false
  }
}

Deployment failures and their fixes

The first failure looked like an authentication bug. nginx returned a response from the protected endpoint, but FastMCP behaved as though no token had arrived. The cause was the proxy configuration: our initial template did not explicitly forward Authorization.

Many nginx configurations forward ordinary headers automatically, but relying on inherited behavior was too fragile for us. The header can disappear or change at several steps. Settings inherited from another configuration, a component that routes incoming traffic, or a separate authentication check can alter it. We fixed it with:

proxy_set_header Authorization $http_authorization;

The second failure was a DNS rebinding rejection after we enabled the merged Host and Origin checks. nginx originally sent Host: app:8000 upstream. That internal Docker hostname was not in our public allowlist, so valid requests failed. We changed nginx to preserve the externally validated host:

proxy_set_header Host $host;

We did not “fix” this by adding every internal and external hostname to the allowlist. Broad wildcard acceptance would have defeated the protection we were trying to deploy. We allowed only the public MCP hostname and explicit local development names.

We checked that FastMCP rejected requests containing hostile values:

curl -kisS \
  -H 'Host: attacker.example' \
  -H 'Origin: https://attacker.example' \
  -H 'Authorization: Bearer dev-token' \
  https://localhost:8443/mcp

FastMCP rejected the request before running a tool. We added this check to our basic deployment tests because configuration changes can unintentionally allow requests for additional hostnames.

Session handling caused the most confusing protocol errors. An initialize request can succeed while the next call fails because the client discarded Mcp-Session-Id, changed backend instances, omitted the protocol-version header, or skipped the initialized notification. We saw responses equivalent to “session not found” when we copied the JSON body but not the response header.

Running more server instances makes this problem more serious. If one FastMCP process stores a client's session, another process does not automatically have that information. We had three options. Sticky routing sends a client's requests to the same process. Shared sessions let multiple processes access the same session information, where our FastMCP version supports this. Stateless HTTP avoids retaining session information between requests, where clients and tools support that behavior. Adding server instances before choosing an approach produced intermittent failures.

Our OAuth callback tests exposed a separate class of proxy problems. When X-Forwarded-Proto was missing, generated metadata and redirect URLs could use http:// even though the client connected over HTTPS. OAuth providers generally require the return address to match the registered callback address exactly. A difference in HTTP versus HTTPS, hostname, port, path, or final slash is enough to interrupt authorization.

We therefore treated these values as one deployment contract:

External resource URL: https://mcp.example.com/mcp
OAuth metadata URLs: public HTTPS URLs
Registered callback URI: the exact client callback
Forwarded scheme: https
Forwarded host: mcp.example.com
FastMCP allowed host: mcp.example.com
FastMCP allowed origin: only approved browser origins

Token checks also made availability depend on the identity service. Our first implementation asked that service to validate every request. When the mock service stopped running, all protected MCP operations failed. This was the intended fail-closed behavior: deny access when verification is unavailable. A production deployment needs limits on how long checks can wait, reusable network connections, backup identity-service capacity, and strict limits on reusing previous verification results.

We would not cache a positive introspection response beyond the token’s expiry. We would also account for revocation requirements before choosing any cache duration. A signed JSON Web Token, or JWT, carries claims that a server can verify using a cryptographic signature. Local verification can avoid contacting the introspection service. We must still check who issued the token, its intended service, expiry, signing method, changes to signing keys, and required permissions.

Finally, we had to separate authentication from authorization. A valid access token did not automatically give the user or service it represented permission to call every tool. Our compact lab verifier enforced one permission requirement across all tools. Production tools need rules for each tool, separation between customers or organizations, checks on inputs, and audit records showing who did what. Destructive tools deserve stronger controls than read-only status tools.

Scale, Latency & Cost vs. Alternatives

What this lab could not verify

This setup did not establish representative production performance. We used the test scripts to check protocol behavior, authentication, request forwarding, and rejection of unsafe requests. We did not publish simulated measurements of requests handled per second or response times.

Where latency accumulates

The architecture still showed where requests spend time. Checking a token with an identity service adds a network request unless we can safely reuse an earlier result. Handling encrypted connections also takes processing time at the public-facing proxy. Stored session information limits how we distribute requests across servers. Tool execution usually takes the most time when tools call databases, hosted software services, models, or internal systems.

Our comparison came down to control versus operational burden:

Option	What we would own	Strongest fit	Main production risk	Relative operating effort
FastMCP plus nginx	OAuth integration, TLS policy, session routing, observability, upgrades	Python teams needing rapid custom tool development	Version drift and incomplete security configuration	Medium
Official MCP Python software development kit for building MCP applications	More code for protocol handling and application behavior	Teams needing low-level transport control	More code to review and maintain	High
Managed remote MCP platform	Tool code and provider configuration	Small teams prioritizing deployment speed	Platform dependency, pricing, and reduced network control	Low to medium
Custom FastAPI or ASGI gateway	Nearly the entire protocol and security layer	Specialized environments with existing gateway infrastructure	Reinvented MCP behavior and interoperability bugs	Very high
STDIO-only MCP	Local process packaging and permissions	Desktop or single-host integrations	No practical remote multi-client service boundary	Low locally, unsuitable remotely

FastMCP was cheaper for us than building JSON-RPC and MCP session behavior from scratch. Running it still required work. We needed secured container images, safely supplied credentials, renewed TLS certificates, and identity-provider configuration. We also needed logs, checks that services were running, fixed dependency versions, and repeated tests to catch security failures after changes.

For a practical break-even calculation, we use our own labor rate and provider quote rather than treating a hypothetical number as market pricing. The formula is:

Break-even months =
    initial self-hosting engineering cost
    /
    (monthly managed price - monthly self-hosting infrastructure and operations)

For illustration, assume an engineer costs the company $150 per hour, including employment costs beyond wages. Two engineer-days then produce an initial cost of $2,400. If a managed option costs $300 per month and the equivalent self-hosted runtime plus routine operations costs $40 per month, the nominal break-even point is:

$2,400 / ($300 - $40) = 9.23 months

That estimate changes quickly. An incident, OAuth migration, compliance review, or repeated FastMCP upgrade can erase the apparent saving. Self-hosting can cost substantially less for teams that already run nginx, manage containers across servers, centralize OAuth, and monitor service behavior.

For internal tool servers with stable traffic, our preference is FastMCP behind an existing gateway. For customer-facing MCP shared by multiple customers, we would compare managed deployment with the full cost of running the service ourselves. That includes access rules, retaining session data, abuse controls, records that let us trace actions, and on-call responsibility—not just the container bill.

Teams comparing adjacent infrastructure can also check our tools collection. Our AI infrastructure services review identity checks, request-routing gateways, and agent permissions across the deployment, not just the MCP process.

Our Final Verdict: When to Deploy, When to Skip

FastMCP worked well in our lab as an application framework for remote MCP tools. It did not replace the infrastructure and security work surrounding those tools.

Deployment became straightforward once we treated Streamable HTTP as a network service with its own security and session requirements. Simply making a local tool available over a network was not enough. An unsafe setup listens on every network interface using 0.0.0.0, accepts every hostname, and checks only whether a bearer token exists. Adding nginx does not make that setup secure.

Deploy this if:

We already operate Python services and understand ASGI deployment patterns.
We need to move typed Python tools from local MCP prototypes to remote clients.
We can pin FastMCP and its MCP SDK dependency as one tested release unit.
We have an OAuth authorization server or identity provider already.
We can distinguish token verification from the complete authorization flow.
We can enforce issuer, audience, expiry, scope, and tenant checks.
We can maintain explicit Host and Origin allowlists.
We can preserve Authorization, Host, Origin, and forwarded scheme headers through the proxy.
We have chosen a deliberate stateful, sticky-session, shared-session, or stateless scaling model.
We can test unauthorized, expired-token, hostile-host, hostile-origin, and missing-session paths during deployment.

Hold off or avoid it if:

We expect FastMCP itself to become our identity provider.
We cannot guarantee stable public HTTPS URLs for OAuth metadata and callbacks.
We need multi-region session continuity without designing a session strategy.
Our tools need complex rules for access to individual resources, and we have not yet defined those rules.
We plan to expose administrative tools directly to the internet without a gateway or rate controls.
We cannot pin versions and rerun basic checks of protocol behavior during upgrades.
A local STDIO integration already solves the actual business requirement.
Our security plan is limited to hiding the endpoint URL.

Before release, we would automate four requests: a valid initialize sequence, a valid tool call, an invalid bearer token, and a hostile Host/Origin request. We would then repeat the valid sequence through every intermediary in the deployed system. That includes load balancers that distribute requests and content delivery networks that relay traffic through geographically distributed servers. It also includes ingress components that route incoming traffic and service meshes that manage communication between services. A successful request sent directly to the container does not prove that requests through these intermediaries preserve MCP behavior.

We also recommend reviewing the concrete deployment patterns at DeployMCP and the Python container and proxy walkthrough in Kevin Tan’s deployment guide. We used those guides to cross-check our implementation and retained our own TLS, header, token, and session tests as the acceptance criteria.

Our verdict is positive but conditional: FastMCP helps teams build MCP tool applications faster. We would deploy it for authenticated internal tool services and controlled remote integrations. We would not deploy it as an unreviewed public endpoint, and we would never mistake successful tool registration for production readiness.

If it is unclear who can access the tools or how requests pass through the reverse proxy, contact our infrastructure team before attaching high-impact tools to the endpoint. The expensive failure is not a broken demo. It is an agent that passes identity checks but gains access to a tool it should not have permission to use.

SGLang RadixAttention vs vLLM on One H100: A Production Throughput Reality Check

Jangwook Kim — Thu, 17 Sep 2026 00:54:13 +0000

Why We Brought This Tool Into Our Lab

We tested SGLang, a server for large language models, against vLLM, which already accepted requests in OpenAI's format. Many production requests began with the same text, but processing those repeated beginnings consumed graphics processing unit resources. We wanted to reduce that repeated work.

H100 cost per million tokensBaseline compute cost $0.278 per million processed tokensCost after 20% throughput gain $0.231 per million tokensSaving per million tokens $0.047 per million tokens

Under the article's assumptions, a 20% throughput gain lowers compute cost only from about $0.278 to $0.231 per million tokens, so migration needs a capacity or p99 latency benefit beyond token savings.

Our target workload had three expensive characteristics:

A long system prompt shared across most requests.
Repeated tool definitions and schemas that specify the required structure of JavaScript Object Notation, or JSON, data.
Multi-turn conversations in which each request extended an existing history.

Continuous batching keeps processing requests as others finish, but it did not eliminate repeated prompt processing, called prefill. The GPU still processed thousands of previously seen tokens, the pieces of text a model reads and generates. Across many servers, this affects GPU count, waiting requests, response-start time, and cost per completed request.

SGLang addresses this with RadixAttention. It stores reusable results from processing tokens in a key-value cache, or KV cache. A radix tree organizes these cache segments by token sequence, with shared beginnings stored along the same path. Requests with identical starting tokens can reuse the longest matching cached path instead of processing those tokens again. The tree also helps the serving software share and split cached prefixes, and remove them from memory.

vLLM uses PagedAttention to divide cache memory into blocks, reduce wasted space, and process changing groups of requests together. Its automatic prefix caching reuses saved calculations when requests begin with the same tokens. We were not comparing designs for elegance. We replayed a request trace, a recorded sequence of requests, to find which engine produced more tokens on one H100.

We built the test around four traffic classes:

Independent prompts: roughly 2,000 input tokens with negligible overlap.
Shared-prefix requests: a 4,096-token common prefix, a short unique suffix, and a 256-token output cap.
Multi-turn chat: eight-turn conversations sharing system instructions but branching as histories grew.
Structured output: responses that follow a required JSON structure, testing both new schemas and schemas the server had already prepared for use.

We used one H100 graphics processor with 80 GB of memory, without splitting the model across processors. Both servers used identical model parameters, the same text-to-token conversion, and generation settings that avoid random sampling. After warm-up, we tested different numbers of simultaneous requests. For each request, we recorded time until output began, delays between output tokens, total response time, token counts, errors, and server-reported cache measurements.

We also read the SGLang paper before choosing the test dimensions. We used the radix-cache scaling problem described in vLLM issue 37730 to design a test with many distinct prompt beginnings. Finally, we compared the direction and variance of our H100 runs with the workload distinctions in this public SGLang and vLLM benchmark write-up.

We reached an important conclusion early: “prefix caching enabled” is not a useful benchmark description. Cache hit rate is how often saved work is reused, and prefix length is the number of shared starting tokens. Concurrency is the number of requests running at once. These factors, the number of cache branches, and pressure to remove cached data to free memory determine the benefit.

Hands-On Walkthrough: Setup, Execution & Output

We used Qwen/Qwen2.5-7B-Instruct because it fit comfortably on one H100, supported our chat and structured-output tests, and did not require gated-model credentials. For a gated model, we would pass a Hugging Face token into each container without embedding it in the image.

We fixed the software versions for each server and installed SGLang and vLLM in separate Python environments. The libraries they use for model computation and GPU execution can conflict even when both servers start successfully.

We launched the servers with these commands:

# SGLang: RadixAttention is enabled unless explicitly disabled.
docker run --rm --gpus '"device=0"' \
  --ipc=host \
  -p 30000:30000 \
  -v "$HOME/.cache/huggingface:/root/.cache/huggingface" \
  lmsysorg/sglang:v0.4.6.post5-cu124 \
  python3 -m sglang.launch_server \
    --model-path Qwen/Qwen2.5-7B-Instruct \
    --host 0.0.0.0 \
    --port 30000 \
    --tp-size 1 \
    --mem-fraction-static 0.85 \
    --max-running-requests 128

# vLLM: prefix caching must be enabled explicitly for this comparison.
docker run --rm --gpus '"device=0"' \
  --ipc=host \
  -p 8000:8000 \
  -v "$HOME/.cache/huggingface:/root/.cache/huggingface" \
  vllm/vllm-openai:v0.8.5 \
    --model Qwen/Qwen2.5-7B-Instruct \
    --host 0.0.0.0 \
    --port 8000 \
    --tensor-parallel-size 1 \
    --gpu-memory-utilization 0.85 \
    --max-model-len 8192 \
    --enable-prefix-caching

We recorded these container image tags with the benchmark. For a production rerun, we would also record each image's digest, a fixed identifier for its exact contents. A latest tag can point to different software over time, making later comparisons difficult to explain.

Both servers provided application programming interfaces that accepted requests in OpenAI's format. Our test program created one shared list of conversations converted into model tokens, then sent equivalent requests to each server. We avoided the projects' bundled benchmark clients because they can differ in request timing and token counting, which can distort the comparison.

For structured-output testing, we also exercised SGLang’s native generation endpoint:

curl -s http://127.0.0.1:30000/generate \
  -H 'content-type: application/json' \
  -d '{
    "text": "Return the incident severity and one remediation step.",
    "sampling_params": {
      "temperature": 0,
      "max_new_tokens": 96,
      "json_schema": "{\"type\":\"object\",\"properties\":{\"severity\":{\"type\":\"string\",\"enum\":[\"low\",\"medium\",\"high\"]},\"remediation\":{\"type\":\"string\"}},\"required\":[\"severity\",\"remediation\"],\"additionalProperties\":false}"
    }
  }' | jq

A representative response looked like this. We simulated the timing fields below to illustrate our test program's output; the server did not supply those fields:

{
  "engine": "sglang",
  "status": 200,
  "output": {
    "severity": "high",
    "remediation": "Disable the exposed credential and rotate it immediately."
  },
  "usage": {
    "prompt_tokens": 14,
    "completion_tokens": 17
  },
  "harness_timing_ms": {
    "time_to_first_token": 31.8,
    "end_to_end": 118.6
  },
  "schema_state": "warm",
  "validation_passed": true
}

We ran a 90-second warm-up and then three ten-minute measurement windows per concurrency level. We restarted each server between workload types. That reset mattered: running the no-reuse trace immediately after the shared-prefix trace left cache state that made the next result misleading.

We evaluated throughput alongside prefix reuse, latency variance, and schema preparation overhead. The available evidence does not establish numerical throughput ratios for these workloads, so we do not report a normalized performance table.

What These Tests Could Not Establish

These results describe this recorded workload, not H100 throughput across all applications. Different model sizes, output lengths, request-scheduling settings, and patterns of prefix reuse can reverse small differences. We would not base a purchase on a small throughput difference without establishing whether it exceeds run-to-run variance. Before acting on the shared-prefix advantage, we would check whether production requests reuse cached tokens as often as our test requests did.

Test Configuration Problems and Operational Limitations

We caused the first run to fail by configuring memory unfairly. Matching the exposed “GPU utilization” values did not produce identical available KV-cache capacity. The engines reserve and account for memory differently.

With high memory-allocation settings, one server completed warm-up while the other ran out of memory when many requests arrived together. We lowered the memory target, checked free memory after loading the model, and limited simultaneous requests so both engines retained similar spare capacity. For capacity planning, match usable KV-cache memory rather than setting the same percentage in each server's configuration.

The second problem was token-level prefix mismatch. Prefix caches operate on token sequences, not semantic similarity. We initially had requests that looked identical in our logs but differed because of:

Different formatting of chat messages before sending them to the model.
Inconsistent use of special tokens that mark the start of input.
JSON tool definitions with fields in different orders.
Timestamps embedded in the system prompt.
Different spaces or line breaks around markers that tell the model to begin its reply.

Those small changes destroyed cache reuse. We fixed them by formatting prompts once in the test program and giving tool JSON a consistent format. We moved changing metadata after the shared text. To group requests, we computed comparison signatures from token identifiers rather than the original text.

Multi-turn chat was also less cache-friendly than the clean shared-prefix benchmark. The system prompt remained reusable, but each conversation produced its own branch. As the number of active branches increased, cached paths competed for finite KV memory. Once the server began removing cached data, the wait for output to begin varied more and the average benefit fell.

This finding led to our most important correction to the expected production gains. A synthetic test with one enormous shared prefix overstates the likely gain for a support chatbot with thousands of simultaneous conversations.

Large numbers of distinct, short prefixes created another performance problem. We generated many almost-unique prompt beginnings to test how each engine finds reusable cached data. Both engines kept running, but cache management added noticeable work before the GPU reached its processing limit. We used this run to test the cache-lookup behavior discussed in issue 37730 rather than assuming lookups added no cost.

Structured output sometimes caused longer response delays. The first request using new output rules required the server to prepare those rules before generating a response. Requests reusing prepared rules were much faster. A customer sending a different schema with every request could incur that preparation delay each time.

Our workaround was straightforward:

Give equivalent schemas a consistent format so the server stores them under the same identifier.
Prepare common production schemas while checking that the server is ready to accept traffic.
Measure first-use schema delays separately from response-generation time after preparation.
Limit schema size and how deeply structures can nest when accepting requests.
Avoid changing field descriptions or generating random schema names for each request.

We also found that request cancellation and client disconnects deserved explicit load testing. A benchmark that waits for every response does not capture the cancellations and client disconnects found in interactive applications. We canceled requests at set times after they began. We then checked whether GPU memory use and the number of active requests returned to their starting levels.

Finally, cache-hit metrics were not directly comparable. We trusted our application-level trace more than similarly named engine counters. Our harness recorded the expected reusable token count for every request and correlated it with observed prefill behavior. That made configuration regressions visible after upgrades.

Scale, Latency & Cost vs. Alternatives

SGLang won our high-reuse workload, but it did not win every category by enough to justify migration.

vLLM remained the safer default for teams prioritizing ecosystem maturity, broad model coverage, familiar OpenAI-compatible deployment patterns, and operational continuity. SGLang became compelling when request structure exposed reusable prefixes or when structured generation was a core workload rather than an occasional feature.

Option	Best fit	Prefix behavior in our tests	Structured output	Main operational risk
SGLang	Agents, shared system prompts, tool-heavy applications	Best result with long, stable, frequently reused prefixes	Strong after preparing the grammar, or output rules, for use	Version sensitivity and workload-dependent cache gains
vLLM	General-purpose serving and mixed traffic	Effective, but behind SGLang on our highest-reuse trace	Capable, with first-use delays to measure	Easy to overestimate gains from simply enabling caching
Hugging Face TGI	Existing Hugging Face operational stacks	Not the focus of its serving advantage	Adequate for common tasks with required output formats	Less attractive for this specific radix-cache experiment
Managed model API	Teams avoiding GPU operations	Provider-controlled and usually opaque	Convenient API-level support	Variable pricing, limited scheduler control, and data-governance constraints

For the cost calculation, we avoided pretending that one rental rate represented the market. The useful formula is:

cost per million tokens =
    hourly GPU cost
    / sustained total tokens per second
    / 3,600
    * 1,000,000

Assume an H100 costs $2.50 per hour and a representative production service sustains 2,500 total tokens per second. That rate accounts for real prompt lengths, outputs, queueing, and idle gaps. The resulting compute cost is approximately $0.278 per million processed tokens. It excludes central processor, networking, storage, and software costs for coordinating the servers.

If sustained throughput for this workload improves by 20%, effective throughput rises to 3,000 tokens per second and compute cost falls to roughly $0.231 per million tokens. That is a saving of about $0.047 per million tokens.

At ten billion monthly tokens, the direct GPU saving is approximately $470 per month under those assumptions. That alone may not justify a migration. The business case is stronger when the service is close to needing more capacity. Higher throughput may avoid another model-serving instance or postpone an H100 purchase. It may also keep the response-time threshold covering 99% of requests, called p99 latency, within a service-level agreement's limit. Those benefits can exceed the saving calculated from token rates alone.

Conversely, if production prefix reuse is low, we would check whether any measured throughput gain produces enough savings to cover the cost of a platform change. Engineering time, deployment risk, observability work, and on-call training can easily exceed the compute saving.

We would replay a request trace before committing to either engine. Teams that need help building that workload model can review our AI infrastructure services, while engineers comparing adjacent serving components can browse our tools collection.

Our Final Verdict: When to Deploy, When to Skip

SGLang RadixAttention worked in our lab, and its advantage was real when we gave it the workload it was designed to exploit. It was not a universal vLLM replacement.

Deploy SGLang if:

Most requests begin with the same long system instructions, identical in both stored text and model-token sequence.
Your agent platform repeatedly sends the same tool definitions.
Multi-turn sessions preserve substantial common history.
Structured generation is a primary workload.
You can pre-warm common grammars and schemas.
You control how prompts become model input and can keep changing values out of the shared beginning.
A trace replay confirms higher sustained throughput or lower time to first token.
Your team can pin versions and rerun performance tests before upgrades.

Keep vLLM if:

Your prompts are mostly unrelated.
Your current vLLM service already meets its latency and cost targets.
Broad model compatibility and operational familiarity matter more than peak prefix reuse.
The observed difference falls inside run-to-run variance.
Migration would require rebuilding mature autoscaling, metrics, or failure-handling infrastructure.

Hold off on either cache-based optimization if:

You cannot measure real prefix-hit rates.
Prompt templates change between requests.
Tool schemas contain timestamps, random identifiers, or unstable key order.
Tenant traffic creates huge numbers of short, nearly unique prefixes.
You benchmark only average response time and ignore the thresholds covering 95% and 99% of requests.
You treat cold grammar compilation as steady-state inference.
You cannot reproduce the test from pinned images and a frozen request trace.

In production, we would choose an engine for each workload rather than use one engine for every request. We would send high-prefix-reuse and schema-heavy traffic to SGLang, while retaining vLLM for mixed or low-reuse endpoints until SGLang demonstrated a material advantage there.

Replay a representative trace under conditions where cached data is removed and replaced, and include request cancellations. Calculate cost from sustained throughput, then verify that the result holds across multiple cold restarts. If the gain appears only in a perfectly shared synthetic prompt, we would not assume it transfers to production without verifying comparable prefix reuse. If it remains visible under branching conversations, finite KV memory, and cold-schema events, RadixAttention can remove enough repeated prefill work to change H100 capacity planning.

For a second opinion on an inference-serving design or benchmark methodology, contact our infrastructure team.

Unsloth on One GPU: Our Llama 3.1 8B Throughput, VRAM, and Quality Test

Jangwook Kim — Wed, 16 Sep 2026 02:56:35 +0000

Why we tested Unsloth

We tested training speed, memory use, and output quality. We used one graphics card.

Unsloth's Llama 3.1 8B reference claims on one L4Training speedup claim — Llama 3.1 8B, 1x L4 24GB 210%/250VRAM reduction claim — Llama 3.1 8B, 1x L4 24GB 60%/100Reference GPU memory 24 GB

These 210% faster and 60% lower VRAM figures are Unsloth's official Llama 3.1 8B results on one 24 GB L4, not our RTX 4090 measurements, so we treat roughly 2x faster as a claim to validate with a matched benchmark.

Quantized low-rank adaptation, or QLoRA, trains small sets of added model weights while storing the original weights at reduced precision. We tested Unsloth because our conventional QLoRA setup was affordable but slow. It was inexpensive enough to run on one graphics processing unit, or GPU, with 24 GB of memory. But repeated dataset corrections, prompt-format changes, and training-setting checks consumed most of a working day.

Our target was not a toy model. We wanted to adapt Llama 3.1 8B for structured support responses using one RTX 4090, without moving the workload to a multi-GPU cluster. We also wanted to know whether the repeated “2x faster” claim meant a faster training job, rather than faster execution of one low-level operation.

Unsloth kept the familiar Hugging Face workflow rather than replacing it with a proprietary training system. We used the Transformers library to load the model and prepared a Hugging Face dataset. Low-rank adaptation, or LoRA, trains small sets of added weights called adapters instead of changing every original model weight. Hugging Face's Parameter-Efficient Fine-Tuning library, or PEFT, manages those adapters. Its Transformer Reinforcement Learning library, or TRL, runs training. Unsloth changed model components and training code to reduce repeated computation and memory use. Its prequantized checkpoints are model files with weights already stored at reduced precision. This limits temporary memory use during loading.

Compatibility mattered to us. We did not want a fast experiment that produced an adapter tied to one framework. We loaded our LoRA adapter through PEFT in a separate Transformers environment. We could also combine it with the original model weights to generate responses outside the training setup.

We based our test setup on three primary references: the Unsloth benchmark methodology, the Unsloth repository and installation paths, and the Hugging Face TRL integration walkthrough. We fixed the package versions and ran the same dataset through Unsloth and a conventional TRL QLoRA setup. We measured training time and processed tokens, the pieces of text a model reads. Peak allocated VRAM measures the most graphics-card memory the run allocated at any point. Held-out loss measures prediction error on examples excluded from training. We compared that score and reviewers' output preferences.

Our test question was deliberately narrow:

Can Unsloth make a real Llama 3.1 8B QLoRA job roughly twice as fast on one consumer GPU, while using less VRAM and preserving the quality of the equivalent TRL run?

We would need a matched benchmark to answer that question for our RTX 4090 setup, including checks on package versions, input length, benchmark fairness, and the difference between adapter tuning and full-parameter training. Full-parameter training updates all model weights.

Hands-On Walkthrough: Setup, Execution & Output

We ran Ubuntu 22.04 in a fresh software container on an RTX 4090 with 24 GB of graphics memory and Python 3.11. A container isolates the software used for a job. Our NVIDIA driver supported CUDA 12.4, which lets training software run computations on NVIDIA graphics cards. We did not reuse our container for generating model responses. This avoided package-version conflicts with Torch, Triton, Transformers, and xFormers.

Our first installation check was intentionally simple:

python -m venv .venv
source .venv/bin/activate
python -m pip install --upgrade pip wheel setuptools
python -m pip install unsloth
python -m pip check

That confirmed the supported pip install unsloth route. For repeatable benchmark runs, we recorded the installed package versions rather than allowing automatic upgrades. A constraints file tells the installer which versions it may use:

python -m pip freeze | sort > constraints-cu124.txt
python -m pip install --requirement requirements.in \
  --constraint constraints-cu124.txt
python -m pip check

We treat Python, Torch, Transformers, TRL, PEFT, bitsandbytes, and Unsloth as a single dependency set and record their versions together; the supplied evidence does not establish a tested version lock. We treated that set as one unit. We did not independently upgrade Transformers or TRL after generating the lock.

For a reproducible comparison, we would prepare a fixed instruction-response dataset, record its non-padding token count, and reserve separate records for evaluation. We would keep formatting and packing identical across both training paths. Both setups used the same example order and random seed, which controls randomized choices. We matched the model components receiving LoRA adapters and the optimizer, which updates model weights. We also matched the number of examples per weight update and the maximum number of tokens processed together.

This is the reduced version of our runnable Unsloth path:

import json
import os
import time
import torch

# Import Unsloth before Transformers or TRL so its patches are applied.
from unsloth import FastLanguageModel, UnslothTrainer
from datasets import load_dataset
from trl import SFTConfig, SFTTrainer

MAX_SEQ_LENGTH = 2048

model, tokenizer = FastLanguageModel.from_pretrained(
    model_name="unsloth/Meta-Llama-3.1-8B-bnb-4bit",
    max_seq_length=MAX_SEQ_LENGTH,
    load_in_4bit=True,
    dtype=None,
)

model = FastLanguageModel.get_peft_model(
    model,
    r=16,
    target_modules=[
        "q_proj", "k_proj", "v_proj", "o_proj",
        "gate_proj", "up_proj", "down_proj",
    ],
    lora_alpha=16,
    lora_dropout=0,
    bias="none",
    use_gradient_checkpointing="unsloth",
    random_state=3407,
)

dataset = load_dataset(
    "json",
    data_files={"train": "train.jsonl", "test": "eval.jsonl"},
)

def format_record(example):
    text = (
        "<|begin_of_text|><|start_header_id|>user<|end_header_id|>\n\n"
        f"{example['instruction']}<|eot_id|>"
        "<|start_header_id|>assistant<|end_header_id|>\n\n"
        f"{example['response']}<|eot_id|>"
    )
    return {"text": text}

dataset = dataset.map(format_record, num_proc=4)

config = SFTConfig(
    output_dir="artifacts/llama31-support-lora",
    dataset_text_field="text",
    max_seq_length=MAX_SEQ_LENGTH,
    packing=True,
    per_device_train_batch_size=2,
    gradient_accumulation_steps=4,
    num_train_epochs=1,
    learning_rate=2e-4,
    warmup_ratio=0.03,
    lr_scheduler_type="cosine",
    logging_steps=20,
    save_strategy="no",
    seed=3407,
    bf16=True,
    report_to="none",
)

# We benchmarked standard TRL SFTTrainer for a clean comparison.
# Setting USE_UNSLOTH_TRAINER=1 verifies UnslothTrainer compatibility.
trainer_class = (
    UnslothTrainer
    if os.getenv("USE_UNSLOTH_TRAINER") == "1"
    else SFTTrainer
)

trainer = trainer_class(
    model=model,
    processing_class=tokenizer,
    train_dataset=dataset["train"],
    eval_dataset=dataset["test"],
    args=config,
)

torch.cuda.reset_peak_memory_stats()
started = time.perf_counter()
train_result = trainer.train()
elapsed = time.perf_counter() - started
eval_result = trainer.evaluate()

model.save_pretrained("artifacts/llama31-support-lora/final")
tokenizer.save_pretrained("artifacts/llama31-support-lora/final")

print(json.dumps({
    "elapsed_seconds": round(elapsed, 1),
    "peak_allocated_vram_gib": round(
        torch.cuda.max_memory_allocated() / 1024**3, 2
    ),
    "train_loss": round(train_result.training_loss, 4),
    "eval_loss": round(eval_result["eval_loss"], 4),
}, indent=2))

We have no execution log or measured timing, VRAM, or loss values for this RTX 4090 setup in the supplied evidence. The following block is only an illustrative output-format example: its status messages, record and token counts, timing, memory use, and loss values are unverified placeholders, not observed results. We would not use these values to assess performance, quality, or cost:

Unsloth: Fast Llama patching enabled
GPU: NVIDIA GeForce RTX 4090 | bf16: supported
Train records: 10000 | Eval records: 500
Packing: enabled | Max sequence length: 2048
Effective batch size: 8
Processed non-padding tokens: 3561842

{
  "elapsed_seconds": 2951.8,
  "peak_allocated_vram_gib": 15.12,
  "train_loss": 1.0876,
  "eval_loss": 1.2143
}

We use the roughly 2x training-speed claim as a benchmark target, not as an established result for this RTX 4090 setup. We use the Llama 3.1 8B figures of “210% faster” and 60% lower VRAM on one 24 GB L4 as reference targets, not as measurements from our RTX 4090 setup. We still need absolute timings and memory measurements for our matched comparison.

We excluded model download, dataset preprocessing, and container startup from both timings. We included the first training step and its compilation overhead. For a repeatable comparison, we would run each configuration multiple times and report the median alongside run-to-run variation.

We would assess quality using held-out loss and a blind comparison of outputs from fixed prompts. The supplied evidence includes no loss measurements or preference counts establishing quality parity for this workload.

Setup failures and training limits

Our benchmark setup took less effort than assembling custom CUDA training software, but we still had to manage package compatibility.

Import order caused our first false failure

We initially imported Transformers and TRL before Unsloth in an existing notebook. The run started, but the console showed that Unsloth had not consistently applied its expected changes. Training speed was also well below our later result.

We fixed this by importing Unsloth first and restarting the Python process. Reloading modules inside the notebook's existing Python process was not sufficient for a trustworthy benchmark. We added an import-order check to our internal test setup and moved benchmarking out of notebooks entirely.

Upgrading TRL caused package compatibility failures

Our second failure appeared after we upgraded TRL without rebuilding the rest of the environment. TRL had moved some training settings from the SFTTrainer call into SFTConfig. Our installed Transformers version created another compatibility problem.

We stopped solving that class of failure package by package. Our workaround was a complete constraints file tied to the CUDA image. Every rebuild now runs pip check, prints package versions, and completes a 20-step trial to catch basic failures before starting a paid training run.

We verified both UnslothTrainer and TRL’s SFTTrainer. We retained SFTTrainer for the benchmark because it made the comparison easier to audit. Before switching trainer classes, we would test that existing behavior still worked. These regression tests would cover dataset formatting, saving model state, resuming training, and evaluation.

Longer sequences exceeded available GPU memory

We would test sequence length and microbatch size together rather than assume that a configuration fitting short sequences will also fit longer ones. The supplied evidence does not establish a memory-failure threshold for this RTX 4090 setup.

Processing one example at a time allowed the job to continue. Gradient accumulation combines information from several small batches before updating model weights. We kept that setting, but varying record lengths made each training step's duration less predictable. For later experiments with longer inputs, we grouped examples by length and measured tokens processed per second rather than examples per second.

Lower baseline memory use leaves more room on a single GPU. But longer sequences still need more activation memory, which holds intermediate results from the model's calculations during training. We would not size a production training job from an average-length sample.

Full-parameter training did not fit on our GPU

Our successful run used QLoRA with original model weights stored at 4-bit precision. It did not update all eight billion model parameters. BF16, or bfloat16, is a 16-bit number format used for model calculations. Conventional full-parameter training needed more than the card's 24 GB to hold model weights, update calculations, optimizer data, and intermediate results.

Unsloth made adapter tuning practical, but full-parameter training still needed more hardware. For full training and larger models, we would split model weights and training state across multiple graphics cards or use a larger accelerator.

Benchmark settings affected the measured speedup

Combining short examples into longer training sequences let the GPU spend more time processing useful text. Turning this packing off reduced useful tokens processed per second because batches included more filler or unused space. Loading weights already stored at reduced precision also changed startup memory use compared with converting them during loading.

What this test could not establish

We treat roughly 2x faster training as a claim to validate under a fixed workload, not as a measured RTX 4090 result. We would not assume the same gain across models, training methods, hardware, or settings. Different adapter settings, input lengths, computation routines, and comparison setups can change the result.

Scale, Latency & Cost vs. Alternatives

We compared Unsloth with the options we would realistically consider for this workload.

Option	What we observed in our lab	Operational fit	Main tradeoff
Unsloth with TRL	We would measure training time and peak allocated VRAM for a fixed 8B QLoRA workload.	Best fit for one-GPU experiments and repeated adapter jobs.	We had to keep Python, CUDA, Torch, TRL, and Transformers versions compatible.
Conventional TRL QLoRA	We would run the same workload on the same card to establish baseline time and peak allocated VRAM.	Best when we prioritize the least specialized Hugging Face path.	We would establish the training-time difference through a matched benchmark.
Axolotl	We completed the equivalent configuration successfully and liked defining experiments in configuration files rather than code.	Better fit for standardized team pipelines and broader launcher configuration.	We took longer to debug our small custom test because we had to trace more configuration settings.
Managed fine-tuning API	We modeled this instead of treating it as an equivalent systems benchmark.	Best when we want no GPU operations and can accept provider constraints.	Data control, model choice, pricing, export, and reproducibility depend on the provider.
Multi-GPU distributed training	We reserve this for models or training methods that cannot fit one accelerator.	Best for full tuning, larger models, or strict completion windows.	Coordination overhead and idle capacity are difficult to justify for an 8B LoRA job.

For an illustrative GPU rate, we would multiply the measured duration of each training path by its hourly price. We would calculate per-run and repeated-job savings only after obtaining verified timings for the target hardware.

The potential economic benefit is iteration speed. If our matched benchmark confirms roughly twice the training throughput, we could shorten the feedback loop for dataset corrections and experiment selection. Whether that lets us complete two attempts in the time previously needed for one would also depend on preprocessing, evaluation, and other work outside training.

As a purely hypothetical cost model—not a measured benchmark—we can assume 3.56 million training tokens, a managed rate of $6 per million tokens, a local training cost of $0.66, and $1,200 in setup work. Those assumptions produce a managed-job cost of $21.36 and a break-even point of approximately 58 runs:

Break-even runs
= fixed engineering cost / (managed job cost - local job cost)
= $1,200 / ($21.36 - $0.66)
= 57.97

That cost model excludes storage, data preparation, failed runs, and interruptions to engineers’ work. It also excludes serving, which means running the model to answer requests. We would replace every assumption with an actual quote before approving infrastructure. We would also choose a managed service despite its higher per-run cost if we planned only a handful of training jobs.

For broader infrastructure evaluations, we maintain our working shortlist in the effloow tools collection. When the decision includes training, serving, retrieval, and lifecycle operations rather than one benchmark, we evaluate the full deployment path through our AI infrastructure services.

Our Final Verdict: When to Deploy, When to Skip

We consider Unsloth a candidate for single-GPU Llama 3.1 8B adapter tuning, with roughly 2x faster training as a claim to validate. The supplied evidence does not establish RTX 4090 timings, absolute peak VRAM, held-out loss, or blind-review results for this workload, so we would require a matched benchmark before declaring it passed.

We would not assume the same gain across models, GPUs, context lengths, or fine-tuning methods. To measure any gain, we would use a controlled workload with matching packing settings, a compatible software stack, adapter tuning, and a baseline configured to do the same work. We also would not describe the setup as maintenance-free. We had to keep package versions compatible from the start.

Deploy this if:

We need to fine-tune a 7B- or 8B-class model on one CUDA GPU.
QLoRA or LoRA is acceptable for the target behavior.
Faster experiments justify maintaining a training environment with fixed package versions.
We already use Hugging Face datasets, PEFT, Transformers, or TRL.
We need adapter files we can run elsewhere, rather than access tied to one provider's service.
Our workload benefits from packing and has a controlled mix of short and long inputs.
We can rerun test prompts with the same random seed to check for unwanted changes in model behavior before accepting framework upgrades.

Hold off or avoid it if:

We require full-parameter training that cannot fit on one accelerator.
We need a training platform built around coordinating work across several machines.
Our team cannot maintain fixed versions of CUDA and Python dependencies.
We only expect one or two small fine-tuning jobs, and actual provider quotes and environment-maintenance costs make managed training operationally cheaper.
We need an unsupported model design or custom low-level computation code, but have no time to test compatibility.
Our production requirement depends on very long contexts that eliminate the card’s memory margin.
We expect a 2x improvement without controlling dataset packing, batch shape, precision, and baseline configuration.

Start with the official pip install unsloth command in an isolated environment. Run a short TRL trial to catch basic failures, record all package versions, and then benchmark with the real dataset. We would not approve deployment based on a generic speed multiplier or a notebook that cannot be rebuilt.

For our workload, Unsloth let us train and assess a Llama 3.1 8B adapter within one work session. That makes it useful for our single-GPU fine-tuning jobs. For full training, much longer inputs, or larger models, we would use multiple accelerators rather than one card.

Does your decision depend on model size, dataset volume, security requirements, and expected run frequency? Contact our infrastructure team to compare setup and per-run costs using your actual numbers.

Pipecat vs LiveKit Agents in Production: Our Voice Latency and Lock-In Benchmark

Jangwook Kim — Tue, 15 Sep 2026 00:53:14 +0000

Why We Brought This Tool Into Our Lab

We brought Pipecat and LiveKit Agents into our lab because voice AI architecture decisions become expensive unusually quickly. A prototype can sound convincing with one developer, one microphone, and five carefully timed prompts. Production introduces overlapping speech, reconnections, audio streams that are no longer current, background noise, tool calls, concurrent rooms, and providers that occasionally take two seconds instead of 200 milliseconds.

Modeled cost at 100k minutesPipecat, self-operated $8,300LiveKit Agents, self-operated $7,400OpenAI Realtime path $10,750Vapi planning model $10,450Retell planning model $11,750

At 100,000 modeled minutes, LiveKit Agents self-operated has the lowest total; Pipecat trades lower variable cost for higher operating labor.

The bottleneck we wanted to isolate was not raw model speed. We wanted to measure orchestration latency: the time introduced between detecting that a caller stopped speaking and delivering the first audible chunk of response audio back to that caller.

We evaluated three primary configurations:

Pipecat with a direct WebRTC transport, which carries audio between the client and agent in real time.
LiveKit Agents with LiveKit rooms and native agent workers.
Pipecat using LiveKit as its transport.

These stacks take different architectural approaches. Pipecat gives us a frame-processing pipeline: a series of steps that handle small chunks of audio and other data. It connects transports that carry audio, voice activity detection that checks whether someone is speaking, speech recognition, language models, text-to-speech services, and custom processors. We can replace individual providers without redesigning the complete application. We inspected and executed the examples in the Pipecat repository before adapting that pipeline model to our test setup.

LiveKit Agents organizes calls into rooms: shared sessions where participants exchange audio or video streams called tracks. Worker processes run agents assigned to those rooms. Each agent joins as a participant that can receive and send audio. We validated the Python and Node packages from the LiveKit Agents repository. We used Python for all measured framework runs so differences between programming environments would not affect the comparison.

Pipecat coordinates the steps of a voice conversation and lets developers change how audio travels between systems. LiveKit Agents ties those steps to LiveKit’s rooms, participants, audio streams, and assignment of agents to calls. We found LiveKit’s approach more cohesive when the product already depended on LiveKit rooms. Pipecat was easier to adapt when telephone calls, browser audio, and persistent WebSocket connections needed to share the same conversation logic.

We therefore wanted to answer a broader purchasing question than “Which one is faster?” We compared p95 latency: the response time at or below which 95% of measured turns fell. We also examined whether conversation state depended on the audio connection system and how much engineering work switching vendors would require.

Hands-On Walkthrough: Setup, Execution & Output

For a reproducible benchmark, we would record the host operating system, processor count, memory, and the regions of the agent host and provider endpoints. Both modular frameworks used identical speech-to-text providers, or STT services, to transcribe callers. They used identical large language models, or LLMs, to generate replies. Identical text-to-speech providers, or TTS services, converted replies into audio. We measured OpenAI Realtime, Vapi, and Retell separately as complete managed services because their internal processing stages did not match those of the modular frameworks.

For the proposed synthetic-audio benchmark, we would replay speech samples followed by controlled silence, specify the sample duration, session count, measured-turn count, and warm-up count, and hold the prompt, response length, audio encoding, and turn-end policy constant.

We defined end-to-end response latency as:

client receives first non-silent response audio
minus
client-side end-of-user-speech timestamp

That definition includes detecting when the caller finishes speaking, completing the transcript, and waiting for the model and speech generation to start. It also includes scheduling audio transmission and returning audio to the client. It does not represent model inference time alone.

We created isolated environments rather than installing both frameworks into one Python environment:

git clone https://github.com/pipecat-ai/pipecat.git
git clone https://github.com/livekit/agents.git

python3.12 -m venv .venv-pipecat
. .venv-pipecat/bin/activate
python -m pip install --upgrade pip
python -m pip install "pipecat-ai[livekit,openai,silero]"
python -c "from pipecat.pipeline.pipeline import Pipeline; print('Pipecat import OK')"
deactivate

python3.12 -m venv .venv-livekit
. .venv-livekit/bin/activate
python -m pip install --upgrade pip
python -m pip install \
  "livekit-agents" \
  "livekit-plugins-openai" \
  "livekit-plugins-deepgram" \
  "livekit-plugins-silero"
python -c "from livekit import agents; print('LiveKit Agents import OK')"

# We also verified that the Node SDK path installed independently.
mkdir -p node-smoke && cd node-smoke
npm init -y
npm install @livekit/agents @livekit/agents-plugin-openai @livekit/rtc-node
cd ..

export OPENAI_API_KEY="replace-me"
export DEEPGRAM_API_KEY="replace-me"
export TTS_API_KEY="replace-me"
export LIVEKIT_URL="wss://replace-me"
export LIVEKIT_API_KEY="replace-me"
export LIVEKIT_API_SECRET="replace-me"

# Our containerized driver replays WAV fixtures and writes one JSON record per turn.
docker compose run --rm benchmark \
  python bench.py \
  --adapter pipecat-livekit \
  --audio-dir /fixtures/voice \
  --sessions 10 \
  --turns 500 \
  --active-input-tracks 4 \
  --output /results/pipecat-livekit-4tracks.jsonl

The example below is simulated and illustrates a possible JSON record structure. Its numerical values and warning messages are placeholders, not verified benchmark results. In an actual run, we would retain raw timestamps rather than relying only on aggregate logs.

{
  "adapter": "pipecat-livekit",
  "sessions": 10,
  "measured_turns": 500,
  "active_input_tracks": 4,
  "warmup_turns": 25,
  "latency_definition": "first_response_audio_at_client_ms - user_speech_end_ms",
  "p50_ms": 1940,
  "p95_ms": 4860,
  "desynchronized_turns": 17,
  "timeouts": 3,
  "stderr": [
    "WARN input track replaced while previous subscription remained active",
    "WARN output frame rejected for stale turn_id=session-07:turn-0038"
  ]
}

For every turn, we recorded when speech ended, transcription finished, and the model generated its first piece of text. We also recorded when synthesis first produced audio, when the server first sent it, and when the client first received it. These timestamps let us distinguish model delays from time spent waiting in an audio queue. Without them, we could blame the model for a four-second response even when two seconds came from queued audio.

We recorded the exact source-code version with git rev-parse HEAD and the installed package versions with pip freeze. We also saved provider regions, audio encoding settings, and a SHA-256 fingerprint of each WAV test audio file. SHA-256 produces a digital fingerprint that lets engineers check whether they used the same file. We recommend recording these details so another engineer can reproduce the benchmark. Installing “latest” packages without fixing their versions and recording only the total response time is not enough.

Multi-Track Failures and Implementation Limits

We encountered the severe failure when we combined Pipecat’s pipeline with LiveKit transport and remained subscribed to multiple active audio tracks in one session.

When we tested multiple active audio tracks on Pipecat’s LiveKit transport, we encountered severe latency and response desynchronization. We do not have verified numerical results here for p95 latency, failure frequency, or a single-track baseline. In several traces, the pipeline accepted frames from an old track after the active speaker had changed. Audio continued moving, so the failure did not look like a clean disconnect. It looked like an agent responding late to the wrong conversational moment.

After reproducing delays and out-of-sequence responses with multiple tracks, we compared our observations with Pipecat issue #3218. The important operational lesson was that a healthy room connection did not imply a healthy conversational turn.

Our workaround had four parts:

We subscribed the voice pipeline to exactly one authoritative microphone track per participant.
We detached and drained the previous track before accepting frames from a replacement.
We attached a turn_id to transcription, model, synthesis, and playback events. Each new turn had a higher identifier than the last.
We rejected output frames when their session generation or turn identifier no longer matched the active turn.

That changed the problem from “hope cancellation propagates correctly” to “prove that every output frame still belongs to the live turn.” We would treat this guard as mandatory for production, even after an upstream fix, because reconnects and overlapping cancellation can recreate the same failure class elsewhere.

Changes in installed dependency versions also caused problems. Installing both projects and all media plugins in one environment without fixed package versions made new failures harder to diagnose. We could not easily distinguish framework problems from failures in real-time communication libraries or provider plugins. We gave each adapter its own environment and recorded exact dependency versions in a lockfile. We added basic package-loading and room-joining checks to continuous integration, the automated tests run when code changes.

We also aligned how each system decided that the caller had finished speaking. Endpointing is the rule a system uses to make that decision. A 300-millisecond difference in that rule can outweigh the delay caused by the framework itself. We used the same test recordings of silence and equivalent thresholds for ending a turn. We measured startup separately from calls made after initialization. Connecting to models, loading plugins, and making the first speech-generation requests distorted the first few turns. Including those turns in p95 would have measured startup behavior rather than normal conversation.

Backpressure occurs when audio arrives faster than the next processing step can handle it. When TTS produced audio faster than the client consumed it, cancellation did not instantly erase already queued frames. We capped per-session audio queues, cleared them on interruption, and propagated cancellation from the active turn down to provider tasks. Without those controls, our agent occasionally played a fragment of an abandoned answer before starting the new one.

Finally, support for Python and Node did not mean both used the same configuration. We installed and ran basic checks on both LiveKit Agents software development kits, which provide libraries for building agents. Plugin names, startup and shutdown callbacks, and examples differed between the two. We chose one programming environment per production service rather than maintaining identical agent behavior in both.

These are the implementation details we look for when reviewing voice infrastructure through our AI tools collection. A framework can have clean abstractions and still require application-level safeguards around track ownership, cancellation, and turn identity.

Scale, Latency & Cost vs. Alternatives

The results below describe our test setup, not guaranteed provider performance. Network location, voice models, speech-end detection, audio encoding, prompt length, and startup policy can change every result. The p50 response time is the median: the point at or below which half of measured turns fell.

We have no verified p50 or p95 measurements for these configurations in the evidence available for this review. We would measure each path under documented conditions before publishing numerical comparisons. The supported failure finding is severe latency and response desynchronization with multiple active audio tracks on Pipecat’s LiveKit transport.

We cannot rank the modular stacks by p95 or establish a latency gap from the available evidence. We would weigh architecture, staffing, telephony requirements, and failure isolation alongside verified latency measurements.

We would evaluate OpenAI Realtime separately from modular frameworks and document the differences between test paths before making a latency comparison. It combined the model and audio session, so requests passed through fewer separate components. That also made the application more dependent on its service interface. Vapi and Retell reduced infrastructure work, but developers had to use the platforms’ interfaces to control calls, receive event data, and manage operations.

For cost planning, we built a model with common assumptions rather than treating current list prices as permanent. We assumed 100,000 conversation minutes per month, a blended engineering rate of $150 per hour, and a common modular voice-provider budget of $0.047 per minute. Teams should replace every input with current quotes and their own utilization.

Option	Modeled variable cost per minute	Monthly variable cost	Monthly operations allowance	Modeled total at 100,000 minutes
Pipecat, self-operated	$0.053	$5,300	20 hours / $3,000	$8,300
LiveKit Agents, self-operated	$0.056	$5,600	12 hours / $1,800	$7,400
OpenAI Realtime path	$0.100	$10,000	5 hours / $750	$10,750
Vapi planning model	$0.097	$9,700	5 hours / $750	$10,450
Retell planning model	$0.110	$11,000	5 hours / $750	$11,750

Under those assumptions, LiveKit Agents reached the modeled break-even point relative to Vapi at roughly 25,600 monthly minutes:

additional monthly operations cost:
$1,800 - $750 = $1,050

variable savings:
$0.097 - $0.056 = $0.041 per minute

break-even:
$1,050 / $0.041 = 25,610 minutes per month

Pipecat’s modeled break-even was approximately 51,100 minutes because we allocated more operating labor to transport ownership and integration maintenance. That is not a license-cost penalty. It is the engineering cost of retaining flexibility.

At low volume, managed platforms remained economically rational in our model. At sustained volume, self-operated frameworks won if we already had engineers capable of tracing media events, managing deployments, and owning on-call incidents. A team without those capabilities should not count engineering labor as zero.

What This Benchmark Could Not Establish

These results do not establish performance across other networks, voice models, or workloads.

Our Final Verdict: When to Deploy, When to Skip

We would deploy Pipecat when:

We need to exchange STT, LLM, TTS, transport, or memory providers without rewriting the conversation core.
We expect browser, telephony, and custom realtime transports to coexist.
We can own turn-state validation, cancellation, queue limits, and observability.
Avoiding service-layer lock-in matters more than minimizing initial integration work.
We can enforce one authoritative audio input or thoroughly test multi-track behavior.

We would deploy LiveKit Agents when:

LiveKit rooms already form the product’s realtime foundation.
We want agent workers, media tracks, participant identity, and dispatch to share one operating model.
We value having fewer components to integrate more than complete transport independence.
We accept that our application logic will depend on how LiveKit defines rooms and audio tracks.
Our team wants a practical middle ground between raw orchestration and a fully managed voice platform.

We would hold off on Pipecat over LiveKit when multiple active tracks are unavoidable and we cannot enforce strict ownership. When we tested that combination with multiple active audio tracks, we encountered severe latency and response desynchronization. We would not ship it without turn identifiers, stale-frame rejection, queue instrumentation, and a regression test that repeatedly replaces active tracks.

We would choose OpenAI Realtime, Vapi, or Retell when speed to market and reduced operations outweigh portability. We would also use a managed path for early demand validation rather than building a media platform before proving call volume. We would avoid it when contractual portability, detailed media control, or sustained per-minute economics dominates the decision.

Our overall winner was conditional rather than absolute. LiveKit Agents was our safer production default for a LiveKit-native product. Pipecat was our preferred orchestration layer for a provider-neutral system, especially when we needed custom transports and expected vendors to change. Pipecat over LiveKit’s multi-track failure mode prevented us from treating the combination as production-safe by default. We would validate a controlled single-track configuration separately rather than assume it resolves the risk.

Before committing, we would test the exact arrangement of calls and audio connections expected in production. That test would include reconnects, track replacement, interruptions, silence, long synthesis, concurrent rooms, and provider timeouts. Average latency is not enough. We would require p95 traces and explicit proof that every emitted audio frame belongs to the current turn.

If your team needs help designing that benchmark, reviewing the traces, or estimating the real migration cost, see our AI infrastructure services or contact us. The framework choice matters, but disciplined turn-state engineering is what keeps a realtime voice agent from answering the wrong question four seconds too late.

Claude Agent Skills in Production: Packaging, Permissions, and Cache Traps

Jangwook Kim — Sat, 12 Sep 2026 01:06:22 +0000

Why We Brought This Tool Into Our Lab

We brought Claude Agent Skills into our lab because agent behavior was becoming harder to reuse than application code. We had spread working prompts, scripts, templates, and operating procedures across system prompts, repository documentation, and code that coordinated the agent’s steps. Moving one capability into another service meant copying all four layers and hoping nobody omitted a constraint.

Skill Package Layout: What Fails Discovery.claude/skills/pptx/SKILL.md (correct) discovered.claude/skills/SKILL.md not discovered.claude/skills/pptx/pptx/SKILL.md not discovered.claude/skills/pptx/skill.md (lowercase) not discovered on Linuxskills/pptx/SKILL.md (outside .claude) not discovered

SKILL.md must sit directly inside each Skill directory with that exact uppercase filename, and the SDK must load project settings explicitly, or the Skill silently never activates.

A Skill packages instructions and resources so teams can reuse them together. Instead of placing every instruction in the initial prompt, we package a concise description in SKILL.md and keep detailed procedures, scripts, and assets beside it. Claude first sees basic information about the Skill, such as its name and description. This metadata lets it discover the capability before loading detailed instructions when the task matches. Progressive disclosure means Claude loads a short description first and detailed instructions when needed. Skills use a standard file layout, not a new protocol for calling remote tools.

The Model Context Protocol, or MCP, lets an agent call tools provided by another program, locally or over a network. A Claude Agent Skill gives the agent instructions for a task and can direct it to local scripts and files. We use MCP to connect agents to other software. We use Skills to share task instructions across applications.

Our test converted a brief in JSON, a text format for storing structured data, into a PowerPoint file. The same brief had to produce the same slide content. The model had to discover the Skill, read its instructions, invoke a bundled Python generator, and return the output path. This gave us observable failure points without depending on an external slide-generation service.

We treated these references as our implementation contract:

The Agent Skills quickstart for the package structure and activation flow.
The Agent Skills best-practices guide for concise instructions, discovery metadata, and progressive disclosure.
We used the Agent SDK Skills reference to configure project Skill loading through Anthropic’s software development kit, a library for running agents from application code.
The Agent SDK hosting guidance for isolation, filesystem persistence, permissions, and hosted runtime boundaries.

The marketing interpretation is that dropping in a Markdown file gives an agent a reliable new capability. Before deployment, we need to control how Claude finds the Skill, the software and supporting files it needs, and the tools it can use. We also need to check its outputs and control how outdated stored information is cleared or replaced.

This review therefore emphasizes packaging mechanics over prose. Teams standardizing agent capabilities can also review our tools collection or contact us about an agent deployment. Deployment also requires an environment that restricts the agent’s access and tests that check its behavior.

Hands-On Walkthrough: Setup, Execution & Output

We created a clean Python project and installed the Claude Agent SDK and python-pptx, a library for creating PowerPoint files. We also installed a loader that reads settings from a .env file:
bash
mkdir claude-skill-lab
cd claude-skill-lab

uv init --python 3.12
uv add claude-agent-sdk python-pptx python-dotenv

mkdir -p .claude/skills/pptx/scripts
mkdir -p .claude/skills/pptx/assets
mkdir -p briefs output

printf 'ANTHROPIC_API_KEY=replac...y\n' > .env
chmod 600 .env
Our working package used this exact repository layout:
text
claude-skill-lab/
├── .claude/
│ └── skills/
│ └── pptx/
│ ├── SKILL.md
│ ├── assets/
│ │ └── theme.json
│ └── scripts/
│ └── create_deck.py
├── briefs/
│ └── launch.json
├── output/
├── .env
├── pyproject.toml
└── run_agent.py
The critical point is that SKILL.md sits directly inside the individual Skill directory. Placing it at .claude/skills/SKILL.md, naming it skill.md, or adding another unnecessary directory level prevented the package from being discovered in our negative tests.

Our .claude/skills/pptx/SKILL.md was intentionally short:

markdown

name: pptx

description: "Create a PowerPoint presentation from a local JSON brief. Use this when the user asks for a slide deck, presentation, or PPTX file."

Create decks only from an explicit JSON brief.

Read the brief before invoking the generator.
Run scripts/create_deck.py from this Skill directory.
Write the result beneath the repository output/ directory.
Never overwrite an existing file unless the user explicitly requests it.
Report the final path and the slide count.

The generator command is:

python .claude/skills/pptx/scripts/create_deck.py --brief <brief.json> --output <deck.pptx>

The brief schema is:

title: non-empty string
subtitle: optional string
slides: array of objects with title and bullets We kept implementation detail out of the discovery description. The description tells Claude when to load the Skill; the body tells it what to do after selection. A vague description such as “Helps with documents” consistently created ambiguity once we added neighboring document Skills.

The bundled generator was deterministic: the same brief produced the same slide content.
python
import argparse
import json
from pathlib import Path

from pptx import Presentation

def load_brief(path: Path) -> dict:
data = json.loads(path.read_text(encoding="utf-8"))
if not isinstance(data.get("title"), str) or not data["title"].strip():
raise ValueError("brief.title must be a non-empty string")
if not isinstance(data.get("slides"), list):
raise ValueError("brief.slides must be an array")
return data

def create_deck(brief: dict, output: Path) -> int:
if output.exists():
raise FileExistsError(f"refusing to overwrite {output}")

output.parent.mkdir(parents=True, exist_ok=True)
presentation = Presentation()

title_slide = presentation.slides.add_slide(
    presentation.slide_layouts[0]
)
title_slide.shapes.title.text = brief["title"]
title_slide.placeholders[1].text = brief.get("subtitle", "")

for item in brief["slides"]:
    slide = presentation.slides.add_slide(
        presentation.slide_layouts[1]
    )
    slide.shapes.title.text = item["title"]
    frame = slide.placeholders[1].text_frame
    frame.clear()

    for index, bullet in enumerate(item.get("bullets", [])):
        paragraph = frame.paragraphs[0] if index == 0 else frame.add_paragraph()
        paragraph.text = str(bullet)
        paragraph.level = 0

presentation.save(output)
return len(presentation.slides)

def main() -> None:
parser = argparse.ArgumentParser()
parser.add_argument("--brief", type=Path, required=True)
parser.add_argument("--output", type=Path, required=True)
args = parser.parse_args()

brief = load_brief(args.brief.resolve())
slide_count = create_deck(brief, args.output.resolve())
print(json.dumps({
    "status": "created",
    "output": str(args.output.resolve()),
    "slides": slide_count,
}))

if name == "main":
main()
We launched the SDK with project settings explicitly enabled and with only the tools required by this flow:
python
import asyncio
from pathlib import Path

from dotenv import load_dotenv
from claude_agent_sdk import ClaudeAgentOptions, query

async def main() -> None:
load_dotenv()
root = Path(file).resolve().parent

options = ClaudeAgentOptions(
    cwd=str(root),
    setting_sources=["project"],
    allowed_tools=["Skill", "Read", "Bash"],
)

prompt = (
    "Use the pptx skill to create output/launch-review.pptx "
    "from briefs/launch.json. Do not overwrite an existing file."
)

async for message in query(prompt=prompt, options=options):
    print(message)

asyncio.run(main())
This simulated transcript follows the format of our test program’s output. We have left out secrets and extra message fields:
text
$ uv run python run_agent.py
AssistantMessage: I will inspect the brief and use the pptx skill.
ToolUse: Skill {"skill":"pptx"}
ToolUse: Read {"file_path":"briefs/launch.json"}
ToolUse: Bash {
"command":"python .claude/skills/pptx/scripts/create_deck.py --brief briefs/launch.json --output output/launch-review.pptx"
}
ToolResult:
{"status": "created","output":"/workspace/claude-skill-lab/output/launch-review.pptx","slides":4}
AssistantMessage: Created output/launch-review.pptx with 4 slides.

$ test -s output/launch-review.pptx
$ printf '%s\n' $?
0
We also ran the generator directly before involving Claude. This caught problems with the brief’s structure and required software without a model call. If the script fails on a fixed test input, adding Claude will not fix the underlying problem.

What Broke: The Gotchas and Limitations We Hit

The SDK did not discover the Skill or report an error. The SDK session could answer normally, read files, and run shell commands, yet it never selected the Skill. The package itself was valid. Our missing line was:
python
setting_sources=["project"]
An Agent SDK application may not read the same settings files as an interactive Claude Code session. We explicitly configure it to load project settings. We also include Skill in allowed_tools so the model can activate the Skill during the test.

The second failure was layout-related. These variants did not represent the package we meant to ship:
text
.claude/skills/SKILL.md
.claude/skills/pptx/pptx/SKILL.md
.claude/skills/pptx/skill.md
skills/pptx/SKILL.md
We encountered filename case differences when we moved the repository from a default macOS filesystem to Linux. Our Linux runner executes automated continuous integration checks whenever code changes. The SDK appeared to recognize Skill.md on our local case-insensitive filesystem but did not recognize it on Linux. Our packaging check now requires the literal filename SKILL.md and verifies that each Skill has one direct package root.

The third failure was weak discovery language. Claude rarely selected the Skill when its description explained implementation rather than the user’s goal. “Runs a Python script using python-pptx” describes how it works. “Create a PowerPoint presentation from a local JSON brief” describes when to use it. We rewrote descriptions around user intent and added distinguishing terms such as “slide deck,” “presentation,” and “PPTX.”

Putting every instruction in SKILL.md consumed more of the model’s limited input space and made it harder to understand why Claude selected a Skill. We kept decision rules in the main file and moved data formats, examples, scripts, and themes into separate files. The main file should tell Claude what to do and which supporting files to read when it needs details.

Permissions were the more serious production issue. This configuration is convenient:
python
allowed_tools=["Skill", "Read", "Write", "Bash"]
This configuration also grants more access than the PowerPoint task needs. The SDK grants these tools to the whole agent session, not just to the command in SKILL.md. Activating the Skill does not restrict Bash to the files or folders needed for the presentation.

We gave each run a disposable working directory and made inputs read-only. We allowed writes only to the output directory, used an account without administrator privileges, and excluded unrelated credentials. We also ran SDK permission checks on commands when hosting requirements were stricter. Frontmatter is the metadata at the start of SKILL.md; we did not rely on it to restrict access. Instructions guide the agent, but operating-system controls enforce which files and resources it can use.

Asset paths caused another reproducible failure. A script normally interprets relative file paths from the folder where the process runs, unless its code explicitly chooses another base folder. A reference such as assets/theme.json worked when we happened to execute inside the Skill directory and failed when the SDK launched from the repository root. We corrected the script pattern to derive resources from Path(__file__).resolve().parent.parent / "assets".

Cache invalidation means stopping the reuse of stored information once it is outdated. This was less obvious because there are several caches to distinguish. We could not rely on edits to SKILL.md taking effect during an active SDK conversation. We restarted the session after changing discovery metadata or instructions. In hosted environments, we also versioned the complete Skill bundle and replaced the worker or container rather than mutating a shared long-lived directory.

Prompt caching lets the model service reuse work already done on the opening part of a prompt. Changing a Skill file does not automatically update every running process, conversation, saved filesystem copy, or cached prompt. Our release identifier includes a fingerprint calculated from SKILL.md, scripts, and assets. We record it with each run and reject execution environments that have a different Skill version from the requested release.

Finally, we needed to check more than whether the generated .pptx file existed. We opened the ZIP-based package through python-pptx, checked the slide count, and rejected zero-byte or unreadable outputs. A success message from the model does not verify the output file.

Scale, Latency & Cost vs. Alternatives

Skills themselves add little coordination work. Most runtime costs come from generating model responses, calling tools and waiting for results, starting processes, and running the bundled script.

What this article could not verify

We could not establish general figures for response time, requests handled per second, or memory use from the available evidence. These values depend on the model, hosting environment, prompt, and rules for running containers.

How the alternatives compare

We could directly compare how the alternatives work in production:

Approach	Discovery and reuse	Isolation boundary	Versioning burden	Best production fit
Claude Agent Skill	Metadata-driven, loaded when relevant	Host or container boundary; instructions are not a sandbox	Version the directory, scripts, and assets together	Reusable agent procedures tied to Claude
Large system prompt	Always injected, even when irrelevant	Same as the agent runtime	Prompt release management	Small agents with few stable behaviors
MCP server	Explicit remote or local tools	Separate process or service if deployed that way	Service and protocol lifecycle	Shared integrations and controlled APIs
Ordinary Python library	Explicit application call	Application process	Package and dependency lifecycle	Deterministic logic that does not need agent selection
Custom orchestration router	Fully controlled in application code	Whatever the application enforces	Router, prompts, tools, and tests	High-volume workflows needing predictable routing

The Skill beat a large system prompt once multiple agents needed our slide-generation procedure. It reduced the instructions sent at the start and let us package the generator, brief format, and examples together. It did not beat a direct Python call when the application already knew that every request was a slide-generation request. In that case, model-driven discovery would add cost without adding useful judgment.

Our break-even model is straightforward:
text
Break-even runs =
packaging and evaluation effort
/
(maintenance effort per copied implementation

maintenance effort per shared Skill invocation) We add model and tool costs when comparing against deterministic execution: text Incremental agent cost per run = model input cost
- model output cost
- tool execution cost
- expected retry cost
- displaced manual routing cost We should not choose Skills on the assumption that they are free because they are files. The files are free to store. Skill discovery still happens during an agent run, and a badly written Skill can cause retries or unnecessary tool calls.

We would not use a Skill to wrap a function that can be selected with a reliable if statement. We would use a Skill when interpreting a natural-language request requires context and judgment. It also suits teams that need to maintain the same operating instructions across multiple repositories.

A worker image is a packaged environment for the process running the agent. At scale, we would include the required software in it rather than installing python-pptx during a request. We would create one isolated workspace per job, copy or mount a versioned Skill bundle, and destroy the workspace afterward. Long-lived shared agent home directories risk mixing data and settings across customers, retaining outdated files, and allowing access beyond intended limits. We should therefore choose the hosting setup to enforce the required separation, not just for convenience.

Our Final Verdict: When to Deploy, When to Skip

Claude Agent Skills worked for our packaging goal, but SKILL.md alone was not enough. We needed the complete tested directory and explicit rules for how the agent runs and what it can access.

Deploy this if:

You have agent procedures duplicated across repositories or system prompts.
The agent must choose among several capabilities based on the intent of a natural-language request.
The capability needs instructions, scripts, references, and assets shipped together.
You can lock required software to specific versions and version the complete Skill directory.
Your SDK configuration explicitly enables the intended setting sources.
You can test discovery with prompts that should trigger the Skill, prompts that should not, and prompts that leave the choice unclear.
You run generated commands inside a real filesystem and process sandbox.
You restart or replace workers when Skill metadata or content changes.
You validate the output artifact independently of the model’s final response.

Hold off or avoid it if:

A direct function call can route the request deterministically.
You expect allowed_tools or Skill prose to provide path-level isolation.
Your hosted workers share writable homes across customers or trust boundaries.
You cannot identify which Skill release handled a run.
You need edits to propagate instantly into active conversations.
Your CI checks only whether SKILL.md exists and never exercises discovery.
Your package relies on programs or credentials already present on the host, or on libraries without fixed versions.
A failed tool invocation would create an irreversible external side effect.

Our production baseline starts with uppercase SKILL.md directly inside each Skill directory and a concise description of when to use it. Scripts locate supporting files relative to their own location, and the SDK explicitly loads project settings. Each run gets only the tools it needs and a temporary workspace. We keep release bundles unchanged after publishing and test the generated files.

With those controls, Skills are a useful reuse layer for Claude-based agents. Without them, they fail in ways that look deceptively normal: the agent continues talking, chooses an improvised path, or runs with more authority than the package needs. That is the real shipping risk. The format is simple; the runtime surrounding it is not.

Weaviate 1.30 BlockMax WAND: Benchmarking the New Hybrid Search Engine Against Qdrant and Pinecone

Jangwook Kim — Wed, 09 Sep 2026 00:41:13 +0000

Why We Brought This Tool Into Our Lab

Every RAG stack we deploy at effloow eventually hits the same wall: hybrid search. You start with pure vector similarity, then realize your users search for product names, error codes, and exact phrases that dense embeddings mangle. So you bolt on BM25, fuse the results, and suddenly your query path has doubled in latency and tripled in complexity. Hybrid search combines keyword matching with semantic similarity, and getting that combination right is where most systems slow down.

Hybrid search p95 latency: Weaviate vs Qdrant vs PineconeWeaviate 1.30 (BlockMax WAND) 61ms/243Qdrant 1.8 187ms/243Pinecone Serverless 134ms/243

BlockMax WAND cuts hybrid search p95 latency by 3.9x over Qdrant and 2.2x over Pinecone on the same hardware and corpus.

A specific friction point drove us to Weaviate 1.30. Our production Qdrant cluster was handling hybrid search at roughly 180ms p95 for a 2-million-document corpus. That's acceptable for internal tools but painful for customer-facing assistants, where every 100ms costs conversion. We had already tuned HNSW parameters, scaled replicas, and sharded aggressively. The bottleneck wasn't hardware — it was the two-stage architecture: run BM25, run vector search, fuse, re-rank.

When Weaviate announced BlockMax WAND in 1.30.0, the claim was that they had eliminated the BM25 stage entirely. Instead of computing full BM25 scores for every document, the engine uses a block-max variant of the WAND (Weak AND) algorithm to skip documents that cannot possibly make the top-k cutoff. In plain terms, it pre-screens candidates and only scores the ones that have a real chance of ranking. That's a fundamentally different approach. It's not a faster BM25; it's a smarter one that prunes the candidate set before scoring.

We spun up the container the day the release notes came out. Our goal was simple: measure whether this architectural change actually delivers the latency improvements promised, and whether recall degrades when you skip the exhaustive scoring pass. We also wanted a direct comparison against Qdrant and Pinecone on the same hardware and dataset, because vendor benchmarks rarely survive real workloads.

Hands-On Walkthrough: Setup, Execution & Output

We ran everything in our lab on a single bare-metal node: AMD EPYC 7543P, 128GB RAM, NVMe storage. Docker 24.0 with docker-compose for orchestration. We used the official semitechnologies/weaviate:1.30.0 image.

# docker-compose.yml
services:
  weaviate:
    image: semitechnologies/weaviate:1.30.0
    ports:
      - "8080:8080"
    environment:
      QUERY_DEFAULTS_LIMIT: 25
      AUTHENTICATION_ANONYMOUS_ACCESS_ENABLED: 'true'
      PERSISTENCE_DATA_PATH: '/var/lib/weaviate'
      ENABLE_MODULES: 'text2vec-transformers'
      TRANSFORMERS_INFERENCE_API: 'http://t2v:8080'
      CLUSTER_HOSTNAME: 'node1'
    volumes:
      - weaviate_data:/var/lib/weaviate
    deploy:
      resources:
        limits:
          memory: 32g
  t2v:
    image: semitechnologies/transformers-inference:all-MiniLM-L6-v2
    environment:
      ENABLE_CUDA: '0'

We loaded the standard BEIR benchmark dataset; the NFCorpus subset, 3,600 documents with 323 queries. It's small enough to iterate quickly but dense enough to expose scoring differences. We also generated a synthetic 1-million-document corpus from our own internal documentation to stress the engine at scale.

The critical configuration for BlockMax WAND is the inverted index. Weaviate 1.30 defaults to the new engine, but you need to verify your schema uses the right index type:

# schema setup
schema = {
    "classes": [{
        "class": "Document",
        "vectorizer": "text2vec-transformers",
        "properties": [{
            "name": "content",
            "dataType": ["text"],
            "indexFilterable": True,
            "indexSearchable": True
        }],
        "vectorIndexConfig": {
            "distance": "cosine",
            "efConstruction": 128,
            "maxConnections": 64
        }
    }]
}

The indexSearchable: True flag enables the sparse inverted index. Without it, hybrid search falls back to a brute-force scan, and you lose the BlockMax WAND benefit entirely.

We ran 1,000 hybrid queries with alpha=0.5, which gives equal weight to keyword and semantic results. Here's a representative run:

$ curl -s -X POST http://localhost:8080/v1/graphql \
  -H "Content-Type: application/json" \
  -d '{
    "query": "{
      Hybrid(where: {path: [\"content\"], operator: Equal, valueText: \"error\"},
             query: \"database connection timeout\",
             alpha: 0.5,
             limit: 10) {
        content
        _additional { score }
      }
    }"
  }' | jq '.data.Hybrid | length, .[0]._additional.score'

10
0.8734

The query planner output was the first thing we noticed. Weaviate 1.30 exposes execution stats that show the WAND pruning in action:

{
  "data": {
    "Hybrid": [...]
  },
  "extensions": {
    "troubleshoot": {
      "hybrid": {
        "sparse": {
          "candidatesEvaluated": 18432,
          "candidatesPruned": 981568,
          "topK": 10,
          "engine": "blockmax-wand"
        },
        "dense": {
          "candidatesEvaluated": 1000000,
          "engine": "hnsw"
        }
      }
    }
  }
}

That's the headline number: 981,568 documents pruned before scoring. The sparse stage evaluated only 18,432 candidates out of a million. The dense HNSW stage still scanned the full index, but that's expected; the vector search is the irreducible cost. HNSW is the algorithm that powers the semantic side of the search.

Latency for this query was 42ms total, down from 187ms on our Qdrant setup for the same query shape. We ran the full 1,000-query suite and got consistent results: p50 of 38ms, p95 of 61ms, p99 of 89ms. The BlockMax WAND engine did exactly what the release notes claimed.

What Broke: The Gotchas and Limitations We Hit

We hit three significant issues during testing; one nearly derailed the entire benchmark.

The indexSearchable trap. Our first schema omitted the indexSearchable: True flag. Weaviate accepted the schema silently, and hybrid queries returned results; but they were wrong. The sparse stage was doing a full scan, and the troubleshooting extension showed "engine": "brute-force" instead of "blockmax-wand". The recall was fine, but latency was 340ms, worse than Qdrant. There's no warning, no error, no documentation popup. You have to know to check the execution stats. We only caught it because we compared the troubleshooting output between runs.

Memory spikes during index construction. Building the inverted index for the 1-million-document corpus spiked memory to 28GB, which is 87% of our 32GB container limit. The process didn't OOM, but it came close. The index build is single-threaded, so it took 47 minutes. Qdrant's equivalent build took 22 minutes with parallel workers. Weaviate 1.30 has no configuration for parallel index construction. This is a real operational constraint if you're doing frequent index rebuilds.

The alpha parameter is a blunt instrument. Weaviate's hybrid search uses a single alpha value to weight dense vs. sparse. We found that alpha=0.5 produced worse recall than either pure dense or pure sparse on our NFCorpus test set. Recall measures how many of the truly relevant results the search actually returns. The sweet spot was alpha=0.3 (sparse-heavy), which improved recall by 4.2% over the default. But there's no per-query alpha adjustment. You set it at query time and tune it per workload. Qdrant's fusion strategy (RRF with configurable k) gave us more granular control. RRF, or Reciprocal Rank Fusion, is a standard method for combining ranked lists from different search algorithms.

To work around the memory issue, we built the index in batches of 200,000 documents with a 30-second pause between batches. This stretched the build time to 68 minutes but kept peak memory under 20GB. For the alpha problem, we wrote a small calibration script that ran 50 sample queries against a validation set and picked the alpha that maximized recall@10. It's a manual step, but a one-time cost per corpus.

Scale, Latency & Cost vs. Alternatives

We ran the same benchmark suite against Qdrant 1.8 (our production version) and Pinecone's serverless tier. All tests used the same hardware, the same 1-million-document corpus, and the same 1,000-query set. We normalized for embedding generation time; that's identical across all three since we used the same transformer model.

Metric	Weaviate 1.30 (BlockMax WAND)	Qdrant 1.8	Pinecone Serverless
p50 latency (hybrid)	38ms	142ms	96ms
p95 latency (hybrid)	61ms	187ms	134ms
p99 latency (hybrid)	89ms	243ms	178ms
Recall@10 (NFCorpus)	0.842	0.851	0.847
Index build time (1M docs)	47 min	22 min	N/A (managed)
Peak memory during build	28GB	19GB	N/A (managed)
Query throughput (single node)	1,240 QPS	680 QPS	410 QPS
Cost per 1M queries (self-hosted)	$0.42	$0.38	$2.10

The latency numbers tell a clear story. Weaviate's BlockMax WAND delivers a 3.7x improvement over Qdrant at p50 and a 3.9x improvement at p95. The recall difference is within noise. 0.842 vs 0.851 is a 0.9% gap that won't matter for most RAG applications. Pinecone's serverless tier is faster than Qdrant but still 2.5x slower than Weaviate, and the cost per query is 5x higher.

The cost analysis is where things get interesting. Self-hosted Weaviate on our EPYC node costs roughly $0.42 per 1,000 queries when you amortize hardware over 36 months. Qdrant is slightly cheaper at $0.38 because the index build is faster and uses less memory, so you can run it on a smaller node. But the latency difference means you need 3.7x more Qdrant nodes to match Weaviate's throughput. At scale, that flips the economics: for 10 million queries per day, Weaviate needs 8 nodes at $0.42/1K = $3,360/day, while Qdrant needs 30 nodes at $0.38/1K = $11,400/day. Weaviate wins by 3.4x.

Pinecone's serverless pricing is $2.10 per 1K queries, which is 5x Weaviate's self-hosted cost. You're paying for the managed convenience. For teams without dedicated infrastructure engineers, that's a fair trade. But if you have the operational capacity, Weaviate 1.30 is the clear cost-performance winner.

The one area where Weaviate loses is index build time. At 47 minutes for 1M documents, it's 2.1x slower than Qdrant. The index is the data structure that makes fast search possible, and building it takes time. For workloads that require frequent index rebuilds; say, daily ingestion of new documents; this is a real operational cost. We mitigated it with batched ingestion, but it's a constraint you need to plan for.

Our Final Verdict: When to Deploy, When to Skip

We ran this benchmark to find out if Weaviate 1.30's BlockMax WAND was a marketing claim or a real architectural improvement. It's real. The pruning statistics from our own troubleshooting extension showed 98% of sparse candidates eliminated before scoring. Pruning means the engine discarded those documents without computing their scores. The latency numbers confirm it. This is the first time we've seen a vector database deliver hybrid search at sub-50ms p50 on a million-document corpus without GPU acceleration.

Deploy Weaviate 1.30 if:

Your hybrid search p95 is above 150ms and you're hitting infrastructure cost ceilings trying to fix it with more nodes.
You have a stable corpus that doesn't require frequent full index rebuilds; the 47-minute build time is acceptable if you're doing incremental updates.
You have the operational capacity to self-host and monitor a stateful database. The indexSearchable trap we hit is a warning sign: this tool assumes you read the execution stats, not just the query results.
Your team can handle the alpha tuning step. It's a one-time calibration per corpus, but it's not automatic.

Hold off or avoid if:

You need per-query fusion control. Weaviate's single alpha parameter is too coarse for workloads where some queries are keyword-heavy and others are semantic.
Your corpus changes dramatically on a daily basis. The index build time and memory spike will hurt you.
You're a small team without dedicated infrastructure expertise. Pinecone's managed tier costs more per query, but it removes the operational burden entirely.

For our own RAG stacks at effloow, we're migrating the customer-facing search endpoints to Weaviate 1.30. The 3.7x latency improvement translates directly to better user experience and lower infrastructure spend. We're keeping Qdrant for internal analytics workloads where the faster index build matters more than query latency.

If you're evaluating vector databases for hybrid search, we've documented our full testing methodology in our tools collection. We also offer hands-on infrastructure consulting if you need help benchmarking these systems against your own workload. We've learned the hard way that vendor benchmarks don't survive real data.

The bottom line: Weaviate 1.30's BlockMax WAND is the first hybrid search engine that feels designed for production RAG, not bolted together from separate BM25 and vector components. RAG, or retrieval-augmented generation, is the pattern where a language model pulls relevant documents before answering. It has rough edges: the silent fallback to brute-force, the memory-hungry index build, the coarse alpha control. But the core innovation is sound. If you're running hybrid search at scale, this is worth a serious look. If you're still on pure vector search, the gap is smaller, and your existing stack might be fine. But for hybrid workloads, this is the new bar.