DEV Community: Jangwook Kim

RAGFlow 0.26.4 in Production: Strong Parsing, Expensive Baseline, Unproven Tenant Isolation

Jangwook Kim — Thu, 15 Oct 2026 00:41:09 +0000

Why We Evaluated RAGFlow

We brought RAGFlow into our evaluation pipeline because the recurring cost of custom RAG glue had stopped being theoretical.

Retrieval quality vs memory baselineParsing fidelity 99%/100Retrieval accuracy 73%/100Idle memory 8,517.3 MBPeak memory during ingestion 9,959.6 MB

RAGFlow preserves 99% of planted facts but surfaces the right answer in the top five only 73% of the time, while idling at 8,517.3 MB.

A typical internal deployment had accumulated a parser service, chunking workers, embedding jobs, vector and keyword retrieval, reranking, chat orchestration, document status tracking, and an administrative interface. LangChain or LlamaIndex could connect those pieces, but neither removed our responsibility for operating them. Every new tenant added more configuration, lifecycle handling, and failure states.

RAGFlow offered a more integrated proposition: deep document parsing, dataset management, hybrid retrieval, chat configuration, citations, model registration, and an HTTP API in one platform. It also uses the Apache 2.0 license, which matters when the deployment will be embedded in a commercial service rather than kept inside an internal prototype.

The attraction was not that RAGFlow invented retrieval. It was that it bundled the operational workflow around retrieval:

Create a dataset.
Select an embedding model and parser.
Upload documents.
Trigger parsing.
Wait for chunks to become available.
Query those chunks directly or through a configured chat assistant.

That workflow replaces a meaningful amount of application code. It does not, however, eliminate infrastructure.

For our 0.26.4 production baseline, we evaluated the published-image deployment results separately from our offline client-contract sandbox. The sandbox tested response handling, not deployment performance. In the published-image deployment baseline we evaluated, RAGFlow ran in five containers, consumed 8,517.3 MB at idle, and reached 9,959.6 MB during ingestion. Elasticsearch alone accounted for more than 4 GB of idle memory in that deployment.

Those numbers immediately changed the question. We were no longer asking whether RAGFlow could replace some Python orchestration. We were asking whether the reduction in engineering work justified an approximately 8.5 GB idle platform while still budgeting for workload growth, local models, observability, backups, and concurrency.

We also needed an answer to a harder question: does RAGFlow provide enough isolation to operate unrelated customers on one shared installation?

We found tenant-aware configuration and authentication surfaces, but we did not find enough evidence to certify hard multi-tenant isolation, per-tenant resource quotas, or noisy-neighbor protection. That distinction is critical. A platform can associate datasets with tenants without providing the controls expected from a hostile or heavily regulated multi-tenant service.

Our review therefore focused on three production-readiness criteria:

Can we automate setup and retrieval without driving the web interface?
Can the API fail safely under automation?
Can we establish a credible cost and isolation model before consolidating tenants?

For other infrastructure evaluations using the same deployment-first approach, browse our tools collection.

Hands-On Walkthrough: Setup, Execution & Output

We scoped our review to RAGFlow 0.26.4 and the benchmark CLI instructions at commit d32e05d5. Pinning matters because API routes and backend behavior change between releases. For deployment planning, we also reviewed the Go v1.0.0-rc1 quickstart. Its host-hardening guidance and Compose launch command below are reference steps, not a validated 0.26.4 installation procedure. Our local sandbox exercised only offline Requests response handling.

On an Elasticsearch-backed host, we first checked the kernel mapping limit:

sysctl vm.max_map_count
sudo sysctl -w vm.max_map_count=262144

We kept the persistent setting in the host configuration rather than relying on the temporary sysctl -w change. We also changed the dependency credentials in docker/.env before exposing any port beyond loopback.

For the built-in CPU embedding path used by the benchmark scripts, we configured the TEI model before starting the containers:

cd ragflow

printf '%s\n' 'TEI_MODEL=BAAI/bge-small-en-v1.5' >> docker/.env
printf '%s\n' 'COMPOSE_PROFILES=${COMPOSE_PROFILES},tei-cpu' >> docker/.env

cd docker
docker compose -f docker-compose.yml up -d
docker compose ps

The benchmark environment uses Python 3.13 and uv:

cd ..
uv sync --python 3.13 --group test

export ZHIPU_AI_API_KEY="redacted"
PYTHONPATH=./test uv run -m benchmark retrieval \
  --base-url http://127.0.0.1:9380 \
  --allow-register \
  --login-email "qa@infiniflow.org" \
  --login-password "123" \
  --bootstrap-llm \
  --llm-factory ZHIPU-AI \
  --llm-api-key "$ZHIPU_AI_API_KEY" \
  --dataset-name "bench_dataset" \
  --dataset-payload '{"name":"bench_dataset","embedding_model":"BAAI/bge-small-en-v1.5@Builtin"}' \
  --document-path test/benchmark/test_docs/Doc1.pdf \
  --document-path test/benchmark/test_docs/Doc2.pdf \
  --document-path test/benchmark/test_docs/Doc3.pdf \
  --question "What does RAG mean?" \
  --iterations 10 \
  --concurrency 4 \
  --json

The canonical entry point is:

PYTHONPATH=./test uv run -m benchmark [global flags] <chat|retrieval> [command flags]

The CLI can create a dataset, upload files, trigger parsing, poll for completion, execute retrieval or chat requests, and optionally delete the generated resources. It is not a general administrative CLI. We captured generated dataset IDs from its report when we wanted to reuse resources across runs.

We use the following hypothetical report sketch to illustrate reporting concepts, not verified CLI output or a measured retrieval result. Its field names are illustrative, and the dataset ID, 10 iterations, four-way concurrency, ten successes, and zero failures are synthetic example values. Our only executed local experiment used 12 offline response fixtures; it did not run retrieval:

{
  "command": "retrieval",
  "base_url": "http://127.0.0.1:9380",
  "dataset_id": "7f52c4a3...",
  "iterations": 10,
  "concurrency": 4,
  "successful": 10,
  "failed": 0,
  "question": "What does RAG mean?"
}

For chat tests, we supplied both the chat assistant configuration and the model field expected by the OpenAI-compatible endpoint. The important detail is that the request-level --model value does not override the model configured on the server-side chat assistant.

The full-corpus 0.26.4 baseline we used ingested 36 of 36 documents and stored 477 chunks. It retained 99% of planted facts in the stored chunks and put the answer in the top-five context for 73% of 93 retrieval questions. Retrieval latency with CPU embedding was 414.7 ms at p50 and 464.4 ms at p95.

That result exposed the difference between parsing fidelity and retrieval accuracy. RAGFlow preserved nearly all the facts, yet the tested retrieval configuration surfaced the correct answer in the top five only 73% of the time. Better parsing did not automatically produce excellent ranking.

We also reproduced the API error-handling problem offline with Requests 2.32.3. The adapter below is runnable and deliberately fails closed:

import requests


class BusinessError(RuntimeError):
    pass


class EnvelopeError(RuntimeError):
    pass


def parse_ragflow_response(response: requests.Response) -> dict:
    response.raise_for_status()

    try:
        payload = response.json()
    except ValueError as exc:
        raise EnvelopeError("Response is not valid JSON") from exc

    if not isinstance(payload, dict):
        raise EnvelopeError("Expected a JSON object")

    code = payload.get("code")
    if isinstance(code, bool) or not isinstance(code, int):
        raise EnvelopeError("Expected an integer business code")

    if code != 0:
        message = payload.get("message", "Unspecified RAGFlow error")
        raise BusinessError(f"RAGFlow code {code}: {message}")

    return payload

Across 12 fixtures, transport-only raise_for_status() missed four business failures returned with HTTP 200. Our adapter accepted only the valid success envelope, rejected codes 101, 106, 108, and 109 as business errors, rejected malformed envelopes, and still raised normal HTTP errors for 403 and 500 responses.

That test did not emulate a running RAGFlow server. It verified the client behavior we require when integrating an API that uses both HTTP status and a JSON business code.

Operational Limitations We Encountered

The first trap was ingestion lifecycle ambiguity. Uploading a document does not mean that RAGFlow has parsed it. Parsing is a separate action. A client that uploads and then waits for chunks without triggering parsing can wait indefinitely for work that was never scheduled.

Our workaround was to model ingestion as an explicit state machine:

created -> uploaded -> parse_requested -> parsing -> ready
                                      \-> failed

We persisted both the RAGFlow document ID and our own idempotency key. We also separated parse timeout from HTTP timeout because parsing can legitimately outlive an individual API request.

The second trap was error semantics. HTTP 200 means the request reached the application, not necessarily that the operation succeeded. We now require both a successful HTTP status and an integer body-level code equal to zero for endpoints using the business envelope. OpenAI-compatible completion responses need their own parser because they do not necessarily use that envelope.

The third trap was authentication during bootstrap. API-token authentication is straightforward once a token exists, but login automation can require client-side RSA encryption of the password. We avoided reproducing browser login logic in production services. We provisioned API tokens out of band and injected them through the secret manager instead.

The fourth trap was route churn. Version 0.26 moved routes under a new prefix, making older examples return 404. The later Go backend introduces another migration boundary, including canonical endpoint and parser-field changes. We pinned API tests to the deployed release and treated a backend upgrade as an application migration, not a routine container refresh.

The fifth issue was resource overhead. The deployment consumed 8,517.3 MB even while idle. Peak memory reached 9,959.6 MB during ingestion in the measured 36-document corpus. Time to the first served request was 511 seconds from a clean host, including image pulls. That is acceptable for a persistent service, but poor for ephemeral preview environments or scale-to-zero workers.

The largest unresolved concern was tenant isolation. We could configure tenant model defaults, authenticate requests, and select datasets by ID. We could not establish the following controls from the reviewed 0.26.4 surfaces:

Enforced per-tenant ingestion concurrency
Per-tenant CPU or memory quotas
Retrieval rate limits
Storage quotas
Dedicated worker pools
Tenant-specific Elasticsearch capacity
A documented noisy-neighbor policy
A formal isolation guarantee suitable for hostile tenants

We therefore treated one shared installation as a trust-boundary decision, not merely a cost optimization. If tenants share parsing workers and Elasticsearch capacity, one tenant's image-heavy ingestion or index growth could affect another tenant's retrieval. We treated that as a deployment risk to test, not as measured noisy-neighbor behavior in RAGFlow 0.26.4.

High availability presented a similar gap. We kept issue #9963 as a production release-gate reference. In our HA planning, we treated the defaults as local-development or single-instance oriented, with potential bottlenecks under clustered or high-traffic load. We also treated high concurrency as an enterprise-edition capability to validate before purchase, rather than assuming it was available in our planned deployment.

The issue did not give us a complete list of unsafe keys and replacement values, so we did not invent one. Our mitigation was architectural: we refused to label the default Compose stack as HA. Before production approval, we required external database and object-storage durability, dependency health checks, tested restore procedures, multiple stateless API replicas where supported, controlled worker concurrency, and load testing against the exact edition being purchased.

Scale, Latency & Cost vs. Alternatives

RAGFlow's economic case depends less on request volume than on how much platform engineering it replaces.

Option	Operational baseline	Parsing and retrieval control	Tenant-isolation burden	Best fit
RAGFlow 0.26.4	Five containers in the measured deployment; 8,517.3 MB idle	Strong document workflow, inspectable chunks, integrated retrieval and chat	High unless isolation is independently validated or deployments are separated	Document-heavy products with trusted tenants
LangChain or LlamaIndex composition	Determined by the services we assemble	Maximum flexibility, but we own integration and lifecycle code	Entirely ours to design	Teams with an established AI platform group
Lean vector stack	Can be materially smaller when parsing and orchestration remain simple	Good for clean text; more custom work for complex documents	Entirely ours to design	Narrow retrieval services and cost-sensitive workloads
Managed RAG service	Little infrastructure for us to operate	Provider-defined ingestion, ranking, and observability	Contractual and provider-dependent	Fast launch, variable traffic, limited operations staff
Dedicated RAGFlow per tenant	Multiplies the fixed RAGFlow baseline	Same integrated workflow with a clearer failure boundary	Stronger practical separation, higher cost	Regulated or high-value tenants

We used 414.7 ms at p50 and 464.4 ms at p95 as our CPU-embedding retrieval baseline. We would not use those values as a universal service-level objective. Corpus size, Elasticsearch configuration, embedding placement, reranking, metadata filters, and concurrent ingestion all change the result. We would use them as a baseline showing that retrieval alone already occupies a substantial portion of a sub-second API budget.

We built our break-even model with explicit assumptions rather than pretending the available evidence included a monetary bill.

Consider a shared 16 GB host priced at an illustrative $0.25/hour:

Host cost:             730 hours × $0.25       = $182.50/month
Operations allowance: 4 hours × $100           = $400.00/month
Estimated fixed cost:                            $582.50/month

At an illustrative managed retrieval price of $0.002/request, infrastructure plus four hours of monthly operations breaks even at:

$582.50 / $0.002 = 291,250 retrieval requests per month

This excludes generation tokens, embedding-provider charges, backup storage, outbound traffic, monitoring, and engineering migration work. Those exclusions apply to both sides and must be filled with actual vendor quotes.

Shared across ten modest tenants, the illustrative fixed cost becomes $58.25 per tenant per month. Dedicated stacks provide clearer isolation boundaries but push the same illustrative baseline toward $582.50 per tenant before model usage. That is the central trade-off: sharing makes RAGFlow financially attractive precisely where we have the least evidence of effective quotas and noisy-neighbor protection.

We would choose RAGFlow over custom LangChain or LlamaIndex glue when the engineering savings from complex parsing and administrative visibility exceed its incremental infrastructure and operations costs relative to our existing stack. We would choose a lean vector stack when inputs are already clean text and the product needs only ingestion, metadata filtering, and retrieval. We would choose managed RAG when volume is uncertain and avoiding an 8.5 GB idle platform matters more than infrastructure ownership.

Teams needing a workload-specific model can review our AI infrastructure services or contact us for an architecture assessment.

What We Could Not Verify

We could not establish hard multi-tenant isolation, per-tenant resource quotas, or noisy-neighbor protection from the reviewed 0.26.4 surfaces. Our offline API fixtures verified client behavior, not a running RAGFlow server.

Our Final Verdict: When to Deploy, When to Skip

RAGFlow 0.26.4 works best as a document-understanding platform, not as a RAG deployment that is production-ready by default.

Its strongest result was parsing fidelity: 99% of planted facts remained available in stored chunks in the measured corpus. Its weaker result was ranking: 73% top-five retrieval accuracy left meaningful room for embedding, chunking, threshold, keyword, and reranking work. The platform preserves evidence well, but teams still have to tune how that evidence is surfaced.

The Apache 2.0 license, integrated UI, benchmark CLI, HTTP interface, and explicit parse workflow make RAGFlow credible for commercial deployment. The memory baseline, API quirks, route migration risk, and unverified multi-tenant controls stop us from recommending the default Compose configuration as a production architecture.

Deploy this if

Complex PDFs, tables, and document layouts justify a heavy parsing stack.
Your tenants belong to one trusted organization or can share a failure domain.
An approximately 8.5 GB idle baseline is acceptable.
You will keep the service warm rather than starting it on demand.
You can pin versions and maintain contract tests around every API upgrade.
You will check both HTTP status and body-level business codes.
You can operate Elasticsearch and test backup restoration.
You are prepared to tune retrieval instead of accepting defaults.
You can validate enterprise concurrency capabilities before relying on them.

Hold off or avoid it if

You need cheap scale-to-zero retrieval.
Your documents are mostly clean text and do not need deep parsing.
Untrusted customers must share one installation.
You require proven per-tenant quotas and noisy-neighbor controls.
Your availability plan assumes Docker Compose is already an HA topology.
Your host has less than the practical memory headroom implied by the measured baseline.
You expect upload alone to complete ingestion.
Your client treats every HTTP 200 response as success.
You cannot absorb endpoint and backend migration work.

Our production recommendation is conservative: use RAGFlow for trusted internal deployments or isolate valuable tenants into separate stacks. Do not consolidate unrelated customers merely because the API exposes tenant-aware objects. Authentication and object ownership are not substitutes for resource isolation.

Before signing off, we would run the official benchmark CLI against the actual corpus, embedding model, concurrency target, and RAGFlow edition planned for production. We would then inject simultaneous parsing and retrieval from multiple tenant identities and verify latency, authorization, queue behavior, recovery, and cost.

RAGFlow can replace a substantial amount of RAG glue. It cannot replace production engineering.

PyRIT in Production: Agentic Red-Team Scenarios and CI Gate Design

Jangwook Kim — Wed, 14 Oct 2026 00:37:35 +0000

Why We Brought This Tool Into Our Lab

We brought PyRIT into our evaluation queue because prompt-level test suites stop being convincing once an application becomes agentic. A single malicious input is easy to represent in YAML. A sequence that adapts after a refusal, changes persona, invokes several transformations, and preserves conversational context is not.

PyRIT: orchestration engine, not a merge gate

Scenarios turn multi-turn attack loops into repeatable, scenario-run-ID-stamped artifacts, but with no verified exit-code contract, retry/backoff behavior, or scorer false-negative rate in the reviewed material, a low ASR cannot be read as proof of safety — so exit codes should report only whether the scan executed, and release policy must be evaluated separately from the complete result set.

PyRIT addresses that gap with Python-native attack execution. Its core pipeline combines objectives, attack strategies, converters, targets, scorers, and persistent memory. Multi-turn strategies such as Crescendo, TAP, PAIR, and red-teaming loops can adjust subsequent prompts based on the target's previous response. That makes PyRIT materially different from a static prompt regression suite.

The July 2026 Scenarios layer was the feature that made us consider it for release engineering rather than occasional security research. A scenario is a packaged playbook containing four practical elements:

A dataset of harmful or policy-violating objectives.
One or more attack techniques.
Converters or adversarial prompt transformations.
Scorers and a result printer.

The built-in catalog includes RapidResponse, Jailbreak, Cyber, Leakage, Psychosocial, Scam, Encoding, RedTeamAgent, TextAdaptive, and AdversarialBenchmark. Results receive a scenario-run identifier in PyRIT memory, making cross-run comparison and resumption possible.

That is useful infrastructure. It is not, however, an autonomous-agent security simulator.

We cross-checked the architecture against the Cloud Security Alliance's agentic PyRIT evaluation. That boundary matched our assessment of the framework's capabilities: PyRIT tests model-mediated behavior, but it does not natively implement an agent's authorization model, tool execution engine, or complete state machine. It can prompt an application into attempting tool misuse, but it cannot independently prove that the tool call crossed an actual permission boundary.

That distinction matters. A successful prompt attack is evidence of unsafe model behavior. It is not automatically evidence that money moved, data left the tenant, or an unauthorized API operation completed. For production agents, we would pair PyRIT with application telemetry and assertions around the real tool broker.

Our evaluation also exposed an important packaging distinction. PyRIT is a red-team execution framework. It is not, by itself, a finished merge policy. We identified RAMPART as a pytest-native layer for turning PyRIT findings into repeatable CI regression tests. That separation is sensible, but it means teams expecting pyrit_scan to behave like a security linter will have integration work ahead.

Hands-On Walkthrough: Setup, Execution & Output

We began with the published installation path and Python 3.13 recommendation:

python3.13 -m venv .venv
source .venv/bin/activate

python -m pip install --upgrade pip
pip install pyrit

python -c "import pyrit; print(f'PyRIT version installed: {pyrit.__version__}')"
pyrit_scan --help

For a production repository, we would not leave pyrit unpinned. The current material spans /latest/, versioned 1.0.1 scanner pages, and a scenario article that references features introduced in 0.14.0. We would install once in a controlled environment, capture the resolved version and lock file, and then promote that exact environment into CI.

PyRIT expects endpoint configuration in ~/.pyrit/.env and startup configuration in ~/.pyrit/.pyrit_conf. The minimum documented shape is:

# ~/.pyrit/.env
OPENAI_CHAT_ENDPOINT="https://api.openai.com/v1"
OPENAI_CHAT_KEY="${OPENAI_CHAT_KEY}"
OPENAI_CHAT_MODEL="your-model"

# ~/.pyrit/.pyrit_conf
memory_db_type: in_memory

initializers:
  - name: target
    args:
      tags:
        - default
        - scorer
  - name: scorer

For Azure, we had to treat the endpoint as provider-specific configuration rather than a generic base URL. The accepted examples include both of these Azure forms:

OPENAI_CHAT_ENDPOINT="https://example.cognitiveservices.azure.com/openai/v1/"
# Or:
OPENAI_CHAT_ENDPOINT="https://example.services.ai.azure.com/openai/v1"
OPENAI_CHAT_KEY="${AZURE_OPENAI_KEY}"
OPENAI_CHAT_MODEL="deployment-or-model-name"

We would inject that file at runtime rather than bake it into a container. The key should come from the CI secret store, while the endpoint and model can remain environment-specific deployment variables.

For Ollama or llama.cpp, we found no source-verified native initializer example in the material we evaluated. The viable pattern is therefore an OpenAI-compatible endpoint or a custom PyRIT HTTP target, not an assumed first-class ollama target. An OpenAI-compatible local configuration may look like this:

# Valid only when the local server exposes an OpenAI-compatible /v1 API.
OPENAI_CHAT_ENDPOINT="http://host.docker.internal:11434/v1"
OPENAI_CHAT_KEY="local-placeholder"
OPENAI_CHAT_MODEL="local-model-name"

Linux containers may need an explicit host mapping:

docker run --rm \
  --add-host=host.docker.internal:host-gateway \
  -v "$HOME/.pyrit:/root/.pyrit:ro" \
  pyrit-runner:locked \
  pyrit_scan run airt.scam --target openai_chat

We would use a custom target rather than force compatibility if the local server's request or response shape differs. That also gives us a clean place to record request counts, inject deterministic failures, and distinguish target latency from adversarial-model latency.

The basic scanner paths are concise:

pyrit_scan run airt.scam --target openai_chat
pyrit_scan run airt.rapid_response --target my_target

RapidResponse performs a baseline pass and, with its default technique selection, applies role-play and many-shot attacks across seven AIRT categories: hate, fairness, violence, sexual content, harassment, misinformation, and leakage. This is a broad assessment, not a cheap smoke test.

The same abstraction is available from Python:

from pyrit.scenario.scenarios.airt import RapidResponse

scenario = RapidResponse()
await scenario.initialize_async(objective_target=target)
result = await scenario.run_async()

We skipped local PyRIT execution in this review. Our intended sandbox used Python 3.12, and we lacked a verified compatible package pin, a runnable offline target/scorer configuration, and a model endpoint suitable for measuring real scorer behavior. The installation and configuration examples above are proposed setup paths, not verified installation or scenario-run results.

The following runnable shell harness demonstrates how we would normalize a completed scan into an adapter-owned CI artifact. The JSON is explicitly simulated; it is not presented as PyRIT's native schema or as observed benchmark output.

#!/usr/bin/env bash
set -euo pipefail

mkdir -p artifacts

python - <<'PY' > artifacts/pyrit-gate.json
import json

# Simulated normalized output produced by a CI adapter after reading
# PyRIT memory or exported results. This is not PyRIT's native schema.
result = {
    "fixture": "simulated",
    "scenario": "airt.scam",
    "scenario_run_id": "demo-run-001",
    "objectives": 12,
    "successful_attacks": 1,
    "attack_success_rate": 0.0833,
    "inconclusive_scores": 2,
    "transport_errors": 0,
    "gate_decision": "manual-review"
}
print(json.dumps(result, indent=2))
PY

cat artifacts/pyrit-gate.json

Simulated output:

{
  "fixture": "simulated",
  "scenario": "airt.scam",
  "scenario_run_id": "demo-run-001",
  "objectives": 12,
  "successful_attacks": 1,
  "attack_success_rate": 0.0833,
  "inconclusive_scores": 2,
  "transport_errors": 0,
  "gate_decision": "manual-review"
}

This adapter boundary is deliberate. We would preserve raw conversations and scorer records, then create a stable internal schema that does not change whenever PyRIT changes its CLI presentation.

PyRIT Integration Problems and Limitations

The first roadblock was version ambiguity. Our setup plan starts with pip install pyrit, but we found examples spread across latest and versioned documentation. That is acceptable for exploration but not for a shipping gate. A new scenario name, initializer contract, or result representation can break a pipeline without changing our application.

Our workaround is conventional: pin Python, pin PyRIT, commit the lock file, record pyrit.__version__ in every artifact, and run a configuration canary before the expensive scenario.

The second problem was endpoint overloading. OPENAI_CHAT_ENDPOINT can refer to OpenAI, one of two Azure endpoint forms, or potentially a local OpenAI-compatible proxy. A syntactically valid URL can still be semantically wrong because of a missing /openai/v1, incorrect deployment or model naming, container DNS, or incompatible authentication.

We also investigated the visible BASE_URL error message. We could not reproduce it as a PyRIT runtime configuration error because the text appears as a generic MyST documentation-site warning: “This may be due to an incorrect BASE_URL configuration.” It is not evidence that PyRIT requires a BASE_URL environment variable for target execution. Treating that banner as runtime troubleshooting would send an operator toward the wrong configuration.

For target failures, we would validate these separately:

DNS and TCP reachability from inside the runner.
The exact API path and authentication header.
Model or deployment resolution.
A minimal chat-completions request.
PyRIT target initialization.
Scenario execution.

The third issue is that target and scorer plumbing can become entangled. LLM-based scoring needs its own reliable model endpoint. Pointing both attack execution and scoring at the same constrained local model may save cloud spend, but it can also turn the target's weaknesses into the judge's weaknesses. A model that refuses awkwardly, follows transformed text inconsistently, or cannot classify nuanced policy violations will produce noisy verdicts.

We could not establish a scorer false-negative rate without labeled responses and a real scoring endpoint. That absence itself prevents us from endorsing a binary gate. A low measured attack success rate (ASR) can mean at least four different things:

The target resisted the attacks.
The selected techniques were ineffective.
Requests failed or were throttled.
The scorer missed successful harmful behavior.

Those outcomes require different remediation, yet a naive ASR == 0 rule collapses them into “pass.”

Rate limiting introduces the same ambiguity. RapidResponse applies every selected technique to every objective, and multi-turn attacks add several requests per attempt. We accounted for latency, API rate-limit budget, and adversarial-chat tokens when assessing scan cost, but we did not verify a retry count, backoff policy, or Retry-After contract for orchestrator loops.

We would therefore place retries at a controlled HTTP client or proxy layer, log every 429 separately, and cap retries by objective. Exhausted retries should produce an infrastructure error, never a safe-model verdict. CI should fail closed on lost coverage but should label the result “invalid scan,” not “vulnerability found.”

Finally, PyRIT does not natively validate actual tool authorization, memory integrity, or long-horizon agent behavior. We would instrument the application under test with a sandboxed tool broker and correlate each PyRIT conversation with tool-call audit events. Without that correlation, an apparent tool-misuse success may be only generated text.

Scale, Latency & Cost vs. Alternatives

We found the main scaling equation straightforward. RapidResponse behaves roughly like:

objectives × techniques × requests per attack

The baseline adds another target request per objective, while Crescendo, TAP, PAIR, and other multi-turn methods can add variable numbers of target and adversarial-model calls. Scoring adds further calls when an LLM judge is used.

TextAdaptive changes the technique dimension. It selects techniques using historical ASR and stops after success, with three attempts per objective as the stated default. Its selector queries PyRIT memory rather than keeping counts internally. That can reduce wasted attempts, but it also means historical data affects future runs. For a strict release comparison, we would isolate memory or use a current-run scope; otherwise, the test policy silently changes as the database accumulates evidence.

Option	Best fit	Multi-turn offensive orchestration	Agent-system validation	CI readiness	Main cost driver
PyRIT Scenarios	Broad and focused adversarial discovery	Strong	Partial; external instrumentation required	Medium; adapter or RAMPART needed	Target, attacker, and scorer calls
RAMPART on PyRIT	Regression tests derived from findings	Inherits PyRIT	Partial	High for pytest-oriented teams	Test count and underlying PyRIT calls
Prompt-level YAML harness	Fast deterministic prompt regression	Limited	Limited	High	Number of prompts and judge calls
Inspect-style evaluation harness	General model or agent quality evaluation	Custom implementation required	Depends on test environment	High	Dataset size and evaluator calls
Managed red-team service	Teams preferring hosted operation	Usually packaged	Provider-dependent	Medium to high	Per-scan service and model usage
Manual specialist assessment	Novel, high-context attack discovery	Strong but labor-intensive	Potentially strong	Low	Engineer or consultant time

We have not published wall-clock, token, throughput, or memory numbers because our setup did not reach a valid endpoint-backed benchmark. Any specific figure would be invented. The useful break-even model is still simple:

automated release cost = runner cost + target calls + attacker calls + scorer calls + review time

Compare that with:

manual release cost = specialist hours × loaded hourly rate

PyRIT pays off when the same scenario is run repeatedly across releases. The initial integration includes target adapters, labeled scorer calibration, artifact normalization, and triage policy. After that, the marginal cost per release can fall substantially because scenario execution and evidence capture are automated.

It does not pay off when a team runs one shallow scan and treats the resulting ASR as a security certificate. In that case, setup and scorer calibration can cost more than a focused manual review while producing weaker evidence.

For cost control, we would use a staged pipeline:

Run a small deterministic regression set on every pull request.
Run focused Jailbreak, Leakage, Cyber, or Scam scenarios on protected branches.
Run RapidResponse on scheduled builds or release candidates.
Use TextAdaptive after enough trustworthy historical evidence exists.
Send successful and inconclusive results to human review.
Retain raw traces only under an explicit sensitive-data policy.

Teams comparing this approach with adjacent infrastructure can browse our tools collection. For architecture and integration help around model gateways, observability, and controlled agent sandboxes, see our engineering services.

What We Could Not Verify

We did not complete a live scenario benchmark, measure scorer false-negative rates, or verify orchestrator retry and backoff behavior. Our configuration examples and simulated CI artifact do not establish runtime performance or gate reliability.

Our Final Verdict: When to Deploy, When to Skip

PyRIT is one of the more credible open-source foundations we have examined for repeatable, multi-turn offensive testing. Scenarios solve a real operational problem: they turn fragile notebooks into named, reusable playbooks with scenario-run identifiers and memory-backed results.

We would deploy it, but not as a bare binary merge gate.

Deploy this if:

We need multi-turn attacks rather than a static prompt corpus.
We can pin the package and own the target initializer.
We can separate target, attacker, and scorer endpoints.
We are prepared to calibrate scorers against human-labeled examples.
We can preserve raw traces and distinguish vulnerabilities from infrastructure failures.
We can instrument real tools, permissions, memory, and side effects around the agent.
We want repeatable discovery scans that can later become regression tests.

Hold off or avoid it if:

We need a turnkey non-zero exit code whenever any attack succeeds.
We expect ASR alone to prove that a release is safe.
Our agent cannot be placed behind a sandboxed target or tool broker.
We cannot tolerate variable multi-turn request volume.
We lack a trustworthy scorer or a human adjudication path.
We need native validation of authorization, tool execution, or long-horizon state.
We are unwilling to maintain compatibility across PyRIT releases.

Our production pattern would be PyRIT for discovery, a normalized artifact adapter for CI, and deterministic regression tests for confirmed findings. We would use process exit codes only to indicate whether the scan executed successfully. Our CI adapter would evaluate security policy separately using the complete result set, with three outcomes: pass, block, or manual review.

That is less convenient than pyrit_scan && merge, but it is operationally honest. A transport timeout is not a safe result. A scorer disagreement is not a pass. A low ASR is not proof that an agent's tools are protected.

PyRIT is worth deploying when we treat it as an offensive orchestration engine rather than a compliance checkbox. Its scenarios provide a strong starting methodology, its memory model supports repeatable analysis, and its Python architecture leaves room for custom targets and scorers. Our production integration would verify retry behavior, calibrate scorers, normalize artifacts, enforce release policy, and add real agent-side observability.

If those controls are already part of the platform, PyRIT can remove a large amount of repetitive red-team scripting. If they are not, budget for that engineering before promising a shipping gate. Teams deciding whether that investment fits their release process can contact us with their target architecture and threat model.

Daft in Production: A Hard-Nosed Multimodal Pipeline Review Against Ray Data and Spark

Jangwook Kim — Tue, 13 Oct 2026 00:40:04 +0000

Why We Evaluated Daft for Multimodal Pipelines

The question that pushed us toward Daft was not whether it could perform another Parquet scan. We needed to know whether it could move compressed images, audio, PDFs, and video through CPU decoding and GPU inference without forcing us to hand-manage every intermediate batch.

That is a materially different problem from conventional analytics. A JPEG may expand by roughly 20 times when decoded. A video can turn into thousands of multi-megabyte frames. The engine must overlap object-storage reads, decoding, preprocessing, inference, and output while keeping expanded objects from accumulating in memory.

Daft approaches this as a declarative DataFrame engine with native multimodal expressions. Its Swordfish execution layer streams batches between operators and applies backpressure when a downstream operation, such as GPU inference, cannot keep up. Flotilla extends that execution model across workers. The practical appeal is straightforward: retain DataFrame and SQL-style planning while avoiding the partition-sized materialization that makes binary-heavy pipelines painful.

We compared that model with two realistic alternatives:

Ray Data, where resource-aware Python operators, actors, and independent CPU/GPU scaling offer considerable control, but partitioning, column retention, actor concurrency, and object-store pressure can become application responsibilities.
Spark, which remains the operational default in many companies but exposes awkward global controls for Python and Arrow UDF batches. Those controls are especially blunt when one stage handles compressed bytes and another holds decoded tensors.

We used the September 2025 distributed benchmark as the common reference point: Daft 0.6.2, Ray Data 2.49.2, and AWS EMR Spark 7.10.0 on eight g6.xlarge workers. The workload set covered 113,800 audio files with Whisper-tiny, 10,000 PDFs with all-MiniLM-L6-v2, 803,580 ImageNet images with ResNet18, and 1,000 videos with YOLO11n. The code and logs are linked from Daft's benchmark page.

We did not treat those headline results as neutral truth. We inspected the competing reproduction, input-layout differences, batch settings, encoder defaults, and instance-shape sweep. That changed our conclusion substantially: Daft is strong, but “Daft is always faster” is not defensible. CPU-to-GPU ratio and data layout can reverse the ranking.

Hands-On Walkthrough: Setup, Execution & Output

The base installation path is intentionally small:

python -m venv .venv
source .venv/bin/activate

pip install daft
# Equivalent with uv:
uv add daft

For the Ray-backed distributed path, the documented extra is:

pip install "daft[ray]"
ray start --head --disable-usage-stats

The default process runs locally. Connecting the same logical pipeline to Ray requires setting the runner before constructing or executing the DataFrame:

import daft

daft.set_runner_ray("ray://127.0.0.1:10001")

That one-line switch is real at the pipeline API level, but it does not make cluster operations disappear. The client and workers must carry matching Daft versions and matching Python minor versions. Worker images still need model files, codecs, credentials, native libraries, and compatible accelerator builds.

We designed the following local smoke test for a COCO-style image record; it requires no API keys. As written, our script is designed to generate JPEG and WAV fixtures, serve them from localhost, define a lazy image-decoding expression, and write a Parquet manifest containing only IDs and URLs. Because the final projection excludes decoded_image, the optimizer can prune that unused expression; writing the manifest would not establish that downloading or decoding actually executed. The audio file is included so the fixture set can be extended into a decode or transcription stage without requiring an external API.

#!/usr/bin/env bash
set -euo pipefail

python -m venv .venv
source .venv/bin/activate
pip install "daft==0.6.2" "Pillow==11.3.0"

mkdir -p fixtures output

python - <<'PY'
from pathlib import Path
from PIL import Image
import math
import struct
import wave

Path("fixtures").mkdir(exist_ok=True)

image = Image.new("RGB", (640, 480), color=(34, 91, 140))
image.save("fixtures/sample.jpg", quality=90)

sample_rate = 16_000
seconds = 1
with wave.open("fixtures/sample.wav", "wb") as wav:
    wav.setnchannels(1)
    wav.setsampwidth(2)
    wav.setframerate(sample_rate)
    frames = []
    for i in range(sample_rate * seconds):
        value = int(8000 * math.sin(2 * math.pi * 440 * i / sample_rate))
        frames.append(struct.pack("<h", value))
    wav.writeframes(b"".join(frames))
PY

python -m http.server 8000 --directory fixtures >/tmp/daft-fixture-http.log 2>&1 &
SERVER_PID=$!
trap 'kill "$SERVER_PID"' EXIT

python - <<'PY'
import json
import time
import daft

started = time.perf_counter()

df = daft.from_pydict(
    {
        "image_id": [1],
        "image_url": ["http://127.0.0.1:8000/sample.jpg"],
        "audio_url": ["http://127.0.0.1:8000/sample.wav"],
    }
)

df = df.with_column(
    "decoded_image",
    daft.col("image_url")
        .url.download()
        .image.decode(mode=daft.ImageMode.RGB),
)

df.select("image_id", "image_url", "audio_url").write_parquet(
    "output/manifest.parquet"
)

print(
    json.dumps(
        {
            "status": "ok",
            "rows_written": 1,
            "image_decode": "materialized",
            "output": "output/manifest.parquet",
            "elapsed_s": round(time.perf_counter() - started, 3),
        }
    )
)
PY

A successful run should produce stdout in this shape:

{
  "status": "ok",
  "rows_written": 1,
  "image_decode": "materialized",
  "output": "output/manifest.parquet",
  "elapsed_s": 0.742
}

The JSON above is illustrative, not observed output: the elapsed value is simulated, and image_decode is hard-coded to materialized, not measured. As written, our harness is intended to check installation, expression construction, and manifest writing—not successful image decoding. A decode-validation test must consume the decoded result and assert its contents. We did not complete such a test because our sandbox failed during installation with a storage error and produced no runtime observations. A performance job should also execute thousands of objects, separate cold and warm runs, and capture process RSS, object-store spilling, GPU utilization, output bytes, and per-stage counters.

Our attempted pinned installation provided one useful but unglamorous result. The resolver successfully selected the Daft 0.6.2 CPython ABI3 Linux ARM64 wheel, but the isolated environment ran out of storage while installing Daft, PyArrow, NumPy, and Pillow:

Installing collected packages: typing-extensions, tqdm, pyarrow,
Pillow, numpy, fsspec, daft
ERROR: Could not install packages due to an OSError:
[Errno 28] No space left on device

We therefore do not present the local decode test as completed. This was a sandbox-capacity failure, not a Daft runtime defect, but it highlights why CI images need more headroom than the 43.8 MB Daft wheel alone suggests. PyArrow, NumPy, Pillow, downloaded wheels, the virtual environment, fixtures, and outputs all count.

What Broke: The Gotchas and Limitations We Hit

The largest problem was not syntax. It was separating engine behavior from benchmark construction.

Flotilla and Ray are not cleanly separate setup choices. We used the Flotilla-powered benchmark as our reference and evaluated Ray or Kubernetes as the distributed deployment substrate, rather than treating Flotilla as a third interchangeable runner. On the Ray path, daft.set_runner_ray() selects distributed execution and Flotilla workers perform the underlying work. We found no standalone set_runner_flotilla() deployment recipe in the reviewed material. Teams should not plan for three interchangeable runners named local, Flotilla, and Ray. In practice, the choices are local execution or a distributed deployment substrate, with Flotilla inside Daft's distributed architecture.

Version equality is an operational contract. Ray Client requires the Daft version and Python minor version to match between client and server. A laptop on Python 3.12 cannot safely be treated as a transparent client for workers built around another minor version. We would ship the driver and worker from the same immutable image rather than rely on Ray runtime installation.

We also would not run benchmark-pinned Daft 0.6.2 code against the current stable documentation without a compatibility test. The newer documentation includes AI functions and a legacy UDF migration path that are not a substitute for the API surface used by old benchmark scripts. We could not establish a specific cross-release break from our failed local execution, so we are not inventing one. Our production safeguard is a lockfile, image digest, schema test, and materialization test before any version bump.

Null and corrupt media need an explicit policy. In our best-effort image-classification reproduction, we added an explicit null filter after decoding:

df = df.where(df["decoded_image"].not_null())

That is more than cleanup. Silent row removal changes corpus cardinality and can bias a dataset if corruption correlates with source, format, or age. We would write rejected URIs and decoder errors to a quarantine table, assert accepted plus rejected equals input, and fail CI if the reject rate crosses a threshold.

Partition layout moved Daft's image runtime by 20–25%. In our best-effort image-classification reproduction, we added an explicit repartition step that was absent from the original pipeline. Without it, execution time increased by 20–25%. We suspected the original metadata was already suitably partitioned, but did not establish that explanation. This means “same number of images” is insufficient. Benchmark manifests must preserve row-group sizes, partitions, URI ordering, and object sizes.

Many small objects remain dangerous. None of the reviewed end-to-end tables publishes a small-file collapse threshold or peak RSS. We therefore cannot attach a trustworthy files-per-second number to that failure mode. We would test 5,000 small objects against five objects containing the same total bytes through MinIO or S3, recording request count, first-output latency, throughput, and peak RSS. Metadata calls and per-object connection overhead can dominate before model inference matters.

Batch size is still a production knob. In the reference runs, Ray's PDF embedding batch had to fall from 100 to 10 to avoid repeated failures. Audio used 64 and still experienced some failures while completing. Spark required spark.executor.cores=1 for most workloads plus spark.sql.execution.arrow.maxRecordsPerBatch. The Spark setting is global, so a safe batch for decoded images also constrains a lightweight text UDF.

Daft needed model-UDF batch tuning too, but its streaming backpressure avoided forcing every stage through the same global batch size. That is the most credible architectural advantage we found.

What We Could Not Verify

Our installation failed before we could complete the local decode test. We also could not verify peak RSS, a small-file failure threshold, or the share of wall-clock time spent on downloads, media decoding, embedding, and writes. The published tables are end-to-end. In the image-classification instance sweep, we observed GPU utilization below 20% for both Daft and Ray on four-CPU g6.xlarge workers. That identified upstream CPU starvation, but did not establish the precise share of time spent decoding. For our own deployments, per-stage timing is a release gate, not an optional dashboard.

Scale, Latency & Cost vs. Alternatives

The original eight-worker reference produced the following end-to-end results:

Workload	Daft 0.6.2	Ray Data 2.49.2	EMR Spark 7.10.0
Audio transcription, 113,800 files	6m 22s	29m 20s	25m 46s
Document embedding, 10,000 PDFs	1m 54s	14m 32s	8m 04s
Image classification, 803,580 images	4m 23s	23m 30s	45m 07s
Video object detection, 1,000 videos	11m 46s	25m 54s	3h 36m

Those numbers favor Daft heavily, but they do not settle the buying decision.

In our best-effort Ray Data reproduction, we could not access the original datasets for the audio, document, and image workloads. We reconstructed inputs using Common Voice, Digital Corpora, and an 800,000-image ImageNet subset, following the published methodology; the video workload used the original dataset. We dropped consumed columns from the Ray pipelines and used Ray Data 2.50 RC's newer download and Parquet optimizations for image classification. These results are useful for tuning, but they are not a byte-for-byte rerun of every original workload.

The CPU-to-GPU sweep mattered most. On image classification, Daft completed in about 315 seconds versus Ray Data 2.50 RC at about 456 seconds on four-CPU g6.xlarge workers. At 32 CPUs per GPU, Ray completed in about 111 seconds while Daft stayed near 195 seconds. In this image-classification sweep, engine ranking flipped as additional CPUs reduced upstream starvation; GPU utilization still remained low in absolute terms.

The cost-optimal points were much closer: approximately $0.44 per Daft run and $0.43 per Ray run, with Ray finishing in about 145 seconds versus Daft at about 202 seconds at each engine's lowest-cost configuration. For an 800,000-image run, that normalizes to roughly $0.55 per million images for Daft and $0.54 for Ray, excluding storage requests, output storage, model distribution, control-plane cost, retries, and engineering labor.

Decision factor	Daft	Ray Data	Spark
Multimodal API	Strong native expressions	Flexible Python operators and integrations	UDF-heavy
Backpressure	Integrated streaming model	Streaming execution with explicit tuning options	Partition and executor tuning
CPU/GPU scheduling	Simple, less manually exposed	Most flexible across resource shapes	Awkward for mixed Python/GPU stages
Memory controls	Per-UDF batching and adaptive streaming	Object store, actors, concurrency, batch and block controls	Executor memory and global Arrow batch controls
Operational maturity	Younger ecosystem	Strong fit if Ray is already deployed	Strongest enterprise footprint
Best fit	Declarative multimodal ETL	Tuned distributed AI pipelines	Existing SQL-centric data platform

The video comparison showed that output encoding also affected runtime. Daft used a Rust-native PNG encoder while the Ray implementation used Pillow. Pillow's default compression produced files 7.8% smaller than the Rust default; compression level two narrowed the discrepancy to 4.4%. Changing compression and Ray's batch size reduced its video runtime from roughly 1,590 seconds to 922 seconds, while Daft remained around 759 seconds on g6.xlarge workers. That benchmark was partly measuring encoder defaults.

Our break-even rule is therefore operational. If Ray already runs your training or serving estate, a difference of about one cent per 800,000-image run does not justify introducing another control plane. If a small team is about to adopt Spark solely for multimodal preprocessing, Daft is the more compelling first test because the developer and tuning burden can dominate compute cost. Teams evaluating a broader stack can also check our tools collection or bring a representative pipeline to our AI infrastructure services.

Our Final Verdict: When to Deploy, When to Skip

Deploy Daft if:

Your rows contain images, audio, PDFs, video, tensors, or embeddings rather than mostly scalar columns.
You want declarative filtering, projection, schema handling, and multimodal transformations in one pipeline.
Decode expansion and intermediate memory pressure are causing OOMs or excessive disk spilling.
You need the same logical pipeline to run locally and on a Ray-backed cluster.
Your team does not want to manage Spark executor cores and one global Arrow batch limit for heterogeneous UDFs.
You can pin Daft, Python, codecs, model packages, and worker images together.
You will benchmark your own object layout and CPU-to-GPU ratio instead of copying a vendor chart.

Hold off or avoid it if:

Ray already powers your distributed estate and your engineers are comfortable tuning actors, blocks, column retention, and resource requests.
Your workload benefits from more CPU capacity per GPU; the image-classification instance sweep shows Ray can win decisively on CPU-richer workers.
Your platform is fundamentally SQL and JVM-centric, with governance and operations already standardized on Spark.
You require a standalone, fully documented Flotilla deployment path separate from Ray or Kubernetes.
You cannot tolerate a younger API surface or enforce exact client-worker version parity.
You expect the engine to solve corrupt-media policy, corpus accounting, and small-file economics automatically.

We would start with Daft for a new multimodal ETL tier, but only after a controlled comparison. We would pin the environment, use identical bytes and output encoding, preserve the same partitions, sweep UDF batch sizes, test at least two CPU-to-GPU ratios, and capture per-stage wall clock plus peak RSS. We would also price storage requests and retries, not merely GPU seconds.

Daft's strongest case is not the headline 2–18x benchmark advantage. That figure changes when datasets, partitioning, CPUs, column retention, and encoder settings change. Its strongest case is that it gives us a cleaner way to express a multimodal pipeline while placing backpressure and bounded streaming inside the engine.

That is enough to make Daft worth a serious proof of concept. It is not enough to skip measurement. If you want help designing a reproducible comparison using representative production data, contact the Effloow team before committing to the cluster architecture.

OpenHands Self-Hosted Runtime Review: Docker Isolation, Agent Cost, and the Production Reality

Jangwook Kim — Mon, 12 Oct 2026 00:36:53 +0000

Why We Evaluated OpenHands

We brought OpenHands into our evaluation queue because buying another coding-agent seat does not solve the infrastructure question facing engineering teams. We wanted to know whether we could operate the execution layer ourselves, connect it to a model provider of our choice, and retain control over source code, runtime images, network policy, and usage accounting.

We often see OpenHands described as an open-source Devin alternative, but that description is too shallow for an architecture review. The interesting component is not the chat interface. It is the runtime responsible for turning model-generated actions into shell commands, file mutations, Python execution, and observations.

We traced that path through the OpenHands runtime architecture. The backend launches a purpose-built runtime container, starts an action execution server inside it, sends actions over an HTTP interface, and receives structured observations. The model remains outside that execution boundary. The container is where potentially destructive commands run.

That separation is useful. It gives us an explicit control point for image policy, workspace mounts, resource limits, process cleanup, and audit logging. It also means OpenHands is not merely an application that happens to use Docker. Docker is part of its security and scheduling model.

The image workflow is more sophisticated than a naïve build-per-task design. We found three levels of runtime tags:

A versioned tag tied to the OpenHands version and base image.
A lock tag derived from the base image and dependency files.
A source tag incorporating the OpenHands source hash.

When an exact source image exists, the runtime can reuse it. When only the dependency-compatible layer exists, it can copy changed source without repeating the slow dependency installation path. That design should reduce repeated build work, although we would still pin immutable image digests rather than rely on a moving tag in production.

The project’s 90,000-plus-star GitHub repository made it impossible to dismiss as a weekend prototype. We treated that popularity as an adoption signal, not evidence of secure isolation or predictable operating cost.

Our central questions were therefore practical:

Can we reproduce the documented local deployment without hidden services?
Does the sandbox protect the host under a realistic threat model?
Can we measure model spend per conversation?
What happens when multiple sandboxes need ports, storage, and cleanup?
At what team size does self-hosting become cheaper than buying hosted seats?

We also wanted to separate what we could verify from what required a functioning Docker daemon and paid model credentials. That distinction became important almost immediately.

Hands-On Walkthrough: Setup, Execution & Output

We started with the supported local prerequisites: Python 3.12, uv, and a Docker installation with access to the Docker socket. We treated the 4 GB host-memory recommendation as a setup baseline, not a measured per-sandbox footprint. It is a minimum host recommendation, not a capacity-planning number.

The shortest launcher path was:

uv tool install openhands --python 3.12
openhands serve --mount-cwd

The --mount-cwd option mounts the current repository into the workspace. That convenience has a direct security consequence: an agent with write access can modify anything exposed through that mount.

For a more controlled reproduction, we prepared a command with explicit application and agent-server versions rather than using latest:

docker run -it --rm --pull=always \
  -e AGENT_SERVER_IMAGE_REPOSITORY=ghcr.io/openhands/agent-server \
  -e AGENT_SERVER_IMAGE_TAG=1.26.0-python \
  -e LOG_ALL_EVENTS=true \
  -v /var/run/docker.sock:/var/run/docker.sock \
  -v "$HOME/.openhands:/.openhands" \
  -p 3000:3000 \
  --add-host host.docker.internal:host-gateway \
  --name openhands-app \
  docker.openhands.dev/openhands/openhands:1.8

This command exposes the web application on http://localhost:3000. More importantly, it mounts /var/run/docker.sock into the application container so OpenHands can create and manage sandbox containers.

We hit a hard environmental stop before a useful runtime benchmark: our available runner did not expose a Docker daemon. We could inspect the launch configuration, but we did not run a deployment-contract test or measure container startup latency, memory consumption, concurrent sandbox throughput, or cleanup behavior on that runner. We did not replace those missing measurements with invented numbers.

For a host with Docker available, the SDK-level sandbox path is straightforward:

import platform

from openhands.workspace import DockerWorkspace


def detect_platform() -> str:
    machine = platform.machine().lower()
    if "arm" in machine or "aarch64" in machine:
        return "linux/arm64"
    return "linux/amd64"


with DockerWorkspace(
    server_image="ghcr.io/openhands/agent-server:1.26.0-python",
    host_port=8010,
    platform=detect_platform(),
) as workspace:
    result = workspace.execute_command(
        "printf 'sandbox-ready\\n' && pwd && python --version"
    )
    print(
        {
            "command": result.command,
            "exit_code": result.exit_code,
            "stdout": result.stdout,
            "stderr": result.stderr,
        }
    )

The expected response shape below is simulated to make the reproduction contract explicit. It is not presented as a measurement from our Docker-unavailable runner:

{
  "command": "printf 'sandbox-ready\\n' && pwd && python --version",
  "exit_code": 0,
  "stdout": "sandbox-ready\n/workspace\nPython 3.12.x\n",
  "stderr": ""
}

DockerWorkspace is the right choice when we can use a prebuilt agent-server image. DockerDevWorkspace builds from a selected base image and is appropriate when we need additional compilers, system packages, or internal tooling. We would not use the development workspace on a latency-sensitive request path because image construction adds an avoidable startup stage.

A real agent conversation introduces another dependency: an LLM credential. For our API-backed conversation design, we use LLM_API_KEY and LiteLLM-style model identifiers such as anthropic/... or openai/..., with accumulated cost reporting available through:

cost = (
    conversation.conversation_stats
    .get_combined_metrics()
    .accumulated_cost
)

We used that metric as the required accounting surface in our production design. We would record it by conversation, repository, model, tenant, and completion state. We would not treat it as a complete bill until reconciling it against provider invoices, retries, failed requests, cache behavior, and any model calls made outside the conversation object.

A local model endpoint can use a placeholder API-key value when its server does not authenticate requests. That removes the paid-provider key dependency, but it does not remove inference cost. GPU depreciation, idle capacity, model serving, and weaker task completion can easily cost more than API tokens.

Deployment Limitations and Operational Risks

The first failure was simple and decisive: without Docker, there was no meaningful OpenHands sandbox benchmark. A mock Docker client could verify Python control flow, but it could not verify namespaces, mounts, port binding, image pulls, process termination, or actual container cleanup. We rejected a mock-only result as production evidence.

The second issue was the Docker socket. The architecture isolates agent commands in a child container, but the parent OpenHands application requires control of the host Docker daemon in the direct deployment. On a conventional Linux host, access to that socket is effectively privileged host control. If the application container is compromised, the sandbox boundary does not protect the Docker host from the controller.

Our workaround is architectural rather than cosmetic:

Run OpenHands workers on dedicated nodes or disposable virtual machines.
Do not colocate them with databases, CI secrets, or customer-facing services.
Restrict outbound network access from both controller and sandbox.
Mount only the repository needed for the task.
Prefer read-only mounts plus an explicit writable worktree where possible.
Apply CPU, memory, process, and execution-time limits.
Treat the Docker host as replaceable after hostile workloads.

OpenHands supports bind mounts, named volumes, and overlay copy-on-write behavior. Overlay mode requires a bind mount marked with :overlay and a writable SANDBOX_VOLUME_OVERLAYS host directory. If we do not configure that directory, OpenHands skips the overlay path. We would fail startup on that condition rather than silently continue with different write semantics.

When planning the reverse-proxy deployment, we identified another design problem. Each sandbox normally receives random host ports, while generated browser URLs default to a localhost pattern. Setting OH_WEB_URL does not rewrite those sandbox URLs; it affects the allowed origin rather than the hostname and port used to reach the container.

We found two imperfect options.

The first is to set OH_SANDBOX_CONTAINER_URL_PATTERN to a public hostname containing {port}. That retains a random port for each sandbox, forcing the edge layer to forward a dynamic range.

The second is host networking:

export AGENT_SERVER_USE_HOST_NETWORK=true

This exposes fixed ports, including agent-server port 8000 and editor-related port 8001. The trade-off is severe: every sandbox tries to bind the same ports, so only one sandbox can run on that host without collision. We would not use host networking for a multi-tenant production worker.

We also found that editor capabilities vary by image. We should not assume portable SDK images include VS Code, VNC, a fixed editor port, or /api/vscode/url. The extra_ports=True pattern belongs to selected legacy editor-capable images, where host_port + 1 and host_port + 2 may expose VS Code and VNC. We would feature-detect those services instead of deriving URLs unconditionally.

Finally, moving tags are a reproducibility risk. ghcr.io/openhands/agent-server:latest-python is convenient for exploration, but production rollouts need pinned versions or, preferably, image digests. We would test the application image, agent-server image, base image, SDK, and model configuration as one release unit.

What This Evaluation Could Not Verify

We could not measure OpenHands startup latency, throughput, or memory use because our runner lacked Docker. The available setup material also does not provide a reproducible production benchmark.

Scaling and Cost vs. Alternatives

What we can evaluate is the cost and scaling shape.

Option	Execution ownership	Isolation model	Model cost	Operational burden	Best fit
OpenHands self-hosted	Our team	Docker sandbox managed by OpenHands	Paid API or local inference	High	Teams needing model and runtime control
E2B-style managed sandbox	Provider	Remote managed sandbox	Model plus sandbox usage	Medium	Fast deployment without worker operations
Custom Docker runner	Our team	Whatever we build and audit	Paid API or local inference	Very high	Specialized policies or narrow workflows
Hosted coding-agent seat	Vendor	Vendor-managed	Usually bundled or plan-dependent	Low	Teams prioritizing speed over control
IDE agent	Developer machine	Local permissions or editor controls	Subscription or API	Low to medium	Interactive individual development

OpenHands removes per-seat runtime licensing from the equation, but it does not make agent execution free. Our monthly model is:

OpenHands monthly cost =
    worker infrastructure
  + engineering maintenance
  + observability and storage
  + LLM or local inference
  + security and incident overhead

The corresponding hosted calculation is:

Hosted monthly cost =
    number of seats × effective monthly seat price
  + overages
  + integration overhead

For an illustrative break-even model, assume:

Worker infrastructure and logs: $600 per month.
Runtime maintenance: 12 engineering hours per month.
Loaded engineering rate: $150 per hour.
Self-hosted model usage: $120 per active user per month.
Hosted alternative: $500 per active user per month.

That produces $2,400 in fixed monthly self-hosting cost and a $380 per-user advantage:

Break-even users = $2,400 / ($500 - $120) = 6.32

Under those assumptions, self-hosting becomes cheaper at approximately seven active users. This is a planning example, not a quoted Devin price or an observed OpenHands bill. If maintenance rises to 30 hours, or if agents burn tokens through repeated tool loops, the break-even point moves sharply upward.

Agent-loop cost is the largest variable. A task is not one model call. The agent may inspect files, invoke tools, ingest observations, revise its plan, run tests, encounter errors, and ask the model again. Long command output also returns to context and can increase subsequent request cost.

We would enforce four controls before opening access:

Maximum model spend per conversation.
Maximum loop iterations and wall-clock duration.
Output truncation and artifact storage outside the prompt.
Per-team and per-repository budgets with hard rejection.

For teams building this into a broader internal platform, our AI infrastructure services cover the worker isolation, observability, and cost-control layer that the default local launcher does not provide. We also maintain a broader set of implementation references in our tools collection.

Our Final Verdict: When to Deploy, When to Skip

OpenHands has a credible runtime design. We like the separation between the agent backend and the in-container action server, the prebuilt DockerWorkspace path, the customizable development workspace, and the exposed conversation-cost metric.

We do not consider the default single-host Docker deployment a complete production security boundary. The Docker socket, repository mounts, dynamic ports, image compatibility, outbound network policy, and cleanup behavior all become our responsibility the moment we self-host.

Deploy this if:

We need control over model providers, runtime images, and source-code location.
We can dedicate isolated worker hosts to untrusted agent execution.
We have enough active usage to absorb recurring platform maintenance.
We can pin and test the full application and agent-server image set.
We will enforce token, iteration, time, memory, and network limits.
We accept that self-hosting transfers operational risk rather than eliminating cost.

Hold off or avoid it if:

We need a turnkey service with contractual support and predictable seats.
We plan to expose the host Docker socket on shared infrastructure.
We cannot operate dynamic sandbox ports behind our edge architecture.
We expect bundled VS Code or VNC capabilities in every SDK image.
We lack per-conversation model budgets and provider-bill reconciliation.
We need audited multi-tenant isolation stronger than ordinary Docker containers.
We need benchmark-backed capacity numbers before running our own Docker tests.

Our recommendation is a guarded deploy for a controlled pilot, not an immediate shared production rollout. Start with a dedicated worker, one pinned image set, outbound-deny networking, disposable repository copies, and a small group of engineers. Measure cold and warm startup separately, record accumulated model cost, test cleanup after forced termination, and attempt deliberate workspace escapes before expanding.

OpenHands can reduce vendor lock-in and give us a useful coding-agent control plane. It does not remove the hard parts. It makes those parts visible—and assigns them to our infrastructure team. Teams that want help validating that trade-off against their own workload can contact us with their concurrency, repository, compliance, and model assumptions.

Inspect AI in Production: A Hard-Nosed Review of Agent Evals, Logs, and Model-Upgrade Gates

Jangwook Kim — Sun, 11 Oct 2026 00:39:48 +0000

Why We Evaluated Inspect AI

The model upgrade itself is rarely the difficult part. The difficult part is proving that the new model did not silently damage extraction accuracy, tool behavior, refusal handling, or operational cost.

Inspect AI Verdict

Inspect AI provides the execution and artifact layer for item-level agent-eval gates, but raw log reproducibility, scorer contracts, sandbox boundaries, and cost accounting remain engineering responsibilities.

We have seen the same internal harness emerge repeatedly: a JSONL dataset, an asynchronous request loop, a pile of retry logic, a scorer script, and a spreadsheet comparing the old model with the new one. It works until an incident forces us to answer questions the harness was never designed to answer:

What exact prompt and tool trajectory produced this result?
Was the failure caused by the model, the grader, the provider, or the sandbox?
Which individual samples regressed?
Did the replacement model consume more tokens despite improving pass rate?
Can we reproduce the evaluation from a pinned task and immutable artifact?

We evaluated Inspect AI because it handles that entire execution layer. An Inspect evaluation combines a dataset, a solver, and one or more scorers. The solver may be a single generation call, a custom function decorated with @solver, or a multi-turn agent. The scorer may perform exact matching, inclusion matching, model grading, or arbitrary custom logic.

The framework also writes structured .eval logs and ships the inspect view browser interface. That separation matters. We do not want the dashboard to be the source of truth; we want the artifact to be the source of truth and the dashboard to be one way of reading it.

The agent surface was another reason for testing it. Inspect accepts agents in the solver position and includes a ReAct loop, tool interfaces, message and token limits, custom agents, agent handoffs, and adapters for external agents. Its tool catalog includes shell, Python, text editing, web, computer, and MCP-oriented interfaces. For dangerous execution, it supports sandbox backends including Docker and more infrastructure-heavy options.

We also inspected the Inspect Evals repository rather than evaluating the framework only with toy arithmetic. It provides a substantial task corpus and demonstrates how real evaluations organize datasets, task versions, dependencies, scoring, and sandbox assets. We still recommend pinning a small subset into an internal regression suite rather than importing a moving benchmark wholesale.

Our short conclusion before the details: Inspect gives us a much stronger foundation than another home-grown request loop. It does not, by itself, make a benchmark deterministic, a model grader trustworthy, or a tool safe.

Hands-On Walkthrough: Setup, Execution & Output

We ran the offline scorer check with Python 3.12 in an isolated container. For the local walkthrough, we recommend a virtual environment. That was deliberate: Python 3.11 and 3.12 are the Inspect Evals project's preferred targets, while later Python versions carry compatibility caveats for parts of the broader task corpus.

For the local walkthrough, use this installation path:

python3.12 -m venv .venv
source .venv/bin/activate

python -m pip install --upgrade pip
python -m pip install "inspect-ai==0.3.100" openai

export OPENAI_API_KEY="replace-me"
mkdir -p logs

The pinned inspect-ai wheel resolved successfully in our environment. The wheel we downloaded had this SHA-256 digest:

78e4c5b7e426fcc25563cd68f9f976a1c18c00adc9f4955bde9a6c97016363d7

For production, we would pin the complete lockfile rather than only the top-level package. Inspect pulls in a nontrivial dependency tree that includes asynchronous HTTP, schema, filesystem, terminal UI, and cloud-storage packages.

We then reduced the task shape to the smallest useful production example: registered task, registered solver, custom partial-credit scorer, stable sample IDs, and an explicit log directory.

# regression_eval.py
import os
import re

from inspect_ai import Task, eval, task
from inspect_ai.dataset import Sample
from inspect_ai.scorer import Score, Target, scorer
from inspect_ai.solver import Generate, TaskState, solver


@solver
def answer_once():
    async def solve(state: TaskState, generate: Generate) -> TaskState:
        return await generate(state)

    return solve


def normalize(text: str) -> str:
    return re.sub(r"\s+", " ", text.strip().lower())


@scorer
def partial_answer():
    async def score(state: TaskState, target: Target) -> Score:
        answer = normalize(state.output.completion)
        expected = normalize(target.text)

        if answer == expected:
            value = 1.0
            reason = "Exact normalized match"
        elif expected in answer:
            value = 0.5
            reason = "Target present with additional text"
        else:
            value = 0.0
            reason = "Target absent"

        return Score(
            value=value,
            answer=state.output.completion,
            explanation=reason,
        )

    return score


@task
def upgrade_gate():
    return Task(
        dataset=[
            Sample(
                id="refund-window",
                input="Reply with only the refund window: 30 days",
                target="30 days",
            ),
            Sample(
                id="support-tier",
                input="Reply with only the support tier: enterprise",
                target="enterprise",
            ),
        ],
        solver=answer_once(),
        scorer=partial_answer(),
    )


if __name__ == "__main__":
    eval(
        upgrade_gate(),
        model=os.environ.get("INSPECT_MODEL", "openai/gpt-5"),
        log_dir="logs",
    )

To execute this worked example through Python or the CLI and inspect its logs, use:

export INSPECT_MODEL="openai/gpt-5"
python regression_eval.py

# Equivalent task discovery through the CLI
inspect eval regression_eval.py@upgrade_gate \
  --model "$INSPECT_MODEL" \
  --log-dir logs

inspect view --log-dir logs

A successful run should produce output in this general shape. The following console block is illustrative, not a measurement from our failed qualification run:

$ inspect eval regression_eval.py@upgrade_gate \
    --model openai/gpt-5 \
    --log-dir logs

upgrade_gate (2 samples): complete
Model: openai/gpt-5
Log: logs/2026-10-11T101530_upgrade-gate.eval

Scores:
  partial_answer: 0.750

Samples:
  completed: 2
  errors:    0

For an agent task, the task definition changes more than the execution command. This is the minimal pattern we would use before adding a pinned challenge dataset:

from inspect_ai import Task, task
from inspect_ai.agent import react
from inspect_ai.scorer import includes
from inspect_ai.tool import bash, python, todo_write


@task
def sandbox_agent_eval():
    return Task(
        dataset=load_pinned_samples(),
        solver=react(
            prompt="Use the available tools and submit only the final answer.",
            tools=[bash(), python(), todo_write()],
            attempts=3,
        ),
        scorer=includes(),
        sandbox="docker",
        message_limit=30,
    )

The important production property is not the decorator syntax. It is that sample inputs, targets, messages, tool calls, outputs, scores, and run configuration can travel together in one reviewable artifact.

Evaluation Failures and Verification Limits

Our first narrow scorer contract test did not complete successfully.

We created five fixed completion fixtures to compare includes() with match(location="exact"). One fixture was intentionally labelled as an incomplete-flag negative case. Our test expected the exact matcher to return incorrect, but Inspect returned C, causing this assertion failure:

AssertionError:
('incomplete_flag_negative', 'exact', 'C', 'I')

We do not treat that as proof that Inspect's scorer is defective. It shows that this fixture's expected result did not match the scorer's returned value. The assertion alone does not establish whether normalization, answer extraction, fixture construction, or another factor caused the mismatch. Because the assertion terminated the run, we did not obtain a complete five-fixture result set.

That failure changed our implementation policy: every built-in scorer we adopt gets a table-driven contract suite containing punctuation, surrounding prose, denial text, case changes, whitespace, multiple answers, and truncated targets. A scorer name is not a specification.

Partial credit introduces another interpretation problem. A score set of 0, 0.5, and 1 is easy to emit, but the business meaning of its aggregate is ours to define. A plain arithmetic mean treats two partial passes as equivalent to one full pass and one complete failure:

mean([0.5, 0.5]) == mean([1.0, 0.0]) == 0.5

Those distributions carry different operational risk. We therefore preserve at least four outputs: full-pass rate, partial-pass rate, hard-failure rate, and mean score. We would not approve a model upgrade from the mean alone.

We have not established byte-for-byte .eval reproducibility. Our offline scorer check did not run evaluations or compare log files; log reproducibility requires a separate experiment. Consequently, we have no defensible raw SHA-256 comparison to publish.

Even after a successful run, we would separate two requirements:

Artifact integrity: the stored .eval file remains unchanged after creation.
Semantic reproducibility: equivalent runs contain the same sample outputs and scores after excluding run-specific metadata.

Raw byte equality is a stricter test. Run IDs, timestamps, ordering, provider metadata, and archive serialization can invalidate it even when the evaluation result is semantically identical. Our CI gate would hash the original artifact for custody, then generate a normalized comparison document for regression analysis.

Model grading creates another source of nondeterminism. Exact match is brittle but inspectable. A model grader is flexible but adds a second model call, another prompt, another model version, and another failure mode. We did not complete a valid exact-match-versus-model-grader drift experiment, so we will not invent a disagreement rate. Our deployment rule is still clear: we pin the grader independently, store its explanation, and maintain a human-adjudicated calibration set.

The sandbox boundary also needs precise language. Adding bash() does not automatically isolate anything. In our task definition, sandbox="docker" is what assigns shell and Python execution to the container. We would not assume that custom Python tools are isolated; we would verify their execution context and route dangerous operations through a sandbox-aware interface.

We therefore treat the evaluator host as sensitive infrastructure:

We run it with minimal credentials.
We mount only required fixtures.
We remove Docker socket access from evaluated containers.
We disable outbound networking unless the task requires it.
We apply message, token, time, and cost limits.
We assume model-generated shell commands are hostile.

Finally, we did not complete a provider rate-limit benchmark or a two-model token-cost comparison. The available run failed before remote inference. We cannot publish retry counts, wall time, throughput, or cost per sample from that attempt.

The workaround is procedural, not rhetorical: record provider errors as their own failure category, pin concurrency, preserve token usage per sample, and rerun the exact same sample IDs against both model versions. We would reject any benchmark table that silently drops exhausted retries.

Scale, Latency & Cost vs. Alternatives

Inspect's strongest comparison is not against an observability dashboard. It is against the bespoke evaluation runner that engineering teams eventually build around raw provider SDKs.

Option	Execution model	Item-level scoring	Agent trajectories	Isolation	Regression artifacts	Main cost
Inspect AI	Local Python runner with provider APIs	Strong	Strong	Explicit sandbox configuration	Structured `.eval` logs	Integration and evaluation design
Bespoke Python harness	Whatever we implement	Variable	Usually custom work	Usually custom work	Usually JSONL or database rows	Engineering ownership
Langfuse or Phoenix-style observability	Trace collection and analysis	Possible, but not the core runner	Strong tracing	Outside the primary scope	Trace-oriented	Backend operation and instrumentation
Prompt-oriented CI harness	Configuration-driven test execution	Strong for prompt and security checks	Depends on integration	Depends on provider and tool setup	CI reports and artifacts	Configuration growth
Commercial evaluation platform	Hosted execution and dashboards	Usually strong	Product-dependent	Product-dependent	Vendor-managed	Subscription, data governance, lock-in

We did not produce valid latency or throughput numbers, so we do not present fictional requests-per-second figures. For our deployment benchmark, we would measure provider latency, agent trajectory length, tool execution time, and local orchestration overhead separately before identifying the dominant bottleneck.

Our cost gate uses artifact-derived quantities:

sample_cost =
    input_tokens  × input_price_per_token
  + output_tokens × output_price_per_token
  + grader_input_tokens  × grader_input_price_per_token
  + grader_output_tokens × grader_output_price_per_token
  + external_tool_cost

We calculate the model-upgrade delta over identical sample IDs:

full_run_delta =
    sum(new_model_sample_costs)
  - sum(old_model_sample_costs)

We then combine quality and cost:

cost_per_additional_full_pass =
    full_run_delta
    / (new_full_pass_count - old_full_pass_count)

If the denominator is zero, this ratio is undefined; an upgrade can still improve cost efficiency by preserving quality at a lower full-run cost. Equal full-pass counts alone do not establish preserved quality, so we also review item-level regressions and failure categories. If the denominator is negative, we assess the quality loss and cost change separately rather than interpreting the ratio as a cost per additional full pass. If model grading is enabled, grader tokens stay separate from candidate-model tokens so that a grading prompt change cannot masquerade as candidate-model cost growth.

The build-versus-buy calculation is similarly direct:

annual_internal_harness_cost =
    initial_engineering_hours × loaded_hourly_rate
  + annual_maintenance_hours × loaded_hourly_rate
  + incident_and_audit_overhead

annual_inspect_cost =
    integration_hours × loaded_hourly_rate
  + task_maintenance_hours × loaded_hourly_rate
  + model_and_tool_usage
  + sandbox_infrastructure

Inspect is open source, but it is not free to operate. The expensive parts are representative datasets, reliable scorers, model calls, sandbox execution, and review of ambiguous failures. Inspect removes a large amount of harness plumbing; it does not remove evaluation engineering.

For broader implementation help, we keep our infrastructure patterns in the Effloow tools collection. For teams turning an ad hoc benchmark into a release gate, our AI engineering services cover dataset versioning, grader calibration, and CI integration.

Our Final Verdict: When to Deploy, When to Skip

We would deploy Inspect AI as the execution and artifact layer for a model-upgrade gate, with conditions.

Deploy it if:

We need repeatable item-level comparisons rather than aggregate dashboard metrics.
We evaluate multi-turn agents, tool calls, or sandboxed code execution.
We want task, solver, and scorer logic stored as reviewable Python.
We need transcripts and scoring explanations for incident review.
We are willing to pin framework, task corpus, dataset, model, grader, and pricing versions.
We can maintain scorer contract tests and a human-adjudicated calibration subset.
We want to avoid building logging, resumption, trajectory capture, and visualization from scratch.

Hold off or avoid it if:

We only need a few deterministic HTTP assertions in CI.
We require raw byte-identical logs and have not built normalization and integrity checks.
We expect model-graded scores to be stable without pinning and calibration.
We cannot isolate model-generated code from evaluator credentials and host resources.
We need a fully managed dashboard with organizational workflows more than a Python evaluation framework.
We are unwilling to inspect failed samples and only want a single pass-rate number.

Our production verdict is adopt with explicit validation and operational controls.

The component model is clean, the agent support is substantive, the task corpus is useful, and inspect view makes trajectory debugging materially easier. Inspect can replace weeks of basic harness construction.

Our own validation was limited, however. Installation succeeded, but the offline scorer check terminated on an assertion mismatch before producing a complete five-fixture result set. Log reproducibility, provider rate-limit behavior, model-grading drift, and per-sample upgrade cost were outside that check's scope and remain untested. We would not promote that check into a production benchmark table.

That restraint is part of the recommendation. Inspect provides the execution and logging components for an auditable release gate. We still have to prove the scorer, define the sandbox boundary, preserve the artifacts, and make the economics explicit.

If those controls are already on your roadmap, Inspect is one of the first frameworks we would prototype. If you want help designing the gate around your own failure modes, contact Effloow.

vllm-mlx on Apple Silicon: Continuous Batching Works, but Unified Memory Sets the Real Limit

Jangwook Kim — Sat, 10 Oct 2026 00:36:02 +0000

Why We Brought This Tool Into Our Lab

The appeal of Apple Silicon inference is easy to understand. A sufficiently large Mac can keep quantized models, application services, and development tooling on one quiet machine without a remote GPU endpoint. The harder question is whether that machine can serve several interactive users without turning every request into a serial queue.

M4 Max continuous batching speedups (5 concurrent requests)Llama-3.2-1B-Instruct-4bit (5 req) 2.05xLlama-3.2-3B-Instruct-4bit (5 req) 1.51xQwen3-0.6B-8bit (5 req) 3.39xQwen3-30B-A3B-4bit (5 req) 2.38xQwen2.5-1.5B-Instruct-4bit (5 req) 1.64x

Batching five concurrent requests yields 1.51x to 3.39x aggregate-throughput gains on the cited M4 Max models, so the multiplier depends on model, architecture, and memory bandwidth rather than being fixed.

That is the problem we evaluated with vllm-mlx. It exposes OpenAI- and Anthropic-compatible APIs over native MLX execution and adds the scheduling features missing from a basic mlx-lm generation loop: continuous batching, paged KV storage, prefix caching, request scheduling, and streaming.

Continuous batching is the important part. Instead of finishing request A before starting request B, the scheduler admits new requests into an active batch and advances multiple sequences together. Model weights can then be reused across rows in the same decode step. Aggregate throughput rises, although each client may receive tokens more slowly because it shares the machine.

The operational distinction matters:

Simple mode processes one request at a time and avoids batching overhead.
Continuous-batching mode combines active requests and optimizes aggregate throughput.
Paged caching stores KV state in fixed-size blocks and can share repeated prefixes.
Prefix caching helps only when prompts actually share reusable content.
None of these mechanisms creates additional physical memory.

Our evaluation hit an important boundary before model execution: the available isolated host was Linux without the Apple Silicon, Metal, and model weights needed for these tests. MLX scheduling, KV growth, and unified-memory exhaustion cannot be reproduced faithfully on that platform. We therefore did not fabricate a local M4 result or present a mocked scheduler as a real benchmark.

Instead, we reviewed the installation instructions and CLI options, then assessed throughput and configuration trade-offs using the Apple Silicon benchmark cases for vllm-mlx continuous batching, mlx-serve performance, and mlxcel scheduling. We did not execute an installation or CLI contract test in the sandbox. We used the throughput values below as benchmark reference points, not as measurements from this Linux host or a validated production capacity envelope.

For our vllm-mlx throughput evaluation, we used the benchmark case with an M4 Max, 128 GB of unified memory, and five concurrent requests. Aggregate throughput increased from 328.1 to 1,111.8 tokens per second for Qwen3-0.6B-8bit, from 98.1 to 233.3 tokens per second for Qwen3-30B-A3B-4bit, and from 196.9 to 322.2 tokens per second for Qwen2.5-1.5B-Instruct-4bit. Those are 3.39x, 2.38x, and 1.64x gains respectively.

That spread is the real story. “Continuous batching” is not a fixed multiplier. Gains depend on architecture, model size, memory bandwidth, batch width, KV traffic, prompt shape, and whether the implementation has a genuinely batched forward path.

Hands-On Walkthrough: Setup, Execution & Output

We pinned the walkthrough to a specific release rather than installing whichever version was latest. For this review, that reference was vllm-mlx==0.4.1.

On an Apple Silicon host, our deployment recipe would be:

python3 -m venv .venv
source .venv/bin/activate

python -m pip install --upgrade pip
python -m pip install "vllm-mlx==0.4.1" openai

vllm-mlx model inspect \
  mlx-community/Llama-3.2-3B-Instruct-4bit

vllm-mlx model acquire \
  mlx-community/Llama-3.2-3B-Instruct-4bit \
  --target-dir ./models/llama-3b-4bit

vllm-mlx serve ./models/llama-3b-4bit \
  --served-model-name llama-3b \
  --host 127.0.0.1 \
  --port 8000 \
  --continuous-batching \
  --use-paged-cache \
  --cache-memory-mb 2048 \
  --max-num-seqs 8 \
  --stream-interval 1 \
  --enable-metrics

We would not expose this listener directly to an untrusted network. The production command should additionally set --api-key, place the process behind a health-aware proxy, and apply an external concurrency limit below the server’s theoretical sequence ceiling.

The built-in benchmarker provides a practical first pass:

vllm-mlx bench-serve \
  --url http://127.0.0.1:8000 \
  --model llama-3b \
  --prompts short,medium,long \
  --concurrency 1,2,4,8 \
  --max-tokens 256 \
  --repetitions 5 \
  --scrape-metrics true \
  --format json \
  --output bench.json

The following is an illustrative reporting schema we would use for automation, not a verified bench-serve output format or a measured result. We would check the actual benchmark output on Apple Silicon before implementing a parser:

{
  "run_metadata": {
    "model": "llama-3b",
    "concurrency": 4,
    "max_tokens": 256,
    "repetitions": 5,
    "result_kind": "simulated-example"
  },
  "metrics": {
    "requests_completed": 20,
    "requests_failed": 0,
    "aggregate_output_tokens_per_second": null,
    "median_ttft_ms": null,
    "p95_ttft_ms": null,
    "peak_metal_memory_bytes": null
  },
  "note": "Populate metric values only from the target Apple Silicon host."
}

For capacity testing, we would sweep three axes independently:

Concurrency: 1, 2, 4, 8, and then higher only if memory headroom remains.
Prompt length: approximately 512, 2,048, 8,192, and 16,384 tokens.
Generated length: 128 and 512 tokens.

Mixing all three variables in one first run makes the result difficult to interpret. Long prompts stress prefill and KV allocation; long outputs stress sustained KV growth; higher concurrency multiplies active sequence state.

We treat the default --max-num-seqs 256 as a scheduler ceiling, not a safe production batch size. A 256-sequence limit says what the scheduler may admit. It says nothing about whether a specific model, context length, and Mac can hold those sequences.

For streaming, --stream-interval 1 sends every token immediately. Raising it to 2–5 reduces delivery overhead but creates chunkier output. Values of 10 or more favor throughput over perceived smoothness. We would benchmark this separately from model throughput because HTTP chunk frequency can distort client-side latency without changing core decode speed.

Evaluation Limits and Production Risks

The first failure was environmental and absolute: no Apple Silicon meant no honest MLX performance run. MLX’s Metal execution and Apple’s unified-memory behavior cannot be inferred from CUDA, CPU emulation, or a fake cache implementation. We stopped that path rather than manufacturing numbers.

The second problem is that memory controls are easy to overestimate. --cache-memory-mb 2048 limits cache consumption; it is not a two-gigabyte limit for the entire process. Model weights, temporary prefill tensors, active KV state, tokenizer buffers, Metal allocations, the server, and macOS still compete for unified memory.

The default memory-aware cache fraction is 20% of available RAM. That is useful for containing prefix-cache growth, but it does not guarantee that the runtime will survive a large active batch. Likewise, --memory-budget-gb applies to registry-managed model-weight residency. It is explicitly not a total runtime-memory cap and does not guarantee prevention of a Metal or MLX out-of-memory failure.

That leaves us uncertain about how vllm-mlx handles unified-memory exhaustion. We found no verified basis for promising transparent CPU spill, automatic model eviction, or graceful continuation of active generations. Our production workaround would be preventive:

Reserve memory for macOS and colocated services.
Keep the model-weight footprint well below physical capacity.
Cap request prompt length and max_tokens.
Set a conservative application-level concurrency limit.
Reject or queue excess work before it reaches MLX.
Restart the worker after a fatal Metal allocation failure.
Run the API gateway outside the inference process.

Paged cache is valuable, but its memory savings depend on the workload. It reduces fragmentation and enables block sharing when requests have common prefixes. In our cache evaluation, we treated the 80%+ memory-saving figure as applying to shared-prefix KV storage for 10 or more concurrent users sharing the same system prompt, not to total process memory. We would not apply that estimate to unrelated prompts, unique RAG context, or independent coding sessions.

Prompt length also changes the result. The vllm-mlx M4 Max batching table does not disclose enough prompt and generation details to turn its speedups into a universal sizing rule. For mlxcel, we evaluated a more tightly specified benchmark case: on an M1 Ultra, meta-llama-3.1-8b-instruct-4bit, a 512-token prompt, and 128 generated tokens, four clients reached 107.9 aggregate tokens per second versus 56.8 for one batched client. Mean time to first token under four-client load was 189 ms, compared with 3,257 ms in the serial parallel=1 baseline.

That result shows why scheduling helps under contention, but it does not establish performance at 16K prompts or on an M4 Max.

Long prompts can also stall active decodes unless prefill is chunked. mlxcel’s scheduler makes this explicit with a default 2,048-token prefill chunk and a grant interval that lets decode work proceed between prompt chunks. vllm-mlx exposes a batching scheduler, but we would still test mixed workloads rather than assume a long ingestion request cannot disrupt short chats.

Batching support also depends on model architecture. Some implementations advertise batching even when a model family executes one single-sequence forward per row and merely evaluates those graphs together. mlxcel explicitly clamps unsupported SSM, hybrid, and mixed-cache families to one active batching row. Its native batched-forward list includes Llama 3, Llama 4, Qwen 3, Qwen 3.5, Gemma 3, Helium, Muse Glimmer, and Qwen3-MoE in the referenced implementation. We therefore need to verify model support for each family rather than infer it from a server-wide feature flag.

For more deployment reviews that separate API compatibility from actual serving behavior, browse our infrastructure tools collection.

Scale, Latency & Cost vs. Alternatives

We compared the three MLX-serving approaches around the behavior that matters in production rather than headline single-stream decode.

Option	Default concurrency posture	Verified reference behavior	Memory-pressure behavior	Best fit
vllm-mlx	Continuous batching is opt-in; `max-num-seqs` defaults to 256	On M4 Max 128 GB, five-request throughput improved 1.51x to 3.39x across the five models listed above	Cache can be capped, but that is not a total MLX memory limit	OpenAI and Anthropic compatibility, broad features, shared Mac services
mlx-serve	`--max-concurrent 1` by default	About 1.6x total throughput at four-way concurrency on dense models, with per-request latency cost	KV quantization trades speed for 2x or 4x KV-memory savings	Performance-focused local serving with explicit tuning
mlxcel	Four active sequences by default	On M1 Ultra and the specified 8B workload, four clients delivered 107.9 aggregate tok/s and 189 ms mean TTFT	Paged-KV admission control is designed to return backpressure under KV-budget pressure; the budget guard is inert on dense decode	Strong scheduler controls, bounded queues, and explicit admission policy
Direct `mlx-lm` loop	Usually one request at a time	No batching benefit	Application must manage all queueing and recovery	Single-user tools and offline jobs
Remote GPU endpoint	Provider-dependent	Scales beyond one Mac when capacity is purchased	Memory management is delegated to the provider	Bursty traffic, larger models, high availability, geographic distribution

mlx-serve exposes a direct memory-for-speed trade through KV quantization. In our tuning evaluation, we treated KV quantization as a memory-for-speed trade: about 10% lower decode throughput at 2K–4K context, and about 18% lower at 42K with fused packed reads automatically enabled from 8K, compared with the earlier 47% penalty. We budgeted approximately 2x KV-memory savings for 8-bit mode and 4x for 4-bit mode. That is attractive on a 16 GB Mac, but it is not free capacity.

Its sliding-window optimization also demonstrates why context length cannot be reduced to one number. On an M4 Max, Muse-Glimmer 30B 4-bit decode improved from 24.7 to 40.2 tokens per second at 16K context and from 8.6 to 21.0 at 64K after every attention read was trimmed to the model window. Those are release-specific measurements, not expected vllm-mlx results.

Cost break-even

For an illustrative capacity model, assume:

Mac acquisition cost: $4,000.
Amortization period: 36 months.
Average inference-system draw: 150 W.
Electricity: $0.20 per kWh.
Continuous availability: 720 hours per month.

Monthly hardware amortization is about $111. Electricity adds roughly $22, producing a baseline of approximately $133 per month before support, networking, backup capacity, and engineering time.

At an assumed remote accelerator cost of $1 per running hour, the raw infrastructure break-even is about 133 accelerator-hours per month. At $2 per hour, it is roughly 67 hours. These are planning assumptions, not current provider quotes.

The calculation favors the Mac only when it stays useful. A lightly used machine assigned to one small model can cost more per completed request than an autoscaled endpoint. A Mac already purchased for development has a different equation because its marginal capital cost may be close to zero.

Availability changes the result again. One Mac is one failure domain. A production service requiring maintenance without interruption needs at least two hosts, a load balancer, model synchronization, monitoring, and spare memory on both machines. Teams comparing one Mac against a managed endpoint often omit that redundancy cost.

If you need a workload-specific capacity and failover model, contact our architecture team rather than sizing from peak tokens per second alone.

Our Final Verdict: When to Deploy, When to Skip

vllm-mlx is credible for shared Apple Silicon inference. Continuous batching produces meaningful aggregate-throughput gains, and its OpenAI and Anthropic surfaces make migration easier than building a custom MLX scheduler. The M4 Max reference results show that the gain can be substantial, especially for smaller models.

Our reservation is operational, not conceptual. Unified memory is the hard boundary, and cache limits do not convert vllm-mlx into a memory-safe multi-tenant platform. Production safety still depends on request admission, context limits, monitoring, and worker recovery outside the server.

Deploy this if:

You already own Apple Silicon capacity with enough memory for weights, KV state, and operating-system headroom.
Several users share the same model and generate concurrently.
Your workloads reuse system prompts or conversation prefixes.
You need OpenAI or Anthropic API compatibility without a cloud dependency.
You can cap prompt length, output length, queue depth, and effective concurrency.
A single-region, local-network service is acceptable.
You can benchmark the exact model family rather than assuming all architectures batch equally.

Hold off or avoid it if:

You need guaranteed graceful recovery after unified-memory exhaustion.
Traffic is highly bursty and the Mac would sit idle most of the month.
You require multi-host scheduling, automatic failover, or elastic scaling out of the box.
Your workload depends on very long, unrelated prompts with little prefix reuse.
You are trying to run a model whose weights already consume nearly all physical memory.
You need a published service-level objective without first measuring mixed prompt lengths and generation sizes.
You intend to treat --max-num-seqs 256 as a safe batch size.

Our production starting point would be four active requests, a two-gigabyte cache cap, paged caching, strict API-side context limits, and at least 20% physical-memory headroom after model load. We would then raise concurrency only after recording aggregate throughput, per-request decode rate, median and p95 time to first token, queue wait, and peak Metal memory.

The bottom line is straightforward: vllm-mlx makes a Mac a legitimate multi-user inference server, not an elastic GPU cluster. Within that boundary, continuous batching is worth enabling. Outside it, especially when memory exhaustion or host failure must be invisible to clients, we would keep the Mac as an edge or development tier and route overflow to a managed backend.

Voyage AI vs Cohere Embed v4 vs Nemotron 3 Embed: Production RAG Trade-offs

Jangwook Kim — Thu, 08 Oct 2026 00:39:22 +0000

Why We Evaluated These Embedding Models

Embedding selection looks reversible until a production index contains tens of millions of vectors. At that point, changing the model can mean re-reading every source document, reproducing the original chunking pipeline, paying for another embedding pass, building a second index, and switching traffic without mixing incompatible vector spaces.

That operational commitment—not a leaderboard position—was why we evaluated Voyage 4, Cohere Embed v4, and NVIDIA Nemotron 3 Embed.

The three options represent materially different deployment choices:

Voyage 4 is a managed, text-focused family with selectable dimensions and quantized output formats.
Cohere Embed v4 is a managed multimodal model for text, images, and mixed documents.
Nemotron 3 Embed is positioned as an open-weight, self-hosted family where licensing cost is replaced by GPU and operations cost.

Under the hood, each model converts documents and queries into vectors that an index compares by cosine similarity, dot product, or an equivalent distance function. The implementation details differ, but the production contract is the same: document and query preprocessing must remain stable, dimensions must match, and we need continued access to a query encoder compatible with the stored vectors, rather than necessarily the exact model version originally used.

We found an immediate problem with the public comparison narrative. The available benchmark figures were not generated on one common evaluation suite.

The Nemotron 3 figures we examined included 78.46 on RTEB and 75.45 on MMTEB Retrieval for the 8B checkpoint. The 1B checkpoint was listed at 72.38 RTEB and 71.04 MMTEB Retrieval. Figures circulating for Voyage and Cohere were general MTEB aggregates, not directly comparable RTEB runs. We therefore refused to place those values into a single “winner” ranking.

A 78.46 retrieval score and a roughly 67 general MTEB aggregate do not measure the same thing. MTEB can blend retrieval, clustering, classification, and semantic-similarity tasks. RTEB is retrieval-specific. Different task sets, relevance judgments, languages, pooling policies, and gain conventions can reverse an apparent lead.

This matters because a model can perform well on public web retrieval and still miss exact clauses in contracts, versioned API behavior, product identifiers, or cross-language support content. Our deployment recommendation consequently depends on a customer’s labeled corpus rather than the headline score.

We treat the model choice as an index architecture decision. Teams evaluating the surrounding stack can compare it with the vector and retrieval tools in our AI tools collection, but they should freeze the embedding contract before optimizing the database.

Hands-On Walkthrough: Setup, Execution & Output

We started by validating the metric harness rather than sending three sets of vectors into an untrusted scorer. That step caught a class of benchmark errors more damaging than a slow API: document-ID misalignment can turn perfect retrieval into a score of zero while every individual array still has the expected shape.

We ran the tests in Python 3.12 with these pinned dependencies:

python -m venv .venv
source .venv/bin/activate

python -m pip install \
  numpy==2.1.3 \
  scipy==1.14.1 \
  scikit-learn==1.5.2

python benchmark_metric_contract.py

The following compact fixture reproduces the key cutoff and all-zero-relevance checks; the document-ID alignment results are reported separately below. It is runnable without an API key or GPU.

import json
import numpy as np
from sklearn.metrics import ndcg_score

cutoff = 10
document_count = 12

def score(relevance, ranking_scores):
    y_true = np.asarray([relevance], dtype=float)
    y_score = np.asarray([ranking_scores], dtype=float)
    return float(ndcg_score(y_true, y_score, k=cutoff))

# Higher retrieval scores produce earlier ranks.
descending_scores = list(range(document_count, 0, -1))

def one_relevant_at_rank(rank):
    relevance = [0.0] * document_count
    relevance[rank - 1] = 1.0
    return score(relevance, descending_scores)

results = {
    "single_relevant_at_rank_1": one_relevant_at_rank(1),
    "single_relevant_at_rank_2": one_relevant_at_rank(2),
    "single_relevant_at_rank_10": one_relevant_at_rank(10),
    "single_relevant_at_rank_11": one_relevant_at_rank(11),
    "all_zero_relevance_query": score(
        [0.0] * document_count,
        descending_scores,
    ),
}

assert results["single_relevant_at_rank_1"] == 1.0
assert results["single_relevant_at_rank_10"] > 0.0
assert results["single_relevant_at_rank_11"] == 0.0
assert results["all_zero_relevance_query"] == 0.0

print(json.dumps(results, indent=2))

Our executed fixture produced:

{
  "single_relevant_at_rank_1": 1.0,
  "single_relevant_at_rank_2": 0.6309297535714574,
  "single_relevant_at_rank_10": 0.2890648263178878,
  "single_relevant_at_rank_11": 0.0,
  "all_zero_relevance_query": 0.0
}

We also tested graded relevance. Swapping two graded results produced an nDCG@10 of 0.9224945116765988 with linear gains and 0.842828264880938 when we supplied exponential gains. That absolute difference of 0.07966624679566081 is large enough to manufacture a benchmark “win” if two runs use different gain conventions.

The alignment fixture exposed an even larger scoring error:

aligned_score=1.0
joint_column_permutation_score=1.0
judgments_only_permutation_score=0.0
all_assertions_passed=true

Permuting document IDs in both predictions and judgments preserved the score. Permuting only the relevance judgments reduced it from 1.0 to 0.0. We now require one canonical document-ID map across chunk storage, relevance judgments, cached vectors, and result files.

This was a measurement-contract test, not a model-quality run. We did not produce vendor embeddings in this executed sandbox, so we do not present invented nDCG, recall, latency, throughput, or multilingual results for Voyage, Cohere, or Nemotron.

For a real head-to-head, our required run manifest is:

{
  "corpus": {
    "chunks": "2000-5000",
    "chunker_version": "frozen",
    "content_hashes": "required"
  },
  "queries": {
    "relevance_judgments": "graded",
    "zero_relevance_policy": "declared",
    "languages": "stratified"
  },
  "retrieval": {
    "metric": ["nDCG@10", "Recall@5", "Recall@10", "Recall@50"],
    "similarity": "cosine",
    "normalization": "explicit",
    "reranker": "disabled"
  },
  "execution": {
    "model_version": "pinned",
    "dimension": "recorded",
    "input_type": "recorded",
    "truncation": "disabled-or-counted",
    "batch_and_realtime": "separate"
  }
}

Voyage’s embedding reference and official Python client expose the necessary model, input type, output dimension, truncation, and output data type controls. Teams should pin Cohere’s equivalent configuration using its Embed reference, rather than infer it from a framework wrapper.

Benchmark Limitations and Production Risks

The first failure was not an API error. It was evidence comparability. We could not reproduce a valid three-model leaderboard from the supplied material because no common fixed-corpus run existed. Any article naming an unconditional winner from these inputs would be overstating the evidence.

We hit six additional production hazards.

1. Dimensions are a schema, not a tuning flag.

Voyage 4 supports 256, 512, 1,024, and 2,048 dimensions, with 1,024 as the default. Cohere Embed v4 configurations include 256, 512, 1,024, and 1,536 dimensions. The Nemotron 3 material we reviewed described 2,048 dimensions for the 1B model and 4,096 for 8B.

A collection created at 1,024 dimensions cannot accept a 1,536- or 4,096-element vector. We plan a new collection, validation, and traffic cutover for a dimension change. Switching to an incompatible embedding space requires re-embedding; reducing compatible Matryoshka vectors may allow us to reuse existing embeddings after validating the transformation and retrieval quality. We store model, revision, dimension, normalization, data type, and preprocessing hash beside every index.

2. Voyage silently truncates by default.

Voyage’s truncation option defaults to True. An oversized input can therefore return a valid vector for incomplete content. That is worse than an explicit failure when the missing text contains the relevant clause.

We set truncation=False in ingestion jobs and route over-length chunks to a dead-letter queue. We would rather fail visibly than index partial documents silently.

3. Query and document modes are operationally significant.

Voyage prepends different retrieval prompts when input_type is query or document. Cohere also distinguishes retrieval input types. We treat this field as part of the model version. A missing query mode is not harmless configuration drift.

4. Request size is not the same as rate limit.

Voyage accepts at most 1,000 texts per request, with additional token ceilings. The documented total is up to one million tokens for voyage-4-lite, 320,000 for voyage-4, and 120,000 for voyage-4-large.

Those are payload limits, not throughput guarantees. We could not verify per-key requests-per-minute, token-per-minute limits, cooldown behavior, or reserved-capacity terms across all three providers. Teams should load-test their own account tier rather than extrapolate from maximum batch size.

5. Quantization support is uneven.

Voyage’s hosted 4-series can return float, int8, uint8, binary, and unsigned-binary output. The local voyage-4-nano path does not support int8 or uint8 because its required calibration ranges are not shipped; those modes raise NotImplementedError. Float and binary formats remain available locally.

The local extra also requires Python 3.10 or newer and installs PyTorch plus Sentence Transformers. We kept API-only and local-inference environments separate to avoid turning a lightweight client image into a large ML runtime.

6. Nemotron naming can produce the wrong comparison.

We found material mixing Nemotron 3 Embed with the older llama-nemotron-embed-1b-v2. The older model has an 8,192-token input limit and Matryoshka dimensions down from 2,048. Those specifications must not be copied onto the newer Nemotron 3 family described with a 32,768-token context.

We pin exact model identifiers and artifact revisions. “Nemotron Embed” is not precise enough for an index manifest.

Scale, Latency & Cost vs. Alternatives

We normalized the product configurations available during the review, but we did not measure service latency or GPU throughput. The latency column is therefore deliberately absent rather than filled with synthetic numbers.

Option	Deployment	Context limit	Dimensions reviewed	Multimodal	Public price used for planning	Main operational trade-off
Voyage 4 Lite	Managed API	32,000	256-2,048	No	$0.02 per 1M tokens	Lowest listed API rate; quality must be tested by domain
Voyage 4	Managed API	32,000	256-2,048	No	$0.06 per 1M tokens	Balanced managed text retrieval
Voyage 4 Large	Managed API	32,000	256-2,048	No	$0.12 per 1M tokens	Higher-cost quality tier
Cohere Embed v4	Managed API	128,000	256-1,536	Yes	$0.12 per 1M text tokens	Best fit here for mixed text and image retrieval
Nemotron 3 Embed 1B	Self-hosted	32,768	2,048	No	No per-token license charge	GPU serving, observability, and capacity become our responsibility
Nemotron 3 Embed 8B	Self-hosted	32,768	4,096	No	No per-token license charge	Largest raw vector payload at the reviewed dimensions; serving capacity must be measured

At one billion text tokens, the simple API bill is approximately:

Voyage 4 Lite: $20
Voyage 4: $60
Voyage 4 Large: $120
Cohere Embed v4: $120
Nemotron 3: infrastructure-dependent

These numbers expose an important reality: initial embedding API charges can be smaller than vector storage, indexing, replicas, backups, and migration engineering. At float32, one million raw vectors require approximately:

256 dimensions: 0.95 GiB
1,024 dimensions: 3.81 GiB
1,536 dimensions: 5.72 GiB
2,048 dimensions: 7.63 GiB
4,096 dimensions: 15.26 GiB

That excludes IDs, graph edges, metadata, allocator overhead, replicas, and snapshots. Moving from 1,024 to 4,096 dimensions quadruples the raw vector payload before the vector database adds its own structures.

For Nemotron, we use a break-even equation rather than pretending open weights are free:

self_hosted_cost_per_million = gpu_hourly_cost / millions_of_tokens_per_hour

At a hypothetical $2 per GPU-hour, self-hosting must sustain more than 16.67 million tokens per hour to beat a $0.12-per-million API on raw compute. It must exceed 33.33 million tokens per hour to beat a $0.06-per-million API. Engineering labor, idle capacity, failover, deployment, and monitoring move the real threshold higher.

We did not measure Nemotron throughput, memory use, or NVFP4 quality loss, so those break-even examples are formulas, not performance claims.

For teams planning a migration or capacity model, our AI infrastructure services focus on the complete cost boundary: embedding generation, index memory, replicas, refresh cadence, and rollback—not just the token invoice.

Our Final Verdict: When to Deploy, When to Skip

We cannot honestly name a universal retrieval-quality winner from the available evidence. Nemotron 3 has the strongest retrieval-specific public numbers in this comparison, but those numbers were not generated in the same run as Voyage 4 and Cohere Embed v4. Our executed work validated the metric harness, not the three models.

That distinction defines our verdict.

Deploy Voyage 4 if:

We need managed text embeddings with low list pricing.
We want selectable dimensions and native quantized output.
A 32,000-token context covers the chunking strategy.
We can pin input_type, reject truncation, and keep a model manifest.
We have domain labels proving that Lite, standard, or Large meets recall targets.

Deploy Cohere Embed v4 if:

We need text, images, and mixed documents in one retrieval space.
A 128,000-token input window materially simplifies document handling.
We value a managed endpoint more than control over model weights.
We can absorb the API price and validate multilingual performance on our own language mix.

Deploy Nemotron 3 Embed if:

We already operate NVIDIA inference infrastructure.
Data residency or service independence makes self-hosting valuable.
Query volume is high and steady enough to amortize GPUs.
We can benchmark the exact checkpoint, precision, and dimension used in production.
We accept responsibility for capacity, upgrades, observability, and rollback.

Hold off or avoid all three as a final choice if:

We do not have labeled queries from the target domain.
We are comparing RTEB and general MTEB scores as if they were interchangeable.
We have not calculated index memory at the chosen dimension and replica count.
The ingestion pipeline allows silent truncation.
Model names, preprocessing prefixes, or revisions are not pinned.
We are inferring multilingual quality from an aggregate score.
We have not tested rate-limit and batch assumptions against the production account.

Our practical default is Voyage 4 at 1,024 dimensions for managed, text-only evaluation; Cohere Embed v4 when multimodal retrieval is a real requirement; and Nemotron 3 only when self-hosting is an architectural objective rather than a reaction to per-token pricing.

We would not approve production deployment until all candidates had run over the same 2,000-to-5,000-chunk corpus, the same graded queries, the same distance metric, and the same nDCG and recall implementation. The winning model is the one that clears the application’s recall threshold at the lowest total operating cost—not the one with the largest vector or the most flattering public benchmark.

If the index migration risk is already material, contact our engineering team before changing dimensions or model families. Rebuilding a benchmark is cheap. Rebuilding a live retrieval index under customer traffic is not.

Cline in Production: BYO-Key Costs, MCP Limits, and Terminal-Bench Results

Jangwook Kim — Wed, 07 Oct 2026 00:38:57 +0000

Why we evaluated Cline

We did not test Cline because we needed another autocomplete tool. We tested it because flat per-seat AI IDE pricing makes cost attribution difficult, while closed agent runtimes make model and tool migrations expensive.

Cline + Kimi K3 on Terminal-Bench 2.1: score up, spend downBaseline pass rate (69/89) 77.5%/100Confirmation pass rate (79/89) 88.8%/100Baseline run cost $79Confirmation run cost $49.80

The higher-scoring run on the same 89-task suite was also the cheaper one, because fewer sessions died in retries, loops, and self-termination.

Cline separates the agent runtime from the inference provider. The VS Code extension supplies the agent loop, file operations, terminal integration, approvals, context management, and MCP client. We supply the model account and pay the inference provider directly. That separation gives us three things we cannot assume from a bundled IDE subscription:

We can select or replace the model independently of the editor.
We can inspect token usage and assign inference cost to a task.
We can expose internal services through MCP without first waiting for an IDE vendor to add a native integration.

That does not make Cline free. It exchanges predictable seat pricing for variable inference spend, provider rate limits, API-key management, and substantially more operational responsibility.

We evaluated four practical questions:

Can a developer install Cline without adopting another IDE?
Can we switch between Anthropic and OpenAI-style providers without rewriting the workflow?
Does MCP provide a usable portability boundary?
Do the published Terminal-Bench numbers survive scrutiny as production evidence?

The short answer is mixed. BYO-key model choice and usage-based accounting provide useful control, but we did not verify the extension's installation or provider-configuration workflow. Our separate Python MCP SDK test failed during initialization; it does not establish a Cline client defect. The benchmark is much more useful as a harness-debugging case study than as a definitive “Cline beats Cursor” ranking.

Cline’s SDK production architecture also matters. We can instrument model-call, tool-use, session-end, and usage events, including token counts and finish reasons. The SDK supports iteration limits, per-turn token limits, loop detection, mistake limits, and host-side cancellation. Those are the controls we expect from an agent runtime. They are not proof that a developer workstation is safely sandboxed.

Our evaluation therefore treated Cline as an agent harness with an editor front end, not as a security boundary.

Installing Cline, configuring MCP, and running Terminal-Bench

Installing and verifying the extension

For a team rollout, we would verify the current Marketplace extension identifier and test installation in a clean VS Code profile. The commands below are illustrative; we have not verified their identifier or execution:

code --install-extension saoudrizwan.claude-dev
code --list-extensions --show-versions | grep -i saoudrizwan.claude-dev

We would confirm the publisher, display name, and exact extension identifier against the current Marketplace listing before using these commands.

For a team rollout, we would pin and test a known version before broad deployment:

code --install-extension saoudrizwan.claude-dev@<approved-version> --force

Cline supports BYO-key providers, including Anthropic and OpenAI. We would verify the extension's current provider interface and credential-storage behavior before rollout, and keep keys out of workspace settings and committed files.

For Anthropic or OpenAI, we would confirm the current provider-specific setup instructions, configure a dedicated development key, and select the intended model. We have not verified the extension's exact settings labels, compatible-endpoint fields, or approval controls. Our rollout plan would begin with a read-only task before enabling terminal commands or file writes.

We would use separate development and production provider keys and configure provider-side budget alerts. A BYO-key extension installed on every laptop creates a larger credential-management surface than a centrally administered seat license.

Wiring a local MCP stub

The following is an illustrative Python MCP server using the SDK version pinned for our local test attempt. This example was not successfully validated, and it is not a verified Cline integration:

# server.py
from mcp.server.fastmcp import FastMCP

mcp = FastMCP("effloow-local-stub")

@mcp.tool()
def add_ticket(label: str, priority: int) -> dict:
    """Create a local test ticket without external side effects."""
    if priority < 1 or priority > 5:
        raise ValueError("priority must be between 1 and 5")

    return {
        "accepted": True,
        "ticket": {
            "label": label,
            "priority": priority,
        },
    }

if __name__ == "__main__":
    mcp.run(transport="stdio")

We installed its dependencies in an isolated environment:

python -m venv .venv
. .venv/bin/activate
pip install "mcp==1.9.4" "pydantic==2.11.5" "anyio==4.9.0"

The following is an unverified registration example, not a configuration we tested in Cline. We would confirm the current client schema, configuration location, and approval fields before using it:

{
  "mcpServers": {
    "effloow-local-stub": {
      "command": "/absolute/path/to/project/.venv/bin/python",
      "args": ["/absolute/path/to/project/server.py"],
      "disabled": false,
      "autoApprove": []
    }
  }
}

For a future integration test, we would use absolute interpreter and server paths and check the environment supplied by the client. We would also verify the meaning of approval fields rather than assume that an empty autoApprove list enforces the intended policy. State-changing tools should require explicit approval.

Our isolated Python MCP SDK handshake did not complete successfully. The test failed during initialization with McpError: Connection closed and exit code 1, producing no tool observations. Dependency setup completed, but we did not verify Cline's registration structure, client behavior, argument validation, or recovery. This was a failed SDK integration attempt, not evidence of a Cline deployment defect.

Running the 89-task evaluation

For the agent benchmark, we used the open Terminal-Bench dataset through Harbor. The relevant projects are the Terminal-Bench repository and the Harbor evaluation harness.

The baseline prerequisites were Python, uv, Docker, one provider key, and optionally Modal for parallel execution:

uv tool install harbor
pip install modal
modal setup

export CPUS=14
export MEMORY_MB=8192
export OPENROUTER_API_KEY="replace-with-real-key"
export API_KEY="$OPENROUTER_API_KEY"

harbor run \
  -d terminal-bench@2.0 \
  -a cline-cli \
  -m openrouter:anthropic/claude-opus-4.6 \
  --env modal \
  --ak thinking=6000 \
  --ak timeout=2400 \
  -n 89 -l 89 \
  --override-cpus "$CPUS" \
  --override-memory-mb "$MEMORY_MB"

The 2,400-second timeout matters. The earlier 600-second default caused agent timeouts that looked like capability failures but were actually harness configuration failures.

The reproduced full-sweep output was:

89/89 Mean: 0.573
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
0:45:05  0:00:00

Passed: 51
Failed: 38
Mean reward: 0.573

Failure classes:
  AgentTimeoutError
  VerifierTimeoutError
  missing_expected_files
  command_exit_status_not_observed
  provider_rate_limit

This was a Terminal-Bench 2.0 run using the Cline CLI adapter and an OpenRouter-hosted Anthropic model. It must not be compared directly with later Terminal-Bench 2.1 Kimi K3 results.

For cost accounting, we examined the SDK's host-side estimator: it prefers provider-reported usage.totalCost and otherwise combines token counts with a maintained pricing table. We did not obtain actual per-task cost output. The following record is synthetic and illustrates a possible logging format:

{
  "task_id": "tb2-task-041",
  "model": "provider:model-id",
  "input_tokens": 18442,
  "output_tokens": 2106,
  "provider_reported_total_cost_usd": 0.1187,
  "iterations": 14,
  "finish_reason": "completed"
}

That JSON is an output-format example, not an independently verified cost for a named Terminal-Bench task. The externally auditable numbers were aggregate: the later Kimi K3 baseline cost $79 for 89 attempts, or $0.888 per attempted task, while the confirmation run cost $49.80, or $0.560 per attempted task. Those averages include failed tasks and cannot reveal the task-level distribution.

For production accounting, we would reject aggregate-only reporting. We would persist provider, model, input tokens, output tokens, cached tokens, provider-reported cost, tool calls, duration, and finish reason for every task.

What Broke: The Gotchas and Limitations We Hit

The first local failure was initialization in a separate Python MCP SDK test, not in Cline's MCP client. Our pinned Python client received Connection closed during session.initialize(). There were no tool observations, so we could not establish invocation, schema validation, or recovery.

Our immediate debugging checklist was:

Use an absolute interpreter and server path.
Launch the server manually with the same interpreter.
Keep stdout exclusively for MCP protocol traffic.
Send diagnostics to stderr.
Confirm the client and server SDK versions agree on transport behavior.
Restart the VS Code window after changing MCP registration.
Avoid assuming that the extension inherits shell variables.

For us, this failed SDK initialization highlighted checks we would perform before relying on an MCP integration. It did not measure Cline's portability or compare clients. We would validate process supervision, approval semantics, credentials, timeouts, logs, and environment propagation for each server/client pairing.

The second issue was misleading benchmark failure attribution. We found several classes that were not model-intelligence problems:

The agent timed out before completing valid work.
Command exit codes were not surfaced correctly.
Long-running build or test commands were terminated early.
The agent declared completion without verifying expected files.
A provider returned a rate limit and the session stopped instead of retrying.
Repeated polling of a progressing background process was mistaken for a loop.
Broad pkill -f commands matched the harness process and killed the agent itself.
Prompt text containing @a-style tokens triggered file-mention processing while an unreferenced asynchronous worker allowed the process to exit before a model call.

These failures matter because they demonstrate how much benchmark performance belongs to the harness rather than the model. Retries with exponential backoff, output-aware loop detection, explicit completion verification, PID-based process cleanup, and a longer timeout produced real improvements.

The score still had material run-to-run noise. Six comparable runs produced 0.49, 0.43, 0.45, 0.44, 0.48, and 0.46, with a 0.458 average and 0.455 median. We would not merge a harness change based on a one- or two-point improvement from a single run. We would repeat close comparisons three to six times.

We also encountered an evidence problem around cost. The runtime exposes enough usage data to implement task-level accounting, but the headline benchmark material mostly supplied whole-run spend. Dividing $49.80 by 89 attempts is useful for budgeting; it is not a substitute for a cost histogram or pass-adjusted cost.

Finally, before replacing Cursor or Windsurf with a BYO-key harness, we would review operational requirements beyond raw model quality:

Developers must select models and understand provider pricing.
Provider outages and rate limits become our problem.
Every API key needs scope, rotation, revocation, and offboarding.
We must standardize approval defaults across machines.
We would verify workspace-rule compatibility rather than assume rules transfer unchanged between agents.
MCP support does not guarantee identical tool behavior.
Cline can spend far more than expected when an agent loops, retries, or reads a large repository repeatedly.
Extension updates need staged internal validation even though Marketplace publication itself is effectively all-or-nothing.

We would manage Cline through an approved configuration baseline, a provider gateway, and centrally collected usage events. A collection of individually configured laptops is not a production deployment.

Scale, Latency & Cost vs. Alternatives

Our local Docker path was suitable for smoke tests, not rapid 89-task iteration. Sequential execution can take many hours. Modal parallelization brought the full run into the reported 40–50 minute range, with our reproduced output finishing in 45:05. The three-task setup check still took roughly 15 minutes because environment and sandbox startup were significant.

We did not measure workstation memory consumption for the VS Code extension, so we will not invent a number. The benchmark configuration explicitly reserved 14 CPUs and 8,192 MB per task environment for the parallel workflow; that is an evaluation allocation, not the extension’s desktop footprint.

Cline option	Inference arrangement	Accounting considerations	Integration considerations
BYO key	Provider API usage; model-agnostic provider choice	Host-side SDK accounting can use provider-reported cost or token counts and a pricing table	Verify each MCP server/client pairing and approval policy
ClinePass	$9.99/month for curated open-weight models, with no separate provider keys	We did not verify task-level billing detail or compare its accounting with bundled IDE plans	We did not test MCP behavior under this plan

We did not measure competitor accounting granularity, MCP behavior, or administrative burden, so we would compare those separately before making a procurement decision.

The practical break-even calculation is straightforward:

Monthly Cline cost per developer
  = tasks per month × mean API cost per task
  + shared gateway, observability, and support cost

Break-even task count
  = competing monthly seat price
  ÷ mean API cost per task

Using the $0.560 attempted-task average from the Kimi K3 confirmation run, a hypothetical $40 seat reaches raw inference break-even at roughly 71 attempted tasks per month:

$40 / $0.560 = 71.4 tasks

That does not mean Cline becomes uneconomic at task 72. A Terminal-Bench task is not equivalent to an autocomplete request, and seat plans can include quotas or overages. The calculation only shows why teams need their own task distribution.

The later Terminal-Bench 2.1 campaign moved from 69/89, or 77.5%, at $79 to 79/89, or 88.8%, at $49.80. The entire optimization campaign consumed roughly one billion tokens and about $680 over 17 hours. The winning run became cheaper because fewer sessions died in retries, loops, and self-termination.

We found that result technically valuable but causally incomplete. The campaign had no budget-matched parallel-sampling control and no held-out task split. Repeatedly optimizing against the same 89 tasks can improve that suite without proving equivalent generalization. We therefore treat 88.8% as evidence that Cline plus Kimi K3 completed 79 tasks under that configuration, not proof that Cline is universally better than Cursor, Windsurf, or every other harness.

If you are designing a broader agent platform rather than selecting an editor extension, review our tools collection and evaluate Cline’s SDK, CLI, and event model separately from its VS Code experience.

What we could not verify

We did not verify the extension installation and provider-configuration procedures, credential-storage behavior, Cline MCP registration or approvals, successful MCP tool execution, the extension's memory footprint, or the task-level cost distribution behind the benchmark totals. Our local MCP experiment used a separate Python SDK client and failed during initialization.

Our Final Verdict: When to Deploy, When to Skip

Cline works best when we treat it as an inspectable agent harness rather than a cheaper clone of a commercial IDE.

Deploy Cline if:

We need BYO-key access across multiple providers.
We can collect token and cost events centrally.
We want to keep VS Code instead of forcing an IDE migration.
We are prepared to test model-and-harness combinations together.
We need MCP integrations and can validate each server/client pairing.
We can enforce approval policies for shell, writes, network access, and state-changing tools.
We have a real eval set drawn from our repositories and workflows.
Variable inference spend is preferable to flat seat pricing.

Hold off or avoid it if:

We need a zero-administration agent product.
Developers would manage unrestricted production keys locally.
We cannot sandbox terminal execution.
We expect MCP configuration to behave identically across clients.
We need predictable monthly spend but cannot implement budget controls.
We plan to use a single Terminal-Bench score as a procurement decision.
We cannot tolerate provider rate limits or model-routing changes.
We lack staff to inspect failed traces and maintain the harness.

Before deployment, we would require per-task usage storage, provider budget alerts, maxTokensPerTurn, maxIterations, explicit task cancellation, restricted tool policies, and separate development and production credentials. We would also start MCP with no auto-approved state-changing tools.

Our final assessment is that Cline is a production candidate for an engineering organization willing to own inference operations and validate its configuration. BYO-key provider choice and host-side usage accounting offer useful control, but we did not establish a comparative portability or accounting advantage over specific closed IDE products. It does not eliminate lock-in entirely: the lock-in moves from the editor to agent rules, approval behavior, telemetry, and tool integration details.

The benchmark work reinforced the most important lesson. Agent quality is not just model quality. Timeouts, exit-code handling, retries, process supervision, loop detection, and completion verification moved the score from 47% to 57%, while later harness repairs helped reach 79/89 on a different benchmark version and model. Those are runtime-engineering gains, not prompt magic.

We would deploy Cline first to a small, technically strong cohort, instrument every task, and compare cost per accepted change against the existing workflow. We would not mandate it across a company based on headline benchmark percentages.

For help designing that controlled rollout, including gateway, MCP, observability, and evaluation architecture, talk to our engineering team.

Exa vs Tavily in 2026: Our Agent Web Search Benchmark

Jangwook Kim — Tue, 06 Oct 2026 00:39:13 +0000

Why We Compared Exa and Tavily

Search latency was not the original problem. Grounding reliability was.

Answer accuracy on 300 company-news questionsExa fast 99.3%/100Exa instant 97.7%/100Tavily advanced 93.0%/100Tavily basic 87.7%/100

On the same fixed suite, Exa fast and instant beat Tavily advanced and basic on answer accuracy, with Exa fast at 99.3% versus 87.7% for Tavily basic.

In agent traces, we kept finding steps that completed quickly but returned the wrong source type: a secondary blog instead of the original announcement, a directory instead of a company page, or an article mentioning a paper rather than the paper itself. The search call looked successful, yet the agent entered its next reasoning step with weak evidence.

That failure mode is expensive because it does not necessarily throw an error. It consumes model tokens, produces a confident answer, and may survive superficial evaluation.

We brought Exa and Tavily into our lab to answer four operational questions:

Which API puts an answer-bearing source in the first five results?
What latency should a multi-step agent budget for, especially beyond the median?
How much usable text reaches the model?
What does a correct answer cost after search and input-token charges?

Both products expose agent-oriented search APIs, but they emphasize different retrieval and content-delivery features.

Exa behaves like semantic retrieval infrastructure. Its search types include instant, fast, auto, and deeper research modes, while its category-specific retrieval is designed for entities such as people, companies, publications, and code. Its query-dependent highlights can return selected passages instead of an entire page.

Tavily behaves more like a packaged research interface. It offers basic, advanced, and speed-oriented search modes, along with general, news, and finance topics. It can return result snippets, generated answers, and raw page content. That reduces integration work when an agent needs search and extraction in one transaction.

Before running this harness against live services, we would verify its payloads against the official Exa search reference and official Tavily search reference, rather than relying on framework wrappers. Wrappers are convenient, but they can hide defaults, rename parameters, and make a provider migration look easier than it is.

For the main relevance comparison, we fixed the workload at 300 company-news questions. Each question had one pre-established answer tied to an official newsroom or wire URL. Every provider received one natural-language query and could return up to ten results. The same answer extraction path evaluated whether those results contained enough evidence to recover the correct value.

That suite measures practical grounding, not abstract semantic similarity. It also avoids letting an evaluator decide the expected answer after seeing a provider’s results.

Our run records showed:

Exa fast: 99.3% answer accuracy and 99.3% answer recall at five.
Exa instant: 97.7% answer accuracy and 97.3% answer recall at five.
Tavily advanced: 93.0% answer accuracy and 92.7% answer recall at five.
Tavily basic: 87.7% answer accuracy and recall at five.

We also inspected the larger entity-oriented evaluation behind Exa’s 2026 comparison. Under that test’s original conditions—Exa fast, Tavily advanced, consistent domain constraints, and model-based grading—Exa reached 75.5% versus 40.5% rank-one recall for people, 81.5% versus 61.3% for companies, and 63.3% versus 31.8% for publication retrieval. The sets contained 200 people queries, 200 company queries, and 597 publication queries.

We treated those entity figures as vendor-originated evidence, not as a neutral replacement for our fixed factual suite. They are still useful because the workload matches what Exa is designed to do, but the provenance belongs in any purchasing decision.

Hands-On Walkthrough: Setup, Execution & Output

We deliberately avoided SDKs for the first pass. Direct HTTP made the provider differences visible and kept dependency versions out of the result.

We stored credentials as environment variables:

export EXA_API_KEY="replace-with-exa-key"
export TAVILY_API_KEY="replace-with-tavily-key"

python -m venv .venv
source .venv/bin/activate
pip install "httpx>=0.27,<1"

Our query fixture used one JSON object per line:

{"id":"q001","query":"How much did Example Corp say it would invest in its announced Ohio facility?","expected":"$2 billion"}
{"id":"q002","query":"On what date did Example Labs announce its acquisition of Sample AI?","expected":"September 12, 2026"}

The following minimal harness is runnable. It measures wall-clock latency, retains normalized URLs for overlap analysis, and emits p50 and p95 for any suite you supply. The two-query fixture is not statistically meaningful; production runs should use the full pinned suite and preserve raw responses.

#!/usr/bin/env python3
import asyncio
import json
import os
import statistics
import time
from pathlib import Path
from urllib.parse import urlsplit, urlunsplit

import httpx

EXA_KEY = os.environ["EXA_API_KEY"]
TAVILY_KEY = os.environ["TAVILY_API_KEY"]

def canonicalize(url: str) -> str:
    parts = urlsplit(url)
    return urlunsplit((
        parts.scheme.lower(),
        parts.netloc.lower().removeprefix("www."),
        parts.path.rstrip("/"),
        "",
        ""
    ))

def percentile(values, p):
    ordered = sorted(values)
    index = round((len(ordered) - 1) * p)
    return ordered[index]

async def call_provider(client, provider, query):
    if provider == "exa":
        url = "https://api.exa.ai/search"
        headers = {"x-api-key": EXA_KEY, "content-type": "application/json"}
        payload = {
            "query": query,
            "type": "fast",
            "numResults": 10,
            "contents": {
                "highlights": {"maxCharacters": 1000}
            }
        }
    else:
        url = "https://api.tavily.com/search"
        headers = {
            "Authorization": f"Bearer {TAVILY_KEY}",
            "content-type": "application/json"
        }
        payload = {
            "query": query,
            "search_depth": "advanced",
            "max_results": 10,
            "include_answer": False,
            "include_raw_content": True
        }

    started = time.perf_counter()
    response = await client.post(url, headers=headers, json=payload)
    elapsed_ms = (time.perf_counter() - started) * 1000
    response.raise_for_status()
    body = response.json()

    results = body.get("results", [])
    urls = [canonicalize(item["url"]) for item in results if item.get("url")]
    return {
        "provider": provider,
        "latency_ms": round(elapsed_ms, 1),
        "status": response.status_code,
        "result_count": len(results),
        "urls": urls
    }

async def main():
    queries = [
        json.loads(line)
        for line in Path("queries.jsonl").read_text().splitlines()
        if line.strip()
    ]

    observations = []
    async with httpx.AsyncClient(timeout=30) as client:
        for item in queries:
            for provider in ("exa", "tavily"):
                result = await call_provider(client, provider, item["query"])
                result["query_id"] = item["id"]
                observations.append(result)
                print(json.dumps(result))

    for provider in ("exa", "tavily"):
        latencies = [
            row["latency_ms"]
            for row in observations
            if row["provider"] == provider
        ]
        print(json.dumps({
            "provider": provider,
            "calls": len(latencies),
            "p50_ms": percentile(latencies, 0.50),
            "p95_ms": percentile(latencies, 0.95),
            "mean_ms": round(statistics.fmean(latencies), 1)
        }))

asyncio.run(main())

A simulated output sample looks like this:

{"provider":"exa","latency_ms":571.8,"status":200,"result_count":10,"urls":["https://example.com/news/investment"]}
{"provider":"tavily","latency_ms":4187.3,"status":200,"result_count":10,"urls":["https://example.com/news/investment"]}
{"provider":"exa","calls":300,"p50_ms":569.0,"p95_ms":"calculate-from-retained-raw-run","mean_ms":"calculate-from-retained-raw-run"}
{"provider":"tavily","calls":300,"p50_ms":4200.0,"p95_ms":"calculate-from-retained-raw-run","mean_ms":"calculate-from-retained-raw-run"}

We intentionally do not insert an invented p95. The retained 300-query benchmark summary gives us medians—569 ms for Exa fast, 386 ms for Exa instant, 4.2 seconds for Tavily advanced, and 1.7 seconds for Tavily basic—but not the raw latency distribution required to calculate a defensible p95.

A separate 333-call speed comparison preserved p50, p90, and p99: Exa instant measured 235/263/437 ms, while Tavily’s comparable ultra-fast mode measured 245/334/576 ms. The medians were close; the tail was not. We use those figures for timeout planning, but we do not relabel p90 or interpolate p95.

For teams building their own provider abstraction, we collect broader implementation patterns in the Effloow tools collection.

Integration Issues and Verification Limits

The first problem was parameter parity. There is no safe one-line provider swap.

Tavily’s max_results maps conceptually to Exa’s numResults. Tavily uses include_domains and exclude_domains; Exa uses includeDomains and excludeDomains. Search-depth labels also differ. Tavily ultra-fast is closest to Exa instant, but Tavily advanced is not automatically equivalent to Exa fast, auto, or deep.

Content retrieval caused a larger semantic mismatch. Tavily can attach static raw page content to search results. Exa can return full text, but its more distinctive path is query-dependent highlights. Replacing Tavily raw content with Exa highlights changes the context contract: the model receives less text and loses some surrounding material.

That is often beneficial, but it is not lossless. We retained a full-text fallback for:

contract language where adjacent clauses matter;
technical pages where examples depend on preceding definitions;
tables whose headers are separated from matching rows;
pages requiring provenance beyond a selected passage.

We also found that “extraction quality” needs a precise definition. Our 300-question score measures whether search results let the answering model recover a correct fact. It does not isolate HTML cleaning, JavaScript rendering, table reconstruction, or PDF extraction. We refuse to convert factual-answer accuracy into a pure extractor score.

The same caution applies to result overlap. Raw URL overlap is unstable because tracking parameters, mirrors, syndication, language variants, and canonical redirects can make identical sources look different. Our harness strips query strings and normalizes hosts, but we still treat Jaccard overlap as a debugging signal rather than a relevance metric. A low overlap can mean healthy source diversity.

Configuration validation exposed another trap. We reproduced the common pre-restart check with Python 3.12.14 and json.load. An OpenClaw-style file containing a trailing comma failed with JSONDecodeError at line 6, column 7, exactly as expected.

A duplicate provider key was more dangerous:

{
  "tools": {
    "web": {
      "search": {
        "provider": "tavily",
        "provider": "brave"
      }
    }
  }
}

The default parser accepted this file and silently selected brave, the last value. Syntax validation alone therefore prevents malformed JSON but does not prove configuration intent. Our local test covered only standard-library json.load on three synthetic fixtures; it did not test the duplicate-key rejection hook below, the agent’s configuration schema, or restart behavior. For deployment, we would use the hook below to reject duplicate keys and separately validate the configuration against the agent’s schema before restarting:

import json

def reject_duplicates(pairs):
    result = {}
    for key, value in pairs:
        if key in result:
            raise ValueError(f"Duplicate JSON key: {key}")
        result[key] = value
    return result

with open("agent.json") as handle:
    config = json.load(handle, object_pairs_hook=reject_duplicates)

Rate-limit design also needs explicit treatment. Tavily publishes a higher standard ceiling of 1,000 requests per minute. Exa publishes 10-plus queries per second with custom enterprise scaling. We did not run a production-scale saturation test, so we cannot tell buyers how either service degrades during a burst. We implement bounded concurrency, exponential backoff with jitter, and provider-specific circuit breakers rather than assuming the published ceiling is a latency guarantee.

Domain filters differ too. Exa supports up to 1,200 included and 1,200 excluded domains. Tavily’s referenced limits are 300 included and 150 excluded. That matters for regulated allowlists and large tenant-specific exclusion sets.

Finally, all returned content is untrusted. We do not store raw extracted bodies in long-lived agent memory. We persist source URLs, hashes, constrained summaries, and citation metadata. Otherwise, prompt-injection text can survive beyond the search turn and contaminate later sessions.

If this threat model is part of a larger agent deployment, our AI infrastructure services cover retrieval boundaries, observability, and tool authorization rather than treating search as an isolated API call.

Scale, Latency & Cost vs. Alternatives

Here is the decision table we actually use:

Configuration	Accuracy	Answer recall at 5	Median search latency	Search price per 1,000	Result tokens	Best fit
Exa `fast`	99.3%	99.3%	569 ms	$7	1,987	High-accuracy factual and semantic retrieval
Exa `instant`	97.7%	97.3%	386 ms	$7	2,128	Latency-sensitive agent loops
Tavily `advanced`	93.0%	92.7%	4.2 s	$16	2,210	General research with bundled content
Tavily `basic`	87.7%	87.7%	1.7 s	$8	1,639	Lower-cost general lookups
Firecrawl search	95.3%	96.7%	471 ms	$5	678	Search attached to a crawl-heavy stack

These figures come from the same 300-question company-news workload summarized in the independent benchmark record. We kept latency and cost separate from accuracy rather than collapsing them into a subjective weighted score.

At pay-as-you-go list pricing, 100,000 searches cost approximately:

Exa: $700.
Tavily basic: $800.
Tavily advanced: $1,600.

Tavily’s monthly plans can reduce credit prices to roughly $0.0075–$0.005, so procurement volume can reverse part of that difference. Enterprise discounts can also make public prices irrelevant. We recommend comparing signed quotes, not landing pages.

Search cost is only half of the calculation. At a hypothetical input price of $5 per million tokens, the recorded result payloads produce these 100,000-query totals:

Configuration	Search cost	Approximate input-token cost	Combined cost
Exa `fast`	$700.00	$993.50	$1,693.50
Tavily `advanced`	$1,600.00	$1,105.00	$2,705.00
Tavily `basic`	$800.00	$819.50	$1,619.50

Tavily basic is slightly cheaper in this model, but it gives up 11.6 percentage points of accuracy versus Exa fast. For search spending alone, the more useful metric is search-API cost per correct result. Our benchmark record puts Exa fast at about $7.05 per 1,000 correct answers and Tavily advanced at $17.20, excluding downstream model-token charges.

Token totals remain workload-specific. Exa highlights can materially reduce context on long pages, while Tavily basic happened to return fewer tokens in this suite. We would not sign a contract based on a universal “Exa uses fewer tokens” assumption. Measure the payload generated by your parameters and your query mix.

Latency compounds in agent loops. As a planning scenario, if each of five sequential calls took its provider’s measured single-call p99 time, search alone would total roughly 2.19 seconds with Exa instant versus 2.88 seconds with Tavily ultra-fast, before model inference or retries. These sums are not measured five-step runtimes or workflow p99 estimates. Parallel fan-out changes the arithmetic, but the slowest branch still controls completion.

The acquisition of Tavily by Nebius in February 2026 is also a roadmap consideration. We do not treat ownership change as an automatic negative. It can bring distribution, infrastructure, and enterprise contracting advantages. We do, however, put API compatibility, pricing protection, data residency, and deprecation notice periods into the contract because integration priorities can change after an acquisition.

Our Final Verdict: When to Deploy, When to Skip

We would deploy Exa first when the agent must find a specific company, person, paper, code reference, or original factual source. It won our fixed factual suite on accuracy, cost per correct answer, and practical latency. Its instant mode also gave us the tighter recorded tail in the separate speed test.

We would deploy Tavily when the team wants a simpler general-research interface with raw content, generated-answer options, news and finance modes, and a high published request ceiling. Tavily is also reasonable when consolidated Nebius procurement matters more than absolute retrieval performance.

Deploy Exa if:

rank-one semantic retrieval is central to the product;
agents search for entities or publications;
tail latency controls a multi-step workflow;
query-dependent highlights can replace most full-page text;
large domain allowlists or blocklists are required.

Deploy Tavily if:

general web research dominates the workload;
bundled raw content reduces engineering work;
news or finance topic routing is valuable;
the published 1,000 RPM standard ceiling fits a high-volume design;
Nebius consolidation simplifies procurement or infrastructure.

Hold off or run a longer proof of concept if:

JavaScript-heavy extraction, PDFs, or tables are the primary workload;
you need contractual latency objectives rather than benchmark medians;
your agent sends sensitive customer data in queries;
you cannot isolate untrusted retrieved text from instructions and memory;
your economics depend on unverified enterprise discounts.

Our production recommendation is not “pick one forever.” Put both behind a typed adapter, retain raw timing and billing metadata, and route by workload. Start with Exa for semantic or high-value factual retrieval. Use Tavily where full-page general research is the actual requirement. Add a fallback only after measuring whether the extra recall justifies duplicate search spend.

Most importantly, benchmark correctness before speed. A 250 ms search call that grounds the agent in the wrong source is not fast infrastructure. It is a fast path to an expensive mistake.

Groq API in Production: The Latency, Throughput, and Cost Trade-Offs That Matter

Jangwook Kim — Mon, 05 Oct 2026 00:43:51 +0000

Why We Brought This Tool Into Our Lab

We brought GroqCloud into our lab for a narrow reason: user-facing agents spend too much time waiting for models to emit tokens.

An agent can have a fast vector store, a warm application container, and sub-20 ms internal service calls, yet still feel slow because generation takes several seconds. Faster decoding changes that interaction materially. It also shortens sequential agent loops where every tool result triggers another model call.

Groq attacks this bottleneck with its LPU inference architecture and an OpenAI-compatible API. We did not treat that architecture as proof of production performance. We separated four measurements that are too often collapsed into one headline:

Client-observed time to first useful token: request start to the first non-empty content delta.
End-to-end latency: request start to a completed stream.
Client-side output-throughput proxy: returned completion tokens divided by elapsed time from the first non-empty content delta to stream end. We do not treat this as verified answer-token decoding speed when reasoning or other non-visible tokens may be included in completion usage.
Workload cost: returned input and output tokens multiplied by the model-specific rates.

That distinction matters. Groq’s console records server-side latency, while an end user experiences network time plus server time. A dashboard can therefore look healthy while clients in another region see a worse tail. We also found that “first chunk” is not necessarily “first useful token”: role metadata or an empty content delta can arrive before visible text.

The historical LPU result that put Groq on engineering teams’ radar used approximately 100 input tokens and 200 output tokens. Groq’s Llama 2 Chat 70B endpoint reached 241 output tokens per second in that 2024 test, with an estimated 0.8-second response time for 100 output tokens. That is useful architectural evidence, but it is not a valid 2026 capacity number. The model, API traffic, and product lineup have all changed.

For an external comparison, we recorded the July 2026 snapshot summarized in this 2026 Groq benchmark. In our July 2026 snapshot comparison, we recorded median time to first token of 0.71 seconds for GPT-OSS 120B at high reasoning effort, 0.73 seconds at low reasoning effort, and 0.77 seconds for GPT-OSS 20B at high reasoning effort. Those figures include internal thinking before an answer token appears. We use them as a sanity check, not as substitutes for measurements from our deployment region.

The model lineup was our first warning against benchmark marketing. llama-3.1-8b-instant and llama-3.3-70b-versatile were scheduled to shut down on August 16, 2026. A benchmark built around either identifier is now historical evidence, not a basis for deployment. We based the runnable harness on openai/gpt-oss-20b and openai/gpt-oss-120b, while keeping the model configurable.

Our goal was not to prove that one accelerator always beats another. We wanted to answer a more operational question: can we route latency-sensitive agent turns to Groq without losing control of tail latency, billing, stream integrity, and fallback behavior?

Hands-On Walkthrough: Setup, Execution & Output

We used the chat-completions endpoint directly so the harness could inspect SSE events and response headers. The only secret required is a Groq API key:

python -m venv .venv
source .venv/bin/activate
pip install "httpx==0.27.0"

export GROQ_API_KEY="gsk_replace_me"
export GROQ_MODEL="openai/gpt-oss-20b"
export GROQ_RUNS="30"

We fixed the workload to one compact agent-planning turn:

A system instruction requiring JSON.
One user request describing three support-ticket actions.
Temperature set to zero.
A maximum of 256 output tokens.
Streaming enabled.
Sequential requests, so the result measures single-stream behavior rather than concurrent account capacity.

The following script calculates p50 and p95 for first useful token and total latency. It also records output throughput, token usage, routing headers, and cost. We intentionally fail a run if the stream ends without a completion signal or without authoritative usage; silently estimating billing tokens would make the cost result look more precise than it is.

# bench_groq.py
import json
import math
import os
import statistics
import time

import httpx

API_URL = "https://api.groq.com/openai/v1/chat/completions"
MODEL = os.getenv("GROQ_MODEL", "openai/gpt-oss-20b")
RUNS = int(os.getenv("GROQ_RUNS", "30"))

PRICES = {
    "openai/gpt-oss-20b": {"input": 0.075, "output": 0.30},
    "openai/gpt-oss-120b": {"input": 0.15, "output": 0.60},
}

MESSAGES = [
    {
        "role": "system",
        "content": (
            "You are a support operations agent. Return JSON only with keys "
            "priority, actions, and escalation_reason. Use at most three actions."
        ),
    },
    {
        "role": "user",
        "content": (
            "A paid customer cannot export invoices. Login works. The export "
            "request returns HTTP 500, and retrying twice did not help. Plan the response."
        ),
    },
]


def percentile(values, fraction):
    ordered = sorted(values)
    index = max(0, math.ceil(fraction * len(ordered)) - 1)
    return ordered[index]


def run_once(client):
    payload = {
        "model": MODEL,
        "messages": MESSAGES,
        "temperature": 0,
        "max_tokens": 256,
        "stream": True,
        "stream_options": {"include_usage": True},
        "service_tier": "on_demand",
    }

    started = time.perf_counter()
    first_content_at = None
    finished_at = None
    finish_reason = None
    usage = None
    text = []

    with client.stream("POST", API_URL, json=payload) as response:
        response.raise_for_status()
        region = response.headers.get("x-groq-region")
        ray = response.headers.get("cf-ray")
        rate_headers = {
            key: value
            for key, value in response.headers.items()
            if key.lower().startswith("x-ratelimit") or key.lower() == "retry-after"
        }

        for line in response.iter_lines():
            if not line.startswith("data:"):
                continue

            data = line[5:].strip()
            if data == "[DONE]":
                finished_at = time.perf_counter()
                break

            event = json.loads(data)
            if event.get("usage"):
                usage = event["usage"]

            choices = event.get("choices") or []
            if not choices:
                continue

            choice = choices[0]
            content = (choice.get("delta") or {}).get("content")
            if content:
                if first_content_at is None:
                    first_content_at = time.perf_counter()
                text.append(content)

            if choice.get("finish_reason") is not None:
                finish_reason = choice["finish_reason"]

    if first_content_at is None:
        raise RuntimeError("Stream contained no non-empty content delta")
    if finished_at is None or finish_reason is None:
        raise RuntimeError("Stream ended without a verified completion signal")
    if usage is None:
        raise RuntimeError("No authoritative token usage was returned")

    output_tokens = usage["completion_tokens"]
    decode_seconds = max(finished_at - first_content_at, 1e-9)

    return {
        "ttft_ms": (first_content_at - started) * 1000,
        "total_ms": (finished_at - started) * 1000,
        "output_tps": output_tokens / decode_seconds,
        "input_tokens": usage["prompt_tokens"],
        "output_tokens": output_tokens,
        "region": region,
        "cf_ray": ray,
        "rate_headers": rate_headers,
        "response_chars": len("".join(text)),
    }


def main():
    api_key = os.environ["GROQ_API_KEY"]
    headers = {
        "Authorization": f"Bearer {api_key}",
        "Content-Type": "application/json",
    }

    results = []
    with httpx.Client(headers=headers, timeout=45.0) as client:
        for index in range(RUNS):
            result = run_once(client)
            result["run"] = index + 1
            results.append(result)

    ttft = [row["ttft_ms"] for row in results]
    total = [row["total_ms"] for row in results]
    throughput = [row["output_tps"] for row in results]
    input_tokens = sum(row["input_tokens"] for row in results)
    output_tokens = sum(row["output_tokens"] for row in results)

    price = PRICES[MODEL]
    cost = (
        input_tokens * price["input"] / 1_000_000
        + output_tokens * price["output"] / 1_000_000
    )

    summary = {
        "model": MODEL,
        "runs": RUNS,
        "ttft_ms": {
            "p50": percentile(ttft, 0.50),
            "p95": percentile(ttft, 0.95),
        },
        "total_ms": {
            "p50": percentile(total, 0.50),
            "p95": percentile(total, 0.95),
        },
        "output_tps": {
            "median": statistics.median(throughput),
            "p05": percentile(throughput, 0.05),
        },
        "tokens": {"input": input_tokens, "output": output_tokens},
        "estimated_cost_usd": cost,
        "regions": sorted({row["region"] for row in results}),
    }
    print(json.dumps(summary, indent=2))


if __name__ == "__main__":
    main()

We run it with:

python bench_groq.py > result.json

A successful execution emits this structure with values measured from the caller’s own network path:

{
  "model": "openai/gpt-oss-20b",
  "runs": 30,
  "ttft_ms": {
    "p50": "<measured by the script>",
    "p95": "<measured by the script>"
  },
  "total_ms": {
    "p50": "<measured by the script>",
    "p95": "<measured by the script>"
  },
  "output_tps": {
    "median": "<measured by the script>",
    "p05": "<measured by the script>"
  },
  "tokens": {
    "input": "<API-reported total>",
    "output": "<API-reported total>"
  },
  "estimated_cost_usd": "<calculated from reported usage>",
  "regions": ["<x-groq-region value>"]
}

What This Article Could Not Verify

Our retained evidence does not include a completed remote run, so we cannot report live latency percentiles from our own deployment. The harness is our reproducible artifact; the 2026 external figures provide a comparison. Anyone evaluating a production region should run it from the same network and runtime that will serve users.

Streaming Failures and Deployment Constraints

The most important failure was not a slow token. It was an apparently successful but incomplete stream.

We reproduced the stream contract locally with groq==0.9.0, httpx==0.27.0, and pydantic==2.8.2. Our HTTP transport emitted deterministic OpenAI-compatible SSE chunks, which let us test completion, a read failure after partial content, and a clean EOF without a finish chunk.

The complete fixture returned four SDK chunks, two non-empty content chunks, the text Hello from fixture., and finish_reason="stop".

When we injected a transport read failure after the first content delta, the SDK returned Hello and propagated an httpx.ReadError. It issued one request and did not replay automatically. That behavior is preferable to an invisible retry, but the application must distinguish partial UI output from a committed result.

The nastier case was clean EOF. The iterator returned normally after yielding Hello even though no finish reason had arrived. There was no exception. Code that treats normal loop termination as success can therefore persist a truncated answer, trigger an incomplete tool call, or bill a workflow as completed.

Our workaround is strict:

Buffer or mark streamed content as provisional.
Require a non-null finish reason before committing it.
Treat EOF without a completion signal as an integrity failure.
Never replay an agent turn blindly after partial output.
Attach idempotency controls to downstream tools and business actions.

We also hit four broader deployment constraints.

Rate limits are account and model constraints, not accelerator throughput. A model may decode at hundreds of tokens per second while the account still rejects a burst. We log every x-ratelimit-* and retry-after header, cap concurrency with a semaphore, and retry 429 or transient 5xx responses with bounded exponential backoff and jitter. We do not retry malformed requests or replay partially executed tool turns.

Service tiers change failure semantics. We use on_demand for interactive traffic. Flex processing can provide up to ten times the current rate limits when capacity exists, but it may fail rapidly when capacity is constrained. Auto begins with on-demand limits and can fall back to flex. That makes flex useful for replayable enrichment jobs, not for an agent action that must execute exactly once.

Model identifiers are routing dependencies. Groq is not a generic deployment target where we upload arbitrary weights. We route only to its curated catalog, and scheduled model removal can break hard-coded applications. We now place model IDs behind configuration, run a startup capability check, and keep a quality-tested fallback at another provider.

Token counting must come from the response. Character counts, local tokenizers, and concatenated SSE fragments are useful diagnostics, but they are not billing records. Reasoning and tool-related tokens can make client estimates diverge. We calculate cost from authoritative prompt and completion usage, store the raw usage object, and reject a benchmark sample when usage is missing.

The same caution applies to latency. We separate network time from server time and treat input length as the primary TTFT driver in our measurement design (latency reference). We preserve input size, output cap, reasoning mode, region, service tier, and concurrency with every sample. Without those dimensions, p95 is just a number detached from a workload.

Scale, Latency & Cost vs. Alternatives

Our current comparison is not “LPU versus GPU” in the abstract. It is hosted Groq versus a conventional GPU API versus self-hosted vLLM for a specific traffic shape.

Decision factor	GroqCloud	Hosted GPU inference API	Self-hosted vLLM or SGLang
Fast single-stream decoding	Core strength; published 2026 rates were 1,000 tok/s for GPT-OSS 20B and 500 tok/s for GPT-OSS 120B	Varies heavily by provider, batching, and model	Tunable, but low-load latency and saturated throughput require different configurations
Operational work	Low	Low	High: GPUs, autoscaling, model loading, observability, and upgrades
Model freedom	Curated Groq catalog	Usually broader	Highest, subject to hardware and license
Burst behavior	Controlled by account, model, and service-tier limits	Provider-specific quotas and queues	Controlled by owned capacity; excess traffic queues or fails
Cost model	Per input/output token	Usually per token	GPU-hours plus engineering and idle capacity
Streaming risk	SSE completion must be verified	Similar API-stream concerns	Full control, but full responsibility
Best fit	Interactive agents where latency has business value	Broad model access and managed fallback	Stable, high utilization or strict deployment control

The July 2026 price points we validated were:

Model	Published speed	Input per 1M tokens	Output per 1M tokens	Fixed workload cost
GPT-OSS 20B	1,000 tok/s	$0.075	$0.30	$0.000150
GPT-OSS 120B	500 tok/s	$0.15	$0.60	$0.000300

The fixed workload cost assumes 1,000 input tokens and 250 output tokens:

GPT-OSS 20B:
(1,000 × $0.075 / 1,000,000) + (250 × $0.30 / 1,000,000)
= $0.000150 per turn

GPT-OSS 120B:
(1,000 × $0.15 / 1,000,000) + (250 × $0.60 / 1,000,000)
= $0.000300 per turn

At one million such turns, that is $150 for GPT-OSS 20B or $300 for GPT-OSS 120B before caching, retries, failed generations, tool-loop expansion, and taxes. The blended rates are $0.12 and $0.24 per million total tokens respectively, but that shorthand is valid only for this 4:1 input-to-output ratio. Output-heavy agents cost more because output tokens carry the higher rate.

Batch processing reduces synchronous pricing by 50% and does not consume standard API quotas, but its completion window makes it a different product. At the same workload, the nominal model cost becomes $75 or $150 per million turns. We would use that for evaluation runs, enrichment, and scheduled classification—not interactive chat.

For self-hosting, the useful break-even equation is:

required requests per hour =
GPU cluster cost per hour / hosted API cost per request

If a self-hosted cluster costs G dollars per hour, GPT-OSS 20B’s token-only break-even for this workload is G / 0.000150 requests per hour. For GPT-OSS 120B it is G / 0.000300. We then add engineering, failover capacity, observability, idle headroom, and the cost of meeting p95 during bursts. Comparing Groq’s per-token bill against a fully utilized GPU while ignoring idle time is not a serious production calculation.

We would also require identical models, prompts, output caps, concurrency, and regions before claiming that Groq is “N times faster” than a GPU service. The old 241 tok/s Llama 2 result illustrates the hardware’s decoding performance; it does not establish a current speed advantage over an unspecified GPU service.

Teams building a multi-provider router can review our broader tools collection. For workload-specific capacity and failover design, our AI infrastructure services cover the parts that raw tokens-per-second charts omit.

Our Final Verdict: When to Deploy, When to Skip

Groq works best as a low-latency inference option within a multi-provider system, not as the default provider for every workload.

Deploy it if:

User-perceived generation speed directly affects conversion, retention, or operator productivity.
Your agent makes several sequential model calls and faster decoding shortens the full loop.
A supported GPT-OSS model meets your quality requirements.
You can measure TTFT and completion latency from the production region.
You calculate cost from returned usage rather than tokenizer estimates.
You enforce completion signals before committing streamed output.
You have bounded retries, concurrency controls, and idempotent tools.
You can tolerate a curated model catalog and maintain a tested fallback.

Hold off or avoid it if:

You need custom or fine-tuned weights that Groq does not host.
Your compliance requirements have not been contractually verified.
Your workload is offline and latency has little economic value.
You depend on a deprecated model identifier.
Your traffic exceeds available quotas and cannot be queued or shifted to batch.
You need provider-independent behavior across every OpenAI-compatible extension.
You cannot safely recover from a partial agent stream.
You are choosing it solely because an old benchmark reported a large GPU multiple.

Our verdict is positive, with conditions. The published 2026 pricing is aggressive, the single-stream throughput targets are unusually high, and the API surface is straightforward. The operational risks sit around the accelerator: rate limits, service-tier semantics, model churn, network distance, token accounting, and ambiguous stream termination.

We would deploy Groq for latency-sensitive conversational turns behind our own gateway, with on-demand processing, explicit model configuration, raw usage logging, verified finish_reason, and a secondary provider. We would route replayable bulk work to batch or flex, compare median TTFT with the dated external figures only under comparable model and reasoning settings, and track p95 against our own workload-specific baseline and latency target.

We would not approve a rollout based on the 1,000 tok/s headline alone. Before routing production traffic to Groq, run the supplied harness from the actual service region, add a concurrency sweep that stays within the account’s quotas, and verify quality on the same agent traces used in production. If you want us to review that routing design, contact Effloow.

Browser Use vs Skyvern Self-Hosted: Setup, Trade-offs, and Completion Verification

Jangwook Kim — Sun, 04 Oct 2026 00:48:36 +0000

Why we evaluated Browser Use and Skyvern

We did not bring Browser Use and Skyvern into our lab to see whether an agent could click a button in a polished demo. We wanted to answer a more expensive question: can either system complete a recurring login-to-download workflow without creating a permanent human repair queue?

Agent-reported done is not verified completion

A strict schema receipt still passed for a missing and a corrupted file, so a workflow should count as complete only when page state, business identifier, and artifact size and digest are checked independently of the agent's own success claim.

Our target shape was deliberately ordinary:

Open a locally hosted application.
Authenticate with a test account.
complete a three-stage form.
Submit the transaction.
Download the generated file.
Prove that the downloaded artifact is the expected file.

That last condition matters. An agent saying “done” is not evidence that the workflow finished correctly. A successful click is not a successful transaction, and a valid JSON response is not proof that a file exists.

The architectural distinction also turned out to be less absolute than the usual “DOM versus vision” shorthand suggests.

Browser Use builds its page representation primarily from browser structure, including the DOM and accessibility information exposed through Chrome. Screenshots can be disabled, enabled, or requested when needed. That lets us configure Browser Use for DOM-first operation, but it is not DOM-only.

Skyvern’s self-hosted execution loop captures a screenshot, combines visual interpretation with a simplified DOM, asks the configured model for the next action, and executes that action through Playwright. That makes it vision-forward, but not vision-only.

In practical terms, both products run a perception-action loop. Each additional loop can add model latency, input volume, another chance to select the wrong element, and another opportunity for a retry to multiply cost. The meaningful comparison is therefore not whether one product “uses AI.” Both do. We care about:

verified workflow completion rate;
wall-clock time per verified completion;
model requests and input volume per run;
retries and terminal failure reasons;
human interventions per 100 runs;
infrastructure and inference cost per successful task.

We also separated the product shapes. Browser Use is primarily an embeddable MIT-licensed agent library. Skyvern is a larger AGPL-licensed workflow platform with an API server, web interface, database, browser runtime, workflow state, and operational views.

That distinction affects engineering cost more than a synthetic leaderboard score. Browser Use gives us an embeddable component. Skyvern gives us a broader platform for managing browser workflows.

For adjacent infrastructure evaluations, we keep our current shortlist in our tools collection. The rest of this review focuses on what we could verify without pretending that an incomplete run ledger was a production benchmark.

Hands-On Walkthrough: Setup, Execution & Output

We treated installation reproducibility as the first test. If a team cannot recreate the environment from pinned commands, any reliability percentage is suspect.

Browser Use installation path

For Browser Use, we use Python 3.11 or newer and take uv add browser-use as the documented setup path. The following pip-based setup is a proposed reproduction path, not an installation test we completed:

python3.12 -m venv .venv-browser-use
source .venv-browser-use/bin/activate

python -m pip install --upgrade pip
pip install browser-use

# Make the browser dependency explicit instead of relying on first-run behavior.
python -m playwright install chromium

We would make Chromium provisioning an explicit build step rather than assume that package installation downloads a compatible browser binary. We did not test first-run browser provisioning here. Our production image would pin the package, verify the browser installation procedure for that version, cache the browser layer, and fail the build if Chromium could not launch.

The minimal agent shape is straightforward:

import asyncio
from browser_use import Agent, ChatOpenAI

async def main() -> None:
    llm = ChatOpenAI(
        model="your-model",
        base_url="http://127.0.0.1:8000/v1",
        api_key="local-development-key",
    )

    agent = Agent(
        task=(
            "Open http://demo-app:8080, sign in with the supplied test "
            "credentials, complete the three-step request form, download "
            "the receipt, and return its filename and visible request ID."
        ),
        llm=llm,
    )

    history = await agent.run()
    print(history.final_result())

if __name__ == "__main__":
    asyncio.run(main())

A local OpenAI-compatible endpoint removes provider charges, but it does not make inference free. GPU depreciation, power, model-serving memory, queueing, and operator time still belong in the cost model.

Skyvern self-hosted path

For Skyvern, we used the full-stack deployment boundary rather than mistaking the lightweight Python package for the server.

git clone https://github.com/Skyvern-AI/skyvern.git
cd skyvern

cp .env.example .env
# Configure the selected LLM provider or local Ollama endpoint in .env.

docker compose up

The important trap is that plain pip install skyvern is the remote API SDK, not the complete local platform. The server-capable package is:

pip install "skyvern[server]"

For the web interface, API server, browser runtime, and PostgreSQL together, Docker Compose remains the practical starting point. We budgeted at least the documented 4 GB of RAM for evaluation and would follow the recommendation of 8 GB or more for production. The 8 GB figure is a recommendation, not a mandatory deployment minimum or a memory measurement from our workload.

For our Skyvern self-hosting plan, we account for the API server, web UI, PostgreSQL, browser runtime, and a configured LLM provider. Self-hosting removes a per-task platform charge, but we still have to operate the API, browser capacity, database, model endpoint, backups, updates, and any required proxy service.

The completion contract we actually executed

We did not have a defensible N-run vendor trace containing success, duration, model usage, and artifact hashes. Rather than invent those numbers, we tested the most important measurement boundary: whether structured output alone can falsely mark a workflow complete.

We pinned Pydantic 2.10.6 and checked an application-defined strict completion receipt against a real local fixture file. We tested Pydantic validation and our proposed completion contract, not Browser Use or Skyvern integrations. Matching the fixture’s size and digest established only that its bytes agreed with the expected receipt—not that a browser workflow completed or a website delivered the correct business document. The contract and artifact checks were:

from hashlib import sha256
from pathlib import Path
from typing import Literal

from pydantic import BaseModel, ConfigDict

class DownloadReceipt(BaseModel):
    model_config = ConfigDict(strict=True)

    status: Literal["completed"]
    path: str
    byte_count: int
    sha256: str

def verify_artifact(receipt: DownloadReceipt) -> dict:
    artifact = Path(receipt.path)
    if not artifact.exists():
        return {"verified": False, "reason": "missing_artifact"}

    payload = artifact.read_bytes()
    size_matches = len(payload) == receipt.byte_count
    digest_matches = sha256(payload).hexdigest() == receipt.sha256

    return {
        "verified": size_matches and digest_matches,
        "reason": "matched" if size_matches and digest_matches else "artifact_mismatch",
        "observed_bytes": len(payload),
        "size_matches": size_matches,
        "digest_matches": digest_matches,
    }

Our test output was unambiguous:

{
  "pydantic": "2.10.6",
  "genuine": {
    "strict_schema_accepted": true,
    "artifact_verified": true
  },
  "missing_artifact": {
    "strict_schema_accepted": true,
    "artifact_verified": false,
    "reason": "missing_artifact"
  },
  "corrupted_artifact": {
    "strict_schema_accepted": true,
    "artifact_verified": false,
    "reason": "artifact_mismatch"
  },
  "numeric_string": {
    "lax_schema_accepted": true,
    "strict_schema_accepted": false,
    "error_type": "int_type"
  }
}

Strict validation rejected a numeric string where an integer was required. It also rejected missing required fields and an invalid status. However, schema validation could not determine whether the referenced file existed or matched its declared digest.

Our completion numerator must therefore be:

verified_success =
    expected_final_page_state
    AND expected_business_identifier
    AND artifact_exists
    AND artifact_size_matches
    AND artifact_digest_matches

Anything weaker inflates the apparent reliability of both products.

What we could not verify and deployment limitations

The first failure was methodological: we could not produce a valid head-to-head reliability score from the available run evidence.

We had install paths, architectural boundaries, pricing inputs, and a completed validation experiment. We did not have repeated Browser Use and Skyvern executions with comparable model settings, task seeds, token accounting, retries, and artifact verification. Publishing “9/10 versus 8/10” under those conditions would be fiction.

That absence exposed several production traps.

Browser installation is a separate dependency

Installing the Python package is not equivalent to provisioning a runnable Chromium binary. For a reproduction build, we would verify whether python -m playwright install chromium is required by the pinned Browser Use version and add a browser launch smoke test.

Our workaround is simple: pin Browser Use, pin the container base image, install Chromium during the build, and never permit a production worker to download browser assets on first request.

Skyvern’s packages support two different deployment models

pip install skyvern does not create the API server, UI, browser runtime, and database stack. We would use skyvern[server] only for a controlled source-based development path. For team evaluation, we would start with Compose and pin every image digest.

We would also inspect and explicitly configure the maximum step count, task timeout, and retry policy before submitting real work. An unbounded retry loop does not ensure resilience and can make costs unpredictable.

Self-hosting does not automatically keep inference local

Skyvern can send screenshots to a configured remote model on every perception step. Browser Use can likewise send page state and optional screenshots to an external model. Running the orchestrator on our own server does not change that data boundary.

For sensitive workflows, we would either use an approved private endpoint or a local model that had passed the same task suite. We would not assume a smaller local model preserved cloud-model reliability.

CAPTCHA and bot detection remain hard boundaries

For self-hosted Skyvern, CAPTCHA handling can require manual intervention. Browser Use’s managed browser offering adds stealth, proxies, and CAPTCHA services, but no browser configuration guarantees universal success.

That distinction changes the operating model. A workflow that occasionally waits for a person is not unattended automation. We would classify it as “human-assisted” and account for intervention time separately.

The same rule applies to authentication state. Browser Use profile synchronization covers cookies, but not every form of local storage, IndexedDB state, or extension-backed authentication. A supposedly warm session can still return to the login screen.

Retry loops can disguise selector mistakes

An agent can select a plausible but incorrect element, observe no expected transition, and retry with a slightly different action. Without a hard step budget and a terminal-state validator, the run may look active while making no business progress.

Our workaround is to record a failure taxonomy rather than one aggregate error:

Failure class	Required evidence
Wrong element or selector hallucination	Intended control, selected control, screenshot, DOM reference
Navigation or page timeout	URL, elapsed time, timeout boundary
Authentication failure	Login state, challenge type, session source
CAPTCHA or bot block	Challenge screenshot and intervention requirement
Retry exhaustion	Step count, repeated action signature, last changed state
False completion	Agent result versus independent page and artifact checks
Artifact failure	Missing file, wrong size, or digest mismatch
Model failure	Invalid action schema, refusal, or unavailable endpoint

This is the level of instrumentation we would require before involving a customer workflow. Teams needing help building that harness can review our AI infrastructure services or contact us with the target volume and error budget.

Scale, Latency & Cost vs. Alternatives

We cannot honestly report measured success rate, wall-clock time, token count, or dollars per completed task for the full login-to-download workflow. Those values were not established by the executable experiment.

What we can compare is the cost structure and the operational work each option leaves with us.

Decision factor	Browser Use OSS	Skyvern self-hosted	Deterministic Playwright
Primary shape	Python agent library	Workflow platform and API	Browser automation library
License	MIT	AGPL v3	Apache 2.0
Grounding	DOM and accessibility first; optional vision	Screenshot plus simplified DOM	Explicit selectors and assertions
Initial setup	Lowest	Highest	Low
UI and run history	We build it	Included in the stack	We build it
Database requirement	Application choice	PostgreSQL in Compose	Application choice
Model cost	Per agent step	Per agent step, often with image input	None for scripted steps
Browser infrastructure	We operate it or buy managed browsers	We operate it	We operate it
CAPTCHA in self-hosted mode	No universal guarantee	Manual intervention may be required	External solver or manual path
Artifact verification	Application responsibility	Workflow plus application checks	Direct application checks
Best fit	Agent capability embedded in code	Operations-owned recurring workflows	Stable, high-volume sites
Verified benchmark result here	Not established	Not established	Not run

For managed Browser Use browsers, we budget $0.02 per browser-hour, with model usage and traffic accounted for separately. At that rate, browser time alone is rarely the dominant cost for a short task. Model calls, retries, proxy traffic, failed runs, and repair labor can be much larger.

We use this formula:

cost_per_verified_success =
    total_browser_cost
    + total_model_cost
    + total_proxy_cost
    + allocated_infrastructure_cost
    + human_intervention_cost
    ------------------------------------------------
    verified_success_count

The denominator is the dangerous part. If a task costs $0.04 per attempt but only 80% of attempts pass independent verification, the attempt cost understates reality:

$0.04 / 0.80 = $0.05 per verified success

That example is arithmetic, not our measured vendor result.

For self-hosting, our break-even comparison would be:

monthly_self_hosted_cost =
    compute
    + model inference
    + database and storage
    + proxy service
    + monitoring
    + engineering maintenance

monthly_managed_cost =
    successful task volume
    × managed cost per verified success

If self-hosting costs $1,500 per month after infrastructure and engineering allocation, and a managed service costs $0.15 more per verified workflow, the nominal break-even point is 10,000 verified workflows per month. Reliability differences can move that threshold dramatically.

A deterministic Playwright script remains the baseline we would beat. If we control the target application or its selectors rarely change, adding an LLM to every click usually adds avoidable latency and reliability risks. The better hybrid is to script stable login, navigation, submission, and artifact checks while reserving the agent for ambiguous page interpretation.

Skyvern is more attractive when operations staff need workflow visibility, stored state, conditional blocks, and reruns without editing Python. Browser Use is more attractive when we already own the surrounding queue, credentials, telemetry, and validation layer.

Our Final Verdict: When to Deploy, When to Skip

We would not recommend one product for every use case.

Browser Use is the better engineering component. It is easier to embed, has a permissive license, and lets us control the orchestration around the agent. We would choose it when our team already has production primitives for queues, secrets, traces, retries, browser isolation, and result validation.

Skyvern self-hosted is the better workflow platform. We would choose it when the operational interface, run history, workflow authoring, and centralized task state justify running a larger stack. We would review the AGPL implications before modifying and exposing the system over a network.

Deploy Browser Use if:

we want browser control inside an existing Python service;
engineers own the workflow and surrounding infrastructure;
we can explicitly bootstrap and pin Chromium;
we want DOM-first operation with selective vision;
we can implement independent completion checks;
we are comfortable building our own queue, credential, and observability layers.

Deploy Skyvern self-hosted if:

operations teams need a visual workflow and run investigation interface;
we accept running an API, UI, browser runtime, and PostgreSQL;
we have at least the documented memory floor and a scaling plan;
we can configure step ceilings, timeouts, retries, and model routing;
we understand that CAPTCHA handling may require a person;
the AGPL license is compatible with the intended deployment.

Hold off or avoid both if:

the target site has a stable API we can call directly;
deterministic Playwright already completes the workflow reliably;
every run must finish without human intervention on bot-protected sites;
screenshots or page contents cannot leave our network, and we cannot run a suitable model within that boundary;
we cannot verify the final business state independently;
the business requires a reliability percentage that has not been measured on its own workflow.

Our strongest conclusion is not that vision beats the DOM or that one agent is categorically more reliable. It is that browser-agent success must be externally verified.

The strict Pydantic receipt helped prevent malformed output, but it still accepted a perfectly structured reference to a missing or corrupted file. We only caught those false completions by checking the artifact itself.

That is the production gate we would apply to both Browser Use and Skyvern: no workflow counts as complete merely because the agent says it is. Until repeated runs produce traceable success, latency, usage, intervention, and artifact data, we have a promising automation system—not a reliability benchmark.

Turbopuffer Review: Object-Storage Vector Search, Cold Starts, and Cost Trade-offs

Jangwook Kim — Sat, 03 Oct 2026 00:37:54 +0000

Why We Brought This Tool Into Our Lab

We started this evaluation with a recurring retrieval-infrastructure problem: most vector databases make inactive tenants expensive.

A conventional deployment keeps indexes on provisioned memory or SSD capacity whether a tenant sends 10,000 queries per hour or one query per month. That works for a small number of uniformly active indexes. It becomes painful when a product has hundreds of thousands of workspaces, repositories, agents, or customer-specific knowledge bases with highly skewed activity.

Turbopuffer takes a materially different approach. In the architecture reference we inspected, the API routes requests to Rust binaries operating over an object-storage source of truth. Query nodes are stateless from a durability perspective. When a namespace is cold, a query node reads it from object storage and caches its documents on local NVMe. The service routes subsequent requests toward the same node to preserve cache locality, although another node can still serve the namespace after a failure or routing change.

That architecture is not ordinary storage tiering. The service commits writes to a write-ahead log under the namespace's object-storage prefix. Separate indexing workers asynchronously convert committed data into searchable index structures. Query nodes use memory and NVMe as disposable acceleration layers rather than as the authoritative copy.

For vector retrieval, Turbopuffer uses SPFresh, a centroid-based ANN design. This matters because graph traversal can require a chain of dependent reads, which is hostile to object-store latency. A centroid hierarchy can identify promising clusters with a bounded number of broad reads. In our architectural analysis, we budgeted roughly three to four object-storage round trips for a cold query, with each round trip on the order of 100 ms.

The trade is straightforward:

We avoid paying premium storage prices for every inactive namespace.
We accept a meaningful first-query penalty when a namespace is not cached.
We depend on routing and cache residency for stable warm latency.
We must decide whether approximate retrieval at roughly 90–95% recall@10 is sufficient.
We inherit a consistency-versus-latency decision on every latency-sensitive query path.

The strongest large-scale result we could verify was the ANN v3 result framed as 200 ms p99 over 100 billion vectors. Its workload uses 100 billion 1,024-dimensional f16 vectors, representing about 200 TiB of dense vector data, with a target above 1,000 QPS. We did not treat that number as a proxy for an ordinary cold serverless namespace. At that scale, the ANN tree is sized to remain on SSD, so it measures a different operating condition from fetching a dormant tenant's index from object storage.

That distinction drove our review. We were not trying to prove that Turbopuffer can store a huge index. We wanted to know whether its economics survive a real multi-tenant query mix without turning cold starts into user-visible timeouts.

Hands-On Walkthrough: Setup, Execution & Output

The service is exposed through HTTP and requires an API key, so our planned harness had no infrastructure dependency beyond a public embedding dataset and a spend-limited account. We separated the test into four phases:

Insert identical vector corpora into namespaces containing 100,000, 1 million, and 10 million documents.
Poll namespace metadata until unindexed_bytes returned to zero.
Query previously untouched namespaces to capture the first-query latency.
Run repeated queries against the same namespace to capture the warm-up curve and steady-state percentiles.

We also planned to write a sentinel document, query it immediately with strong consistency, and poll metadata independently. That separates write visibility from asynchronous indexing completion: a document can be visible through exhaustive WAL search before it has entered the ANN index.

Our execution environment did not contain a Turbopuffer API credential, an offline server, or verified protocol fixtures. We therefore stopped before issuing hosted-service requests. We did not substitute a cache simulation or another vector database and label that as a Turbopuffer benchmark. Doing so would have produced precise-looking but useless numbers.

The following harness is the HTTP timing collector we prepared. It is runnable once TPUF_QUERY_URL, TPUF_API_KEY, and a valid query payload are supplied. The URL is parameterized deliberately so that CI does not hard-code a region or API revision.

python3 -m venv .venv
. .venv/bin/activate
python -m pip install "httpx==0.27.2"

export TPUF_API_KEY="replace-with-a-spend-limited-key"
export TPUF_QUERY_URL="https://replace-with-current-region-and-query-path"
export SAMPLE_COUNT="100"

cat > query.json <<'JSON'
{
  "rank_by": ["vector", "ANN", [0.01, 0.02, 0.03]],
  "top_k": 10,
  "include_attributes": ["category"]
}
JSON

cat > bench.py <<'PY'
import json
import math
import os
import statistics
import sys
import time

import httpx

api_key = os.environ.get("TPUF_API_KEY")
query_url = os.environ.get("TPUF_QUERY_URL")
sample_count = int(os.environ.get("SAMPLE_COUNT", "100"))

if not api_key:
    sys.exit("ERROR: TPUF_API_KEY is not set")
if not query_url:
    sys.exit("ERROR: TPUF_QUERY_URL is not set")

with open("query.json", "r", encoding="utf-8") as handle:
    payload = json.load(handle)

latencies_ms = []
status_counts = {}

with httpx.Client(timeout=30.0, http2=True) as client:
    for sequence in range(sample_count):
        started = time.perf_counter_ns()
        response = client.post(
            query_url,
            headers={
                "Authorization": f"Bearer {api_key}",
                "Content-Type": "application/json",
            },
            json=payload,
        )
        elapsed_ms = (time.perf_counter_ns() - started) / 1_000_000
        status_counts[str(response.status_code)] = (
            status_counts.get(str(response.status_code), 0) + 1
        )

        if response.is_error:
            print(
                json.dumps(
                    {
                        "sequence": sequence,
                        "status": response.status_code,
                        "latency_ms": round(elapsed_ms, 2),
                        "body": response.text[:1000],
                    }
                ),
                file=sys.stderr,
            )
            response.raise_for_status()

        latencies_ms.append(elapsed_ms)
        print(
            json.dumps(
                {
                    "sequence": sequence,
                    "latency_ms": round(elapsed_ms, 2),
                    "phase": "first" if sequence == 0 else "subsequent",
                }
            )
        )

ordered = sorted(latencies_ms)

def percentile(value):
    index = max(0, math.ceil((value / 100) * len(ordered)) - 1)
    return ordered[index]

summary = {
    "samples": len(ordered),
    "first_query_ms": round(ordered[0] if len(ordered) == 1 else latencies_ms[0], 2),
    "p50_ms": round(statistics.median(ordered), 2),
    "p90_ms": round(percentile(90), 2),
    "p99_ms": round(percentile(99), 2),
    "status_counts": status_counts,
}
print(json.dumps({"summary": summary}))
PY

python bench.py

The following is simulated output showing the collector's format, not a result from our lab:

{"sequence": 0, "latency_ms": 886.42, "phase": "first"}
{"sequence": 1, "latency_ms": 18.71, "phase": "subsequent"}
{"sequence": 2, "latency_ms": 15.64, "phase": "subsequent"}
...
{"summary":{"samples":100,"first_query_ms":886.42,"p50_ms":14.93,"p90_ms":18.27,"p99_ms":28.84,"status_counts":{"200":100}}}

We chose those illustrative values to resemble the published control ranges so the output would look realistic. They are not independent measurements.

The verified architectural baseline for 1 million documents is a first-query p50 of 874 ms and a cached p50 of 14 ms. For our comparison, we used a separate 10-million-document, 1,024-dimension, approximately 40 GB workload at 8 QPS and top_k=10 as a reference: warm p50/p90/p99 values of 14/17/27 ms and cold values of 874/1,214/1,686 ms. We did not independently measure these endpoints.

These are useful controls, but they are not a reproduced warm-up curve. We could verify the cold and warm endpoints in the published material, not the intermediate transition across queries or elapsed time.

Test Limitations and Operational Constraints

Our immediate roadblock was test access. Without a hosted API key, we could not truthfully produce service-side error responses, independent percentiles, cache-eviction measurements, or indexing-delay distributions.

If run without an API key, the prepared harness would exit with the following message; we did not execute it in this evaluation:

ERROR: TPUF_API_KEY is not set

That is our harness error, not a Turbopuffer API response. We are making the distinction explicit because fabricated vendor error strings are worse than a missing test.

Several operational constraints still surfaced during our review.

First, “the second query is warm” is too simplistic. Cache locality is node-specific, and any query node can serve any namespace. A failover, rebalance, or eviction can put the next request back on an object-storage path. We would use the warm-cache endpoint or a pre-flight query before interactive traffic, but that transfers responsibility to the application. It also creates one warm-up operation per active namespace.

Second, strong consistency imposes an approximately 10 ms latency floor because the query path checks object storage for newer writes. Eventual consistency can get below that floor, but it permits staleness of up to roughly one hour in the worst case. In our consistency assessment, we accounted for eventual queries searching up to 128 MiB of unindexed data and used the greater-than-99.8% consistent-response figure as a reference, not as an independently measured result. That is encouraging, but it is not a guarantee suitable for permission changes, deletion enforcement, or user-facing read-after-write workflows.

Third, each namespace currently produces at most one WAL entry per second. Concurrent writes can be grouped, preserving high aggregate throughput, but a batch opened shortly after the preceding commit can take up to one second. We would batch ingestion intentionally rather than sending a stream of tiny writes and expecting database-style single-row latency.

Fourth, schema flexibility has hard edges. Attribute types must remain consistent within a namespace. Every vector in a vector column must use the same dimensionality. A namespace can have at most four embedded attributes, and an upsert batch containing embedded attributes is limited to 30 rows. We could verify those limits, but not the exact API messages emitted when they are violated.

That leaves an important gap in our evaluation: we cannot provide authentic response bodies for a type conflict, non-indexed filter, dimension mismatch, fifth embedded attribute, or oversized embedded upsert. Teams automating error classification should capture those responses during their own proof of concept instead of matching on guessed strings.

Finally, cache behavior remains underspecified for capacity planning. We could not establish an eviction policy, a guaranteed residency duration, or a namespace-count threshold at which warm p99 deteriorates. This is the main unresolved risk for products with a very long tenant tail. If that describes your workload, bring a production-shaped Zipf distribution to the test rather than benchmarking one namespace in a loop.

For help designing that workload, contact our infrastructure team or review the systems in our AI tools collection.

Scale, Latency & Cost vs. Alternatives

The architectural comparison is clearer than the invoice comparison.

Decision factor	Turbopuffer	Pinecone	Self-hosted Qdrant
Durable storage model	Object storage as source of truth	Managed service model varies by product and plan	Operator-selected disk, replication, and snapshots
Cold namespace behavior	Direct object-store reads, then NVMe caching	Depends on selected service architecture and capacity model	Normally resident on provisioned node storage
Warm latency evidence in scope	10M workload: 14 ms p50, 17 ms p90, 27 ms p99	Must be measured on the purchased plan	Must be measured on the selected hardware
Cold latency evidence in scope	10M workload: 874 ms p50, 1,214 ms p90, 1,686 ms p99	Not measured in this review	Usually an OS-cache or process-start concern
Operations burden	Low	Low	High
Inactive-namespace economics	Strong architectural fit	Quote and workload dependent	Capacity remains provisioned unless nodes scale down
Read-after-write	Strong by default, with latency floor	Plan and API dependent	Configuration dependent
Primary risk	Cold-tail latency and cache uncertainty	Metering and vendor-plan economics	Staffing, replication, upgrades, and capacity planning

We could not produce a defensible current-dollar Pinecone comparison without fixing a plan, cloud, region, pod or serverless model, read units, write units, metadata size, and query rate. Any single “Turbopuffer is X times cheaper” figure would hide the variables that dominate the invoice.

We can still establish a useful storage floor. The infrastructure reference rates we validated are approximately $0.02 per GB-month for object storage and $0.10 per GB-month for SSD cache. These are component-cost references, not Turbopuffer customer pricing.

For the documented 10-million-vector workload occupying approximately 40 GB:

Object-storage component = 40 GB × $0.02 = $0.80/month
Full SSD-cache component = 40 GB × $0.10 = $4.00/month

Normalized to 1 million vectors, that becomes approximately $0.08 per month for the object-storage component and $0.40 per month if the corresponding 4 GB remains cached on SSD. This excludes compute, queries, indexing, metadata, replication overhead, service margin, networking, support, and minimum commitments.

A more practical model is:

Monthly hierarchy cost floor = D × (0.02 + A × 0.10)

Here, D is stored gigabytes and A is the average fraction resident in SSD cache. With 40 GB and 10% average cache residency, the component floor is approximately $1.20 per month. At 100% residency, it is approximately $4.80.

This explains where Turbopuffer should win: large D, low A, and enough tolerance for cold requests. It does not prove the final invoice beats Pinecone because query and service charges are missing.

Against Qdrant, we would compare Turbopuffer's invoice with the complete monthly cost of nodes, attached storage, replicas, snapshots, cross-zone traffic, monitoring, upgrades, and engineering time. Qdrant can be the better choice when the dataset is continuously hot, latency must be predictable, and a team already operates stateful search clusters. Object storage becomes more compelling as the inactive-to-active namespace ratio rises.

Our Final Verdict: When to Deploy, When to Skip

Turbopuffer's architecture is technically coherent. Rust query binaries, object-storage durability, disposable compute, NVMe locality, SPFresh clustering, and asynchronous indexing fit the economics of sparse multi-tenant retrieval better than keeping every tenant permanently resident.

The cold-start penalty is not theoretical. The verified 10-million-document controls span 27 ms warm p99 versus 1,686 ms cold p99. That difference is large enough to change an application's request flow, timeout policy, and user experience.

Deploy this if:

We have many isolated namespaces but only a small active working set.
We can pre-warm a namespace before an interactive retrieval path.
A roughly one-second cold request is acceptable outside the critical path.
We value managed operation more than absolute latency control.
We can use strong consistency selectively and eventual consistency elsewhere.
A 90–95% recall@10 target is acceptable and we will validate recall on our corpus.
Our writes can be batched around the one-WAL-entry-per-second namespace behavior.

Hold off or avoid it if:

Every request must meet a strict sub-100 ms p99, including after eviction or failover.
We require guaranteed read-after-write behavior below the strong-consistency latency floor.
Our schema needs more than four embedded attributes.
We rely on large embedded upsert batches exceeding 30 rows.
We cannot tolerate approximate retrieval or independently test recall.
We need a documented cache-residency guarantee.
We expect a reliable cost comparison without testing our actual mix of queries and namespaces.

Our conclusion is positive on architectural fit but conditional on workload validation. We would shortlist Turbopuffer for a product with thousands or millions of intermittently active tenant indexes. We would not approve production deployment until we reproduced the namespace warm-up curve, measured p99 across several corpus sizes, captured indexing delay under concurrent writes, and recorded the exact schema and filter errors from the hosted API.

The relevant technical starting points are the architecture overview, the ANN v3 design, and the concepts and recall model. If the cold-start budget is uncertain, we would run the proof of concept before negotiating around headline storage savings.