DEV Community: Yuri Pocepaev

When Offloading Stalls Inference: Queue Capacity, CUDA Event Ownership, and Misleading Health Checks

Yuri Pocepaev — Mon, 05 Oct 2026 08:29:24 +0000

An inference server stopped generating tokens while more than 200 GiB of host memory remained available. Its health endpoint kept returning HTTP 200. The first visible exception was queue.Full inside a CPU-offload connector; the engine reached a fatal RPC timeout roughly five minutes later.

The useful lesson is about concurrency contracts. A connector built around one outstanding model pass was running inside a runtime that could prepare overlapping passes. Its queue, synchronization event, and input lifetime needed to follow that execution model together.

Increasing the queue capacity addressed one symptom. In a deterministic reproduction, it left a possible circular wait intact.

This article walks through the evidence, the failure mechanism, and the design rules that follow. The exact CUDA interleaving during the incident was not captured. CPU reproductions establish that the mechanisms are possible in the inspected code; they do not replace GPU qualification.

What was being offloaded

The investigated PLE path prepared input tokens, computed the required embeddings on the CPU, and supplied the result to GPU execution. A background thread received notifications through an internal queue.

This queue contained work for model passes. It was distinct from the serving scheduler's queue of user requests. A low user-facing waiting count therefore said little about whether the connector could accept another notification.

The inspected implementation used this pattern:

request_queue = queue.Queue(maxsize=1)
# Later, when launching offload work:
request_queue.put_nowait(request)

A bounded queue has a software-defined capacity. Available RAM does not expand it. Python documents that a nonblocking insertion raises Full when no slot is available. Python queue documentation.

The queue enforced its limit correctly. The surrounding code assumed that a second insertion could not arrive while the first notification still occupied the slot.

The assumption hidden inside “one pass at a time”

The connector's rationale relied on each forward consuming its offload result before the next forward. It also referred to disabling dual batch overlap, or DBO.

That constraint did not disable asynchronous scheduling.

CPU-side preparation can advance after GPU operations have been submitted, before those operations complete. In the inspected runtime, the asynchronous execution path allowed two unfinished batches even with a single pipeline stage.

The effective async mode was reconstructed from launch settings and configuration logic. The live configuration object was not captured during the incident, which limits how directly that part of the execution can be established.

A minimal problematic schedule is nevertheless simple:

Pass A inserts its offload notification.
The background thread has not removed it yet.
The runtime prepares pass B.
B attempts another nonblocking insertion.
The only queue slot is occupied, so insertion fails.

A large number of users is unnecessary. Successive passes serving one user can also overlap in preparation.

The important bound is the maximum number of outstanding work items the producer may create before the consumer advances. That bound must account for the actual scheduling path.

The shared event creates a second problem

Queue depth was only one part of the connector's implicit single-pass state.

The main thread reused one CUDA event to mark input readiness. The background copy thread waited on that event before transferring inputs to the CPU. If the producer advanced before the consumer established the wait, another pass could record the same event again.

An event handle needs an ownership rule when work overlaps. The relevant CUDA recording and waiting semantics are specified in the CUDA event documentation.

In the ordering model used for reproduction, the following interleaving produced a cycle:

A records the shared readiness event and queues its notification.
GPU execution reaches the point where A needs the CPU result.
The background thread has not yet established A's event wait.
Preparation of B records the same event again, behind A in the model stream.
The background thread processes A's notification but waits on the newer recording.
That recording cannot complete until A advances, while A cannot advance without the CPU result.

GPU pass A
  -> needs CPU result A
  -> CPU needs copied inputs A
  -> input copy waits for event recorded for B
  -> B's event is behind unfinished GPU pass A

There is also an ordering trap at admission: in the inspected code, the event was recorded before put_nowait().

B could therefore fail to enter the queue after already changing synchronization state that A still needed. Rejecting a work item did not undo that earlier mutation.

This is why the correction needs to cover both admission and resource ownership. A queue can reject an item safely only if the failed attempt has not invalidated resources belonging to accepted work.

What the reproductions showed

The investigation used actual connector methods inside a deterministic CPU model of CUDA operation ordering. It compared the original path, a queue-only change, and a candidate that also separated event ownership and preserved inputs.

Reproduced configuration	Queue insertion failure	Circular wait	Result
Original connector, capacity 1	Yes	Yes	No batch completed in the forced schedule
Original connector, capacity increased to 8	No	Yes	The same dependency cycle remained
Candidate, capacity 8 with per-job events and input snapshots	No	No in the modeled schedule	A and B completed with their own inputs

The input check matters. Completing both batches would not be enough if B's preparation overwrote data still needed by A. The candidate reproduction verified that A consumed A's inputs and B consumed B's inputs.

These are results for deliberately constructed schedules. They establish a defect mechanism and a useful regression case. They do not establish how often the race occurs, prove that this exact interleaving caused the production stall, or qualify the candidate on CUDA hardware.

The inspected production logs directly confirmed the queue exception, stopped generation, later timeout, and automatic restart. They did not include an event trace sufficient to prove the circular wait.

Why the engine learned about failure late

There was another boundary between the worker and the engine.

Successful results and failures shared a response queue. The response-delivery thread could take A's result first and then wait for its GPU work to finish. B's exception could already be queued behind it.

Response queue:
  [A: result that requires completion] [B: failure]
          |
          +-> delivery thread waits here

If A never completes, the thread does not reach B's failure. An exception can appear in worker logs while the engine is still waiting for a response.

A separate CPU reproduction using actual worker methods demonstrated this ordering. While A remained unfinished, no response was delivered and B's failure stayed queued. After the reproduction released A, delivery produced A's success followed by B's failure.

The incident did not capture the delivery thread's stack. This remains an experimentally demonstrated explanation for the delay, rather than a recorded production execution trace.

The general design concern is head-of-line blocking in failure reporting. A failure notification should have a path that can progress without completing the work that may already be stuck. That might require a separate failure channel or another bounded mechanism; changing response order blindly can violate other runtime contracts.

A healthy process can still be unable to generate

During the stall, the health endpoint continued returning HTTP 200. In the inspected path, it checked whether the engine had been declared dead. Before the fatal timeout, that state had not changed.

The endpoint could answer while inference made no progress.

The observed recovery sequence was approximately:

Relative stage	Observation
Initial failure	`queue.Full` recorded by the worker
Shortly afterward	Generation reached zero with two active requests and no waiting users
Following minutes	Health checks remained successful; RPC broadcast warnings appeared
About five minutes after the first exception	Fatal sampling RPC timeout
Shortly afterward	Automatic container restart
About ten minutes after the first exception	API ready again
Roughly twenty seconds later	A recorded generation sample was nonzero

Those intervals describe this incident. They are not expected recovery times for other deployments.

The distinction suggests separate checks for process liveness, engine readiness, and work progress. A progress watchdog should track the completion signal relevant to the active stage. Zero output tokens alone is insufficient: legitimate prefill, idle periods, or long computations can also produce no tokens for a while.

For active work, track how long the oldest outstanding operation has remained in its current stage, whether completions advance, and whether worker failures have reached the engine. Thresholds need to accommodate the supported workload.

Memory and CPU metrics helped narrow the investigation

Before the fatal timeout, available host RAM stayed around 226–233 GiB. KV-cache utilization was approximately 53.4% near the beginning of the stall. These observations did not support memory exhaustion as the explanation for the queue exception.

CPU utilization was around 97–98%. A busy host could delay the background thread and widen the race window, but the defect reproduction did not require that load.

High CPU usage is therefore a possible amplifier, not an established immediate trigger. The investigation did not identify the external event that opened the race window in production.

Keeping that distinction prevents a tempting but incomplete response: add CPU capacity, make the queue larger, and call the incident fixed. Neither change alone repairs ownership of overlapping work.

Make resources follow the work item

A robust correction needs several properties to agree:

Admission: queue capacity and available resource slots cover the supported in-flight work bound, with deliberate overload behavior.
Event ownership: each outstanding job has a distinct event recording, or an exclusive pooled slot that cannot be reused prematurely.
Input lifetime: buffers remain stable until the copy and all consumers that depend on them have completed.
Failure propagation: timeouts and worker errors remain observable even when earlier work cannot finish.
Cleanup: failed admission, cancellation, shutdown, and worker failure release resources according to their actual completion state.

An input snapshot is one approach to preserving data. Another design can use bounded stable slots, provided slot reuse follows completion. Additional copies have a cost, so correctness and performance both need measurement.

Bounded blocking insertion also deserves care. Waiting for space while holding a lock or executing on a thread required by the consumer can introduce another deadlock. Backpressure needs a dependency analysis, not just a timeout argument.

A depth of eight was the candidate's tested setting. It is not a universal recommendation. Derive the capacity from the supported runtime's outstanding-work bound and the resources available to each job.

Public code supports the diagnosis, with a version boundary

A public correction commit addresses PLE offload under async MRV2 scheduling. Its changes include capacity based on concurrent batches, per-batch events, and input copying ordered through the model stream.

That independently supports the concurrency concern. It does not mean the correction exists in every installed runtime.

The associated PLE-offload PR is closed without merging. Maintainer discussion points toward a UVA-based implementation in the main branch. This article concerns the inspected connector path; it should not be read as a claim that every current PLE implementation has the same defect.

For a serving image, check the source actually installed and the effective execution mode. A package name or a public commit link cannot establish that a particular correction is present.

Qualification must finish on a GPU

The candidate passed twelve targeted tests according to the investigation report. Additional CPU checks covered module import and a tensor-based scenario with 24 batches of 8,192 tokens at queue depth eight. CUDA behavior was emulated in those checks.

At the evidence boundary of this write-up, the candidate had not been installed in production or qualified on a real GPU.

The next validation should exercise overlapping passes on isolated hardware, deliberately delay the consumer, and verify both completion and input identity. It should also cover capacity exhaustion, cancellation, worker failure, event-slot reuse, and shutdown. Recovery behavior needs to be checked through the serving API, including failure propagation and progress monitoring.

A passing ordering model is a strong reason to continue qualification. It is not permission to describe an untested CUDA patch as production-proven.

The lasting lesson is that asynchronous execution changes the lifetime of resources. Queues, events, and buffers must belong to outstanding work until that work has safely finished. Health reporting must then distinguish an alive process from an inference pipeline that can still advance.

More engineering notes: Hashnode · DEV · Telegram · GitHub · Hugging Face · Instagram

Originally published on Hashnode.

One Model, Many Roles: Specializing LLM Inference Without Training More Models

Yuri Pocepaev — Tue, 29 Sep 2026 18:30:35 +0000

I ran the same ten tool-selection tasks with four different tool catalogues against one shared model backend: 40 requests in total.

With five tools, the requests used 6,208 input tokens in total. With fifty tools, they used 38,288. Every condition returned the expected tool names and arguments, including correctly making no calls when tools were unnecessary.

That is 83.79% fewer input tokens with no selection errors in either condition on this small test. It is not evidence that fewer tools made the model more accurate. The larger catalogue also passed every task.

This result gives a concrete starting point for a broader architecture: one model, multiple inference roles.

Consider a user asking an assistant to investigate a failing integration, check the documentation, and propose a patch.

One approach puts everything into one request: the conversation history, repository context, search tools, execution tools, memory tools, and instructions for every possible task. The model must decide what matters, which tools to use, and when to stop.

Another approach gives each stage a narrower job. A research stage collects evidence. A coding stage inspects the relevant implementation. A final stage reconciles their findings and answers the user.

All three can use the same model weights.

The architectural unit is an inference profile: the instructions, context, tools, budgets, and output contract surrounding a model call. Specialization can happen at this boundary without training another model or loading another copy of its weights.

The experiment below isolates one part of that design: the tool schemas supplied to a request. The role architecture is the design under discussion; the measured result is a tool-selection pilot, not a benchmark of a complete routed agent system.

One checkpoint, several operating roles

A model checkpoint and an application role describe different things.

The checkpoint provides the model's learned capabilities. An inference profile determines which of those capabilities a request asks it to exercise, what information it receives, and which actions the application permits.

Consider four illustrative profiles:

Profile	Context supplied	Available actions	Expected result
Chat	Relevant conversation and authorized user context	A small set of everyday capabilities	A direct user-facing answer
Reasoning	The problem, constraints, and selected evidence	Only tools needed to resolve remaining uncertainty	An analyzed answer within an explicit budget
Code	Relevant files, interfaces, errors, and test results	Scoped repository operations and sandboxed execution	A patch, diagnosis, or verification report
Mediator	A bounded research question and source constraints	Authorized retrieval and evidence collection	Structured evidence for another stage

These are design examples, not a prescription for four permanent agents.

User request
     |
     v
Routing and authorization
     |
     +--> Chat profile --------+
     +--> Reasoning profile ---+
     +--> Code profile --------+--> Shared inference server
     +--> Mediator profile ----+          |
                                          v
                                   One checkpoint

The routing layer can select one profile or arrange several calls into a workflow. It need not send every request through every role.

Keep three identifiers distinct: the user-facing entry point, the worker's responsibility, and the model endpoint or alias. A “code” entry point might need a research worker before a coding worker. Several workers might call the same endpoint with different request parameters. Naming four aliases does not, by itself, create four different behaviors.

Make the inference profile an executable contract

A useful profile specifies more than a personality prompt:

Component	Decision to make
Instructions	What task should this stage complete, and what is outside its responsibility?
Context policy	Which messages, files, memories, and retrieved evidence belong in this request?
Tool policy	Which schemas are exposed, and which operations may execute?
Generation policy	What sampling settings and supported reasoning controls apply?
Resource budget	How much input, output, elapsed time, and how many iterations are allowed?
Output contract	Should the result be prose, tool calls, a patch, or validated structured data?
Admission policy	How many requests may this role submit, and what happens under overload?

Configuration has to reach the actual call. A declared output limit is ineffective if a worker adapter never passes it to the model request. A tool allowlist is ineffective if another path attaches the global catalogue afterward.

Resolve the effective profile before dispatch and record its version. Include the actual tool set, context size, generation limits, and timeout in request-level diagnostics without logging private content unnecessarily.

Reasoning controls also depend on model and serving support. Raising an output-token limit can allow a longer response; it does not automatically enable a distinct reasoning mode.

A smaller action space is a testable hypothesis

Suppose the model needs to inspect a repository. Alongside repository tools, it receives schemas for messaging, calendars, customer records, web search, and document management.

Every schema adds information to process. Some tools introduce plausible but inappropriate alternatives. Closely related tools can be especially difficult to distinguish: searching a file, searching its commit history, and searching an external knowledge base all sound relevant to “find where this behavior came from.”

The hypothesis is straightforward:

Accuracy may improve because the action space becomes smaller, even when the model weights stay unchanged.

“May” matters. Fewer tools do not guarantee better answers. A router can exclude the necessary tool, or a small set can still contain highly ambiguous choices. Conversely, a large collection of clearly distinct tools may be easier than a small collection with overlapping descriptions.

Tool count is therefore only one variable. Measure semantic similarity, schema length, description quality, and whether the required capabilities survive selection.

MetaTool treats deciding whether to use tools and selecting the appropriate tools as evaluation problems, including single-tool and multi-tool requests. That distinction is useful here: choosing a valid tool is not enough when the correct action is to answer without one.

There is external evidence that selective loading can help. In its advanced tool-use report, Anthropic describes reducing context consumption from roughly 77K to 8.7K tokens with tool search. It also reports improved accuracy on internal MCP evaluations. Those are vendor results for its models and evaluation setup, not measurements of this architecture or a guarantee for another checkpoint.

Static roles and deferred discovery solve different problems

A static role exposes a known subset from the beginning. This is easy to inspect and useful when task boundaries are stable: a repository diagnosis normally needs repository tools, not a calendar API.

Deferred discovery begins with a small discovery interface and loads schemas when needed. It can handle a larger and less predictable catalogue, but introduces another decision: finding the right capability before using it.

AnyTool explores hierarchical API retrieval, a solver operating on selected candidates, and self-reflection when an initial solution is unsuccessful. It is relevant as an example of separating capability discovery from task execution, rather than treating every API definition as permanent prompt content.

A practical combination is a small core tool set plus a controlled discovery path. If a task needs something outside the current role, the worker can request a handoff or additional authorized capability.

That fallback needs to appear in evaluation results. A system that succeeds only after repeatedly expanding back to the full catalogue has different costs from one that completes the task within its original subset.

Tool visibility is not authorization. The execution layer must independently enforce permissions, even when a schema was omitted from the prompt. A model-generated tool name must never bypass those checks.

The mediator should return evidence, not another confident opinion

The mediator makes the separation of responsibilities concrete. Its job is to collect support for another stage's decision.

An illustrative pipeline is:

Research question
  -> Select relevant domains
  -> Retrieve authorized source material
  -> Expose suitable evidence-collection tools
  -> Produce and validate a structured plan
  -> Execute permitted actions
  -> Return evidence, gaps, and source references

Its response can distinguish supported claims, unresolved questions, and conflicting sources. For each claim, preserve the source identifier and enough evidence for the next stage to check it. A concise evidence package should retain a path back to the original material.

Schema-constrained JSON helps enforce the response shape. It does not establish that a claim is true or that its citation supports it. Those checks remain separate.

If retrieval finds nothing useful, the contract should permit “insufficient evidence.” Otherwise, a stage designed to gather evidence may fill the gap with its own unsupported answer.

The final answering stage should also be able to request more evidence. A mediator that aggressively compresses away qualifications can make downstream answers faster and less accurate at the same time.

Run independent stages concurrently

In the integration example, reading the relevant documentation and inspecting the existing implementation can often start independently. Producing the final patch may depend on both. Testing the patch depends on the patch existing.

                  +--> Documentation evidence --+
Request and scope |                              +--> Patch --> Tests --> Answer
                  +--> Repository inspection ---+

This is a dependency graph. Execute independent stages concurrently and preserve ordering where one stage needs another's output. Check side effects too: two operations without a data dependency may still conflict if they write the same resource.

Do not equate asynchronous application code with parallel model execution. Concurrent submissions reach a shared scheduler, which may batch, interleave, or queue them. Network retrieval can overlap with inference, but several generation requests still compete for accelerator capacity.

LLMCompiler studies planned parallel function execution and reports latency improvements of up to 3.7× over ReAct in its evaluated tasks. That supports studying dependency-aware execution; it does not establish the same speedup for several inference roles sharing one backend.

Shared weights also mean shared constraints

One serving process avoids keeping a separate weight copy for every logical role. It does not eliminate the memory and compute costs of their requests.

Active sequences still need state. Long generations occupy capacity. Large prompts consume prefill work. A research burst can increase queueing for interactive chat.

Different service targets therefore require enforcement: per-role concurrency caps, admission control, deadlines, cancellation, bounded fan-out, and overload behavior. A role label alone does not reserve capacity or guarantee an SLA.

The server is also a shared failure domain. If it becomes unavailable, every role depending on it is affected. Hard isolation requirements may justify separate serving instances even when those instances use identical weights.

One checkpoint simplifies weight management, but each profile still needs evaluation. Different prompts, context policies, tools, and budgets produce different behavior. A checkpoint upgrade can regress coding, retrieval, or abstention independently.

What I actually tested

The pilot used ten synthetic requests: five in English and five in Russian. Six requested a single tool, two required no tool, and two requested two independent operations.

The core catalogue exposed five repository-related capabilities: reading current file contents, searching source text, reading commit history, running a named test target, and listing files beneath a directory. All tools were schemas only. No repository was read, no test target executed, and no external service modified by a generated call.

Each task ran once in each condition:

Condition	Schemas exposed	Calls
A	Five core tools	10
B	The core tools plus fifteen repository-related alternatives	10
C	The core tools plus fifteen tools from unrelated domains	10
D	Fifty mixed tools, including the same core set	10

Repository-related alternatives included historical file reads, commit-message search, documentation search, and reading an existing test report. The unrelated tools covered domains such as weather, calendars, music, inventory, and travel. Their descriptions were synthetic and often made the distinction explicit.

For each task, the system instruction and user prompt stayed the same across conditions. The calls used the same endpoint and requested model alias, temperature zero, automatic tool choice, an output limit of 256 tokens, and a fixed seed. The request explicitly disabled thinking through the serving template option. Both condition order and schema order were shuffled deterministically.

This was one run per task and condition, not repeated trials across several seeds. The responses identified the same backend model alias and serving fingerprint; an immutable weight digest was not captured. Endpoint identity is therefore the recorded control, not an independently verified checkpoint revision.

Requests ran sequentially, with pauses of at least fifteen seconds, on a backend already serving other traffic. This avoided a concurrency stress test but did not create a controlled latency environment.

Results: the same selections with less input

Tool catalogue	Expected tool selection	Exact arguments	Median input tokens	Total input tokens
5 core tools	10/10	10/10	619.5	6,208
20, with repository alternatives	10/10	10/10	1,691.5	16,928
20, with cross-domain alternatives	10/10	10/10	1,759.5	17,608
50 mixed tools	10/10	10/10	3,827.5	38,288

Input tokens are the server's reported prompt_tokens for the complete request, including instructions, the user message, and tool definitions. They are not a separate measurement of schema tokens or uncached compute.

Across the same ten tasks, the reduction from the fifty-tool condition to the five-tool condition was:

1 − 6,208 / 38,288 = 83.79% fewer reported input tokens

The fifty-tool condition used about 6.17 times as many input tokens. This is a request-context result, not a measured reduction in GPU time, billed cost, or end-to-end agent cost.

The scorer compared the returned tool names and parsed arguments against the expected calls, ignoring call order for the two independent-operation tasks. It also checked allowed names, required argument keys, and string value types. For no-tool tasks, success meant returning no tool calls; the prose answer was not independently scored.

All forty responses met those checks. None hit the output limit. Multi-tool requests returned both expected calls in the first response; the experiment did not run a tool loop afterward.

The honest conclusion is narrow: the extra schemas were unnecessary for these tasks, and removing them reduced reported input tokens without introducing selection errors in this run.

What the pilot does not establish

The test has a ceiling effect: all conditions scored perfectly. It provides no evidence of improved selection accuracy from a smaller tool space and cannot establish equivalence on a broader workload.

The tasks are short and explicit. “Read the current contents” gives a strong clue against a historical-file tool. Several unrelated descriptions explicitly distinguish themselves from repository operations. Real requests may be ambiguous, incomplete, or require capabilities the router initially excludes.

The small catalogue was selected by the experiment author. No router had to discover it. Consequently, the result excludes routing errors and routing overhead. It also excludes deferred discovery, real execution, grounded final answers, memory-policy changes, context pollution from tool results, and parallel inference.

Recorded wall time includes shared-backend queueing and transport overhead. Requests with more tools sometimes completed sooner because they encountered different traffic. Treating those times as a catalogue-size speed comparison would be misleading. Streaming TTFT and isolated prefill time were not measured.

The next useful experiment should add ambiguous near-neighbor tools, less explicit requests, missing-capability cases, and repeated shuffled trials. A real router and a deferred-discovery path should then be evaluated separately, counting their failures, fallback expansion, tokens, and latency.

Context policy and scheduling need their own comparisons. Changing tools, prompts, memory, reasoning limits, and concurrency together can measure an overall system difference, but cannot identify which change caused it.

Measure successful work, including the extra stages

Use several levels of correctness:

Selection: Did the model choose an acceptable tool or correctly abstain?
Arguments: Did the call satisfy both the schema and the task's actual constraints?
Execution: Did the authorized operation succeed against a controlled fixture?
Answer: Did the final response solve the task using the returned evidence?

The Berkeley Function Calling evaluation distinguishes multiple candidate functions, parallel calls, relevance detection, and executable checks. These categories help prevent a single tool-name score from standing in for complete task success.

Also record tool-schema tokens, total input and output tokens across every stage, iteration counts, retries, time to first token, queue time, total latency, finish reasons, and failure rates. Report latency distributions and uncertainty on accuracy differences, not just one average from a small run.

For staged systems, distinguish first visible text from the time to a useful, completed answer. An early “working on it” message should not make a slow workflow look fast.

Measure cold and warm cache behavior separately. A shorter prompt can reduce uncached prefill work, but role-specific prefixes may reduce sharing between requests. Prefix-cache reuse depends on the actual serving implementation and prompt structure; it is not an automatic benefit of specialization.

Use isolated synthetic fixtures for operations with side effects. Store the corpus, schemas, ordering seeds, effective profiles, scoring rules, and raw responses with the results. Request-level usage must be collected from actual calls; placeholder zero counters are missing telemetry, not free inference.

The boundary is the design

The pattern is useful when a capable model serves tasks with different information needs, action sets, and latency expectations. It is less attractive when routing adds more overhead than it removes, or when most tasks repeatedly need the same broad context and tools.

The pilot supports a practical reason to start with a focused tool set: on these tasks, the larger catalogue consumed substantially more request context and produced no better selections. It does not settle whether role routing improves overall accuracy, cost, or latency.

A good role has a clear responsibility, sufficient authorized context, an appropriate action set, and a checkable result. That makes specialization something the application can inspect and test, instead of a label attached to another prompt.

More engineering notes: Hashnode · DEV · GitHub · Hugging Face · Telegram · Instagram

Originally published on Hashnode.

Prevent Context Leakage in Multi-Tenant LLM Systems

Yuri Pocepaev — Tue, 29 Sep 2026 01:21:55 +0000

Two users can ask the same question and be entitled to completely different answers.

“Summarize the renewal terms” might refer to a public template, a private contract, or a document shared with only one team. The words alone do not tell you which answer is safe to return.

In an LLM application, isolation must follow the information through conversation history, retrieval, agent memory, tool calls, and cached responses. Authentication establishes the caller. Every component that reads or returns context must enforce what that caller is allowed to access.

Why context isolation matters

Context changes what a model says and what an agent may decide to do. Mixing it between users creates several kinds of failure:

Private information reaches the wrong person. A summary can disclose contract terms, customer records, or internal decisions without quoting the original document. Generated answers need protection just as their source documents do.
An answer is plausible but belongs to someone else. Another team's pricing, constraints, or preferences can quietly influence a recommendation. The user may see a confident answer with no obvious sign of contamination.
An agent acts on the wrong premise. Contaminated context can lead it to propose or attempt an inappropriate message, update, or workflow. Tool-level authorization remains essential, but it cannot guarantee that an otherwise permitted action makes sense for the user's actual request.
Revoked access survives in derived data. Removing access to a document is incomplete if summaries, memories, or cached answers still expose it. Permissions must govern derived content as well as the original record.

These failures do not require a sophisticated prompt injection. Two ordinary users asking the same question can be enough to reveal a missing boundary. Asking the model to “never reveal another user's data” is not an access-control mechanism: unauthorized context should never enter its input in the first place.

Isolation also makes the system easier to debug. When every response has an identifiable authorized scope and context revision, an unexpected answer can be traced to the information it was allowed to use. Without that provenance, a context-selection bug can look like a model hallucination.

Build a trusted request context

Resolve identity from verified credentials, then construct an immutable request context on the server:

# Illustrative structure, not a complete authentication implementation.
@dataclass(frozen=True)
class RequestContext:
    tenant_id: str
    principal_id: str
    permission_revision: str

Validate the credential's signature, issuer, audience, and expiration as applicable. Resolve current tenant membership and permissions. A tenant selected in a URL or header is a requested scope to authorize, not proof of membership.

Client-supplied fields such as user, owner_id, or tenant_id must not override that context. Pass it explicitly to storage, retrieval, tools, and cache operations. Avoid mutable process-wide variables for the current user: concurrent requests must never share that state.

Delegated agent credentials should bind the principal, tenant, audience, permitted operations, and expiration. Downstream services must verify them rather than trust identity fields supplied by the agent.

Authorize resources before building the prompt

A conversation ID identifies an object; it does not authorize access to it.

Read a conversation through its tenant and access policy. Enforce the same policy when listing messages, retrieving attachments, exporting history, or opening a streaming endpoint. Returning 404 for inaccessible objects can limit information about their existence, but the status code is not the authorization check.

For RAG, apply tenant and document permissions before retrieved content reaches a reranker, model, or client. Include the filters in the search where supported, and verify the returned resources against current authorization. Persisting an owner_id field without enforcing it on reads provides no isolation.

Define memory scopes deliberately:

Conversation memory belongs to an authorized conversation.
Personal memory may follow a user across conversations if that is an explicit product behavior.
Team memory follows membership and document permissions.
Public knowledge is explicitly approved for shared access.

Do not collapse these into one global session. Server-controlled session identifiers should include the relevant tenant, principal, and conversation scope. Background jobs and tool callbacks need the same context as the request that created them. OWASP covers these boundaries across data, sessions, caches, and background processing in its multi-tenant security guidance.

Treat a cached answer as protected data

A response cache can bypass the checks in a retrieval pipeline. That makes the cache an authorization boundary of its own.

Imagine Alice asks a question about her private contract. The answer is cached. Bob asks an identical question about a different contract. If the cache considers only prompt similarity, Alice's answer can become eligible for Bob.

A stricter similarity threshold does not solve this. Identical prompts can depend on different private context.

Use this rule:

Establish which entries the caller may reuse before comparing semantic similarity.

A conservative default is to scope private answers to both the tenant and principal. Broader sharing requires an explicit policy and equivalent access to the information behind the answer.

For context-dependent responses, reuse also needs to account for:

Dimension	Why it matters
Tenant and principal or authorized sharing scope	Defines who may reuse the response
Conversation and context revision	Distinguishes histories and subsequent turns
Permission revision	Prevents reuse after access changes
Source revisions	Distinguishes updated or removed documents
Model, instructions, tools, and relevant generation settings	Distinguishes how the answer was produced

A conversation ID alone is insufficient: its history changes. A source ID alone is insufficient: its contents and permissions change. Use canonical, unambiguous serialization when computing fingerprints, and never use a fingerprint as a substitute for access control.

If the cache lookup happens before retrieval and cannot establish the relevant source context, bypass it for that request or move it to a point where those dependencies are known. Do not label unknown context as empty context.

Make unsafe cache calls difficult to express

Prefer a cache API that requires an authenticated scope over one where omitting an owner silently selects a shared cache.

# Pseudocode: scope is derived by trusted server-side code.
scope = build_private_cache_scope(request_context, authorized_context)

if scope is None:
    return await generate_with_authorized_context()

candidate = await cache.find(query, required_scope=scope)
if candidate and await may_reuse(candidate, request_context, scope):
    return candidate.answer

answer = await generate_with_authorized_context()
await cache.store(query, answer, required_scope=scope)
return answer

Use the same scope construction for reads and writes. Enforce mandatory filters in the storage adapter, not just at individual call sites. Redis's semantic-cache documentation describes metadata filters for user and conversation scoping; the application must supply the correct values on every path.

Missing or invalid authorization should reject the request. An unavailable cache can fall back to normal authorized generation. Those are different failure modes: neither should trigger a search across all users.

Keep deliberately public caching separate. “Plain text,” “no tools,” and “no retrieved documents” are not evidence that an answer is public. System instructions, prior messages, and user preferences can all affect it.

When introducing a stricter scope, make older entries ineligible unless their provenance can be established. A cache schema version or fresh namespace can separate them. TTL limits age; it does not prove permission.

Test through the real request boundary

A unit test that passes two different owner IDs to a cache verifies that the cache can isolate them. It does not verify that the HTTP handler supplies those IDs.

Use synthetic identities and private markers in an isolated integration environment:

Authenticate as Alice and generate an answer containing a marker from her private context.
Confirm that the answer entered the intended cache partition.
Authenticate as Bob and send the identical final prompt, then a paraphrase.
Assert that Alice's cache entry is ineligible and that Bob receives no Alice marker.
Repeat as Alice with unchanged authorized context and confirm that legitimate reuse still works.

Inspect cache provenance as well as response text. An unrelated output does not prove that an unauthorized entry was never selected.

Extend the matrix to different conversations, different tenants, revoked permissions, updated documents, missing identity, forged client identity fields, and concurrent requests. Test alternate routes: streaming, non-streaming, tools, and background jobs can have different implementations.

Keep these checks deterministic where possible. A stub generator can return the synthetic marker from its authorized input, making an isolation failure observable without depending on model behavior.

Audit the remaining copies of context

Application response caches, agent memory, and inference KV caches have different lifecycles. Review each separately. A shared model server should keep request sequences isolated; any cross-request prefix reuse needs its own assessment of the serving implementation and isolation controls.

Also review logs, traces, exports, and debugging tools. Avoid recording raw private prompts merely to diagnose cache hits. Prefer access-controlled metadata such as request IDs, scope identifiers, cache decisions, and policy versions, with appropriate retention.

The practical standard is consistent: a fast path must enforce the same access policy as the full path. Optimize only after the system can explain why this caller is allowed to receive this context.

More engineering notes: Hashnode · DEV · GitHub · Hugging Face · Telegram · Instagram

Originally published on Hashnode.

Discuss this article on Telegram.

Smaller Context, Recoverable History: Inside an Agent Memory Handoff

Yuri Pocepaev — Mon, 28 Sep 2026 02:24:34 +0000

A memory handoff replaces a large conversation window while keeping retained source segments available for recall.

The goal is to reduce how much history the LLM has to carry in its active context, while preserving the information it can recover when needed.

That goal is different from shrinking a database. Keeping exact source text outside the prompt can be the right trade-off: the model stops processing the entire history on every turn, but retains a path back to the details.

I checked the implementation, reran its targeted tests, and measured the replacement handoffs stored by the system. Across 10,241 completed runs, the selected source windows contained 1,137,387,720 characters. Their handoffs contained 15,375,395 characters in total—a 98.65% reduction in the replaced portion of context, measured in characters.

I also recomputed the stored text hashes for 416,472 retained segments. None lacked exact source text, none had an exact or reduced hash mismatch, and none were missing a projection timestamp.

I then ran a controlled synthetic comparison with a live answering model and real semantic embeddings. Full history and handoff-plus-recall both answered all 12 questions correctly. Handoff alone answered only the one question whose correct response was UNKNOWN. The recalled-context arm used substantially fewer input tokens, including the evidence retrieved for each answer.

This is a small controlled result, not a claim of general lossless agent memory. The retrieval calls were made by the test harness, not chosen autonomously by an agent.

KEEP does not mean “keep in the prompt”

This is the most important detail in the implementation.

The pipeline routes segments through three decisions:

Route	What happens to the segment
KEEP	Preserve exact text in external storage.
COMPRESS	Store an accepted reduced representation alongside the exact source.
DROP	Omit the segment from retained content.

All three decisions belong to the processing of a selected conversation window. Once its retained material has been durably handed off, that window can leave the active context—including its KEEP segments.

The replacement is a compact handoff containing a short navigation summary, a manifest identifier, selected content hashes, and instructions for retrieving the omitted material.

Consequently, a run with nothing but KEEP decisions can still substantially reduce active context. It retains information externally instead of repeatedly presenting all of it to the LLM.

What the operational measurements show

I queried aggregate lengths from completed runs without exporting user message text:

Measurement	Result
Completed runs	10,241
Selected source-window characters, summed	1,137,387,720
Replacement handoff characters, summed	15,375,395
Character reduction across those windows	98.6482%
Smallest handoff	584 characters
Largest handoff	1,897 characters

The percentage is calculated from the summed lengths. It is not an average of per-run percentages.

It also applies only to the replaced windows. System instructions, protected recent messages, tool definitions, and subsequent recall results still occupy context. The measurement is neither a whole-prompt reduction nor a token or inference-cost benchmark. Repeated runs can contain related material, so the source total is not a count of unique information.

The earlier segment counters showed 416,459 KEEP decisions, 13 COMPRESS decisions, and no DROP decisions. The reduced storage representation was only 818 characters shorter than its source.

That tiny difference answers a different question. It measures reduction within the externalized content, not the reduction from replacing the source window with its handoff. The large active-context change comes from externalization.

A small run that makes the distinction visible

I also ran the planner inside the deployed API environment on a synthetic window of 24 tool messages. Each contained an exact endpoint, port, command, and repeated diagnostic prose.

Measurement	Result
Messages / segments	24 / 24
Source characters	56,443
Handoff characters	1,313
Reduction in the selected window	97.67%
KEEP / COMPRESS / DROP	24 / 0 / 0
Exact segment hashes valid	24 / 24
Planner elapsed time	3.242 seconds

Every segment was retained verbatim, yet the resulting handoff was much smaller than the selected window.

This was a planner smoke test, without database writes. Protected tool content bypassed classification; the default summary and NLI paths were invoked. The recorded output does not distinguish an accepted generated summary from its safe fallback. The timing is one observation, not a latency distribution, and the test did not exercise semantic retrieval or downstream question answering.

The path out of active context

The agent selects a message-safe middle window. Secrets are redacted before persistence, and messages are divided into role-aware segments of at most 3,800 characters.

Technical content receives conservative handling. Tool, system, and developer segments are protected, as are recognized patterns such as code fences, commands, paths, hashes, and error evidence. These protections resolve to KEEP before model classification.

Other segments go through GLiClass routing. Weak decisions or model failures fall back to KEEP. COMPRESS candidates pass through LLMLingua, then an NLI gate. Empty output, output that does not shorten the segment, or a verdict other than entailment preserves the exact source instead.

The short handoff summary is separately checked. If its generation or validation fails, the planner uses a deterministic excerpt. This summary is a navigation aid: it is not intended to contain every fact from the discarded window.

“Exact source” here means the retained text after mandatory secret redaction. It does not mean an unredacted archive.

How the details come back

PostgreSQL stores authoritative exact/reduced pairs and their hashes. Qdrant holds the reduced retrieval representation and integrity metadata.

The agent has three recall paths:

Semantic query: search for relevant segments, with optional session scoping.
Manifest lookup: enumerate retained pieces from a particular compaction run.
Content-hash lookup: recover a specific exact segment.

Search candidates are checked against PostgreSQL before being returned. Reranking is used when available. The exact source is authoritative when it differs from its reduced representation.

This design lets the agent start with a small handoff and request detail as needed. It also makes retrieval behavior central to the outcome. Preserving a command byte-for-byte is valuable, but the agent still needs to find that command at the appropriate moment.

Removing the window has a durability condition

The source window is removed only after a successful, valid handoff response.

PostgreSQL is the authority, but successful persistence alone is insufficient for the API acknowledgment. Projection must also complete. A projection failure returns a retryable error, and the original context remains available to the agent. Retrying the same operation repairs outstanding projection work; a completed duplicate returns the full stored handoff.

The integration also rejects malformed responses, empty handoffs, and invalid manifests. It disables an earlier tool-output pruning pass so that a failed compaction does not leave the conversation partially truncated.

This boundary protects against losing the source window during a failed handoff. It does not prove that the downstream agent will answer every question as well as it would with full history.

Checks I reran

The targeted suite finished with 28 passed in 7.60 seconds. It covered routing and fallback behavior, persistence/projection ordering, retry behavior, stale retrieval-candidate rejection, API behavior, observability, and the agent integration. The alert configuration also passed validation with five rules found.

The suite uses test doubles for models and storage, and the agent failure test uses a stand-in for the upstream compressor. Its results verify the tested contracts; they are not a live full-stack benchmark.

Separately, the database integrity scan produced:

Check	Result
Retained segments checked	416,472
Missing exact text	0
Recomputed exact SHA-256 mismatches	0
Recomputed reduced SHA-256 mismatches	0
Missing projection timestamp	0

This scan validates stored text against stored hashes. It does not compare the database with an independent original transcript, and a projection timestamp alone does not verify every live vector-index payload.

Testing answers after recall

The synthetic history contained 49 segments: operational facts, an explicit later correction, and 36 distracting records. All segments had the tool role and therefore took the protected KEEP route. This deliberately isolates externalization and recall from classifier or lossy-reduction quality.

The planner and storage/recall functions matched the inspected implementation. I used connection-local temporary PostgreSQL tables and a local Qdrant index, with real embeddings, the configured reranking path, and the same live answering-model endpoint for all three arms. The test wrote no synthetic records into the production memory tables or collections.

For each question, the harness compared full history, the handoff alone, and the handoff plus exact records returned by semantic top-5 retrieval. It supplied no expected answers to the model or retriever. The query was the question itself. Temperature was zero, and scoring compared the returned value with a predetermined answer after basic whitespace, case, and terminal-punctuation normalization.

Input available to the answering model	Correct answers
Full history	12 / 12
Handoff only	1 / 12
Handoff + semantic recall, top 5	12 / 12

The endpoint reported 96,819 input tokens across the 12 full-history calls and 22,296 across the 12 recall calls: 76.97% fewer input tokens, with retrieved evidence included. Per question, the averages were 8,068.25 and 1,858 tokens. These are reported request usage counts, not independently tokenized measurements or a cost benchmark; embedding, reranking, compaction, and output-generation costs are not included in that percentage.

The questions covered an IP address, port, exact command, full SHA-256, responsible person, database decision, prohibited operation, Russian-language value, superseding revision, staging-versus-production distinction, prerequisite, and an unknown password. The designated evidence segment appeared in the top five for all 12 questions. For the revision question, the designated evidence was the newer record.

All 49 segments were also recovered through exact-hash lookup with matching text, and manifest lookup returned all 49 rows. The selected history was 49,897 characters; its handoff was 1,329 characters.

This is an intentionally limited check. It does not exercise COMPRESS or DROP quality, a production-sized vector collection, authentication, concurrent retrieval, multiple successive compactions, or an autonomous decision to call the recall tool. The unknown-answer case also does not test resistance to plausible contradictory evidence. The live response reports a serving alias rather than a pinned model-weight revision.

What “without losing memory quality” still needs to demonstrate

Preserving memory quality is the objective. The next comparison must measure it directly:

Give an agent the full history and record its answers.
Replace the eligible window with the handoff and enable recall.
Ask the same questions, including exact values, earlier decisions, corrections, and conflicting updates.
Compare answer accuracy, retrieved evidence, latency, and tokens consumed—including the recall results brought back into context.

The context measurement should include both the immediate reduction and the cost of subsequent retrieval. A system that starts with a tiny handoff but repeatedly reloads the entire history may save much less over a complete task.

A verbatim-externalization baseline is useful here: it separates the benefit of moving history out of context from the effect of reducing the retrieval representation.

The evidence so far supports a concrete claim: large selected conversation windows can be replaced with compact handoffs while retaining integrity-checked exact segments for recall; on this 12-question synthetic check, recall preserved answer accuracy. Broader unchanged answer quality remains a benchmark question, not an inference from the storage checks or this small corpus.

The system is designed around a smaller working context and recoverable history. That is the trade-off worth measuring.

Reproduce the comparison

The public benchmark includes the synthetic corpus, expected answers, Python runner, tests, and recorded responses. The README explains offline verification and live endpoint configuration.

The original recorded run is separate from the portable live run. The portable version uses SQLite, exhaustive cosine search without reranking, and a deterministic handoff. It also scored 12/12 with full history and recall, using 96,819 versus 14,208 input tokens (85.33% fewer). Its handoff-only score was 2/12: the excerpt exposes the first answer, and UNKNOWN is also correct. These are different protocol variants; their token totals should not be mixed.

The repository reproduces the comparison procedure, not the private production stack. Weight revisions were not captured, so identical outputs are not guaranteed. Operational aggregates and deployment tests above are separate observations, not reproduced by the public fixture.

Elsewhere

Telegram: discussion and updates · GitHub · Hugging Face · DEV · Instagram

Original on Hashnode · Discuss this article

How I Took GLiClass FP8 from 59 ms to 16 ms on an RTX 4050

Yuri Pocepaev — Sun, 27 Sep 2026 14:40:04 +0000

Multilingual zero-shot classification on a 6 GB laptop GPU, with native FP8, Triton, CUDA Graphs, and reproducible measurements.

I quantized Knowledgator’s GLiClass Multilang Ultra into a custom FP8 W8A8 checkpoint. Then I wanted to find out how much of that compression could translate into faster local inference.

The weights already took up a third less disk space. The first native FP8 implementation, however, needed about 59 ms per request. The original BF16 model took roughly 24 ms. Reducing precision had made the artifact smaller, but the execution path still needed work.

After fusing auxiliary operations with Triton and adding CUDA Graphs, median request latency reached 16.10 ms. BF16 with the same graph wrapper measured 23.79 ms. On this particular workload, the optimized FP8 path was 1.48× faster, with roughly a third less memory allocated to tensors.

Here is what the model does, what I changed, and what these measurements actually establish. The weights, implementation, and benchmark report are on Hugging Face.

What GLiClass does

GLiClass Multilang Ultra, developed by Knowledgator, classifies text against labels supplied at inference time. You can provide categories such as “delivery,” “refund,” “payment,” and “other” without training a new classification head for that particular list.

For example:

text = "The courier has postponed my delivery for the second day."
labels = ["delivery", "payment", "refund", "other"]

The useful output is a class and its score. This makes classification a practical component before retrieval, application logic, or a more expensive model call.

Potential applications include support-ticket routing, document categorization, intent detection, and filtering text collections. Those are examples of where this interface can help; this experiment did not validate every one of those tasks.

I used the multilingual Ultra model. My quality evaluation covers English and Russian topic classification. It does not establish performance across every language supported by the upstream model or readiness for a particular production workflow.

What is inside the FP8 checkpoint?

I started from the original GLiClass model and built the FP8 W8A8 quantization used here. A separate phase then audited the tensors, measured quality, and optimized its inference path with Triton and CUDA Graphs. Knowledgator remains the developer of the original model; the quantization and local inference work are my derivative contribution.

The checkpoint quantizes 168 matrices across 24 mT5 encoder blocks: the attention projections q, k, v, and o, plus the feed-forward projections wi_0, wi_1, and wo.

Weights use FP8 E4M3 with a separate scale for each output channel. Activations are quantized dynamically for each token. Embeddings, normalization layers, and classification components remain in BF16.

W8A8 therefore describes the selected linear projections. It does not mean every tensor in the model uses eight bits.

Weight file	Size
Original BF16	3,416,522,340 bytes
FP8 W8A8	2,259,902,516 bytes

That is a 33.85% reduction, or 1.51× compression. The shared embedding matrix alone accounts for approximately 1.02 GB of the remaining size.

I also checked the unquantized tensors against a pinned snapshot of the original model. All 80 original tensors expected to remain unchanged matched byte for byte. The exact upstream revision used during the historical quantization was not recorded, so the report states that explicitly. The BF16 snapshot used for the comparison is pinned.

Why FP8 was initially slower

The storage format does not tell you which operations the GPU will execute.

One loading configuration reconstructed the weights in BF16 while retaining activation-quantization emulation. That path took around 264 ms per request. It was useful as a diagnostic, but it was not a native FP8 performance measurement.

The next adapter used actual FP8 matrix multiplication through torch._scaled_mm and cuBLAS. A separate profiler pass recorded 168 FP8 GEMM kernels for sm89, one for each replaced projection. Yet the complete request still took 59.23 ms.

The trace showed an obvious place to investigate: 3,269 GPU kernel executions for one request. Around the matrix multiplications, separate operations computed maxima, calculated scales, converted types, padded rows, and processed outputs.

At batch size 1, the cost of running that sequence matters. Matrix multiplication is only part of request latency. The profile suggested reducing auxiliary operations and launch overhead; the subsequent measurements tested those changes at the request level.

The changes that made a difference

Fuse activation quantization

For each token, the adapter needs an absolute maximum, a scale, and an FP8 representation of the activations. The original implementation expressed these as separate PyTorch operations.

I moved them into one Triton kernel. It performs the reduction, calculates the scale, converts values to E4M3, and fills the additional rows needed for matrix alignment.

Fuse the output scaling

The native FP8 GEMM produces an FP32 result. The adapter then applies the activation scale for each token and the weight scale for each output channel before converting back to BF16.

That sequence became a second Triton kernel. The stored FP8 weights are used directly, without another round of weight quantization. GEMM continues to use use_fast_accum=False.

Together, these changes reduced median latency to 37.97 ms.

Replay the encoder with CUDA Graphs

Next, I added CUDA Graphs around the encoder. A graph replays a captured sequence of GPU work, reducing the overhead of issuing its operations separately from Python.

Text lengths vary, so the adapter uses four buckets: 64, 128, 192, and 256 tokens. Inputs are padded to the appropriate bucket, and the extra positions are masked. Before classification, the encoder output is sliced back to the original token count.

Every request copies fresh tokens and a fresh mask into the graph's input buffers. Predictions are not cached. The output is cloned before it is returned to the caller.

One compatibility issue needed attention: in the tested Transformers version, attention-mask construction copied a scalar from CPU to GPU during graph capture. Preparing the mask outside the captured region resolved it.

With these changes, median latency reached 16.10 ms. The profiled request contained 1,425 GPU kernel executions, down from 3,269, while retaining all 168 native FP8 GEMMs.

How I measured it

The machine was an NVIDIA RTX 4050 Laptop with 6 GB VRAM, running Windows. The environment used Python 3.12.9, PyTorch 2.11.0 with CUDA 12.8, GLiClass 0.1.20, Transformers 5.14.1, and Triton Windows 3.6.0.post26.

Latency was measured on one fixed AG News example with four candidate labels, at batch size 1. After warmup and quality evaluation, I ran 50 additional requests. Each request was synchronized with CUDA before and after timing. The measurement includes tokenization and postprocessing.

Model loading, Triton compilation, and graph preparation are excluded from these steady-state timings. All four graph buckets were prepared in advance. FP8 graph preparation took 2.18 seconds in this run; that is not the complete cold-start time.

Runtime	Median	p95	Peak allocated tensor memory
BF16, eager	24.18 ms	29.67 ms	3.219 GiB
BF16 + CUDA Graphs	23.79 ms	24.45 ms	3.238 GiB
FP8, initial native adapter	59.23 ms	66.36 ms	2.145 GiB
FP8 + Triton	37.97 ms	46.62 ms	2.144 GiB
FP8 + Triton + CUDA Graphs	16.10 ms	16.44 ms	2.166 GiB

For the main comparison, I applied the same CUDA Graph wrapper to the original BF16 model. Optimized FP8 was 1.48× faster than that control. The 3.68× improvement compares the optimized implementation with my initial native FP8 adapter; it describes progress in the adapter implementation.

The memory column measures allocated PyTorch tensors. It is different from total device memory shown by nvidia-smi, which also reflects allocator reservations and other overhead. The current loader temporarily reconstructs weights in BF16 before replacing the projections, so 2.17 GiB is not a loading-memory requirement.

Laptop clocks and thermals were not locked. These are measurements of this configuration, not performance confidence intervals across independent sessions. I did not measure sustained throughput at larger batch sizes.

What happened to quality?

The paired evaluation used 664 identical examples: a seeded subset of 256 AG News test examples and the full English and Russian SIB-200 test splits, with 204 examples each. SIB-200 candidate labels were in English for both languages.

Dataset revisions, example indices, and hashes are recorded in the repository. The original and quantized tokenizers produced identical token IDs. No example was truncated at the 256-token limit.

Dataset	BF16 macro-F1	Optimized FP8 macro-F1	Difference
AG News	79.08%	79.49%	+0.41 pp
SIB-200 English	84.57%	84.04%	−0.53 pp
SIB-200 Russian	84.09%	83.42%	−0.67 pp

Optimized FP8 changed 5 of 664 top-1 predictions relative to BF16, for 99.25% agreement. Agreement measures whether the two variants choose the same label; it is not classification accuracy. BF16 with and without the graph wrapper produced identical top-1 predictions on all 664 examples.

I would not interpret the small positive AG News difference as an improvement in model quality. The other two datasets show declines, and this evaluation is too narrow to establish general quality preservation. A support-ticket router, for example, would need its own evaluation on representative tickets and candidate labels.

The single-label pipeline retained only the winning label and its probability. Full score distributions were not saved, so this report makes no claims about changes across all logits or probability calibration.

Try it locally

The Hugging Face repository contains the weights, tokenizer, dependencies, adapter, and an inference.py entry point.

After downloading the repository and installing the environment described in the model card, run:

python inference.py --text "NASA launched a spacecraft to study Mars." --labels space politics sports business

Or use the Python API:

import torch
from inference import load_pipeline

classifier = load_pipeline()

with torch.inference_mode():
    result = classifier(
        "NASA launched a spacecraft to study Mars.",
        ["space", "politics", "sports", "business"],
        batch_size=1,
        threshold=0.0,
    )

print(result)

The tested adapter supports sequential inference, batch size 1, and at most 256 input tokens, including labels and prompt formatting. The tokenizer truncates longer inputs. Concurrent requests to one instance require synchronization or separate runners.

This is currently a specialized inference path for the measured workload. Other GPUs, Linux execution, and installation in a fresh environment were not independently validated for this release. The measured configuration is Windows on an RTX 4050 Laptop.

What this experiment taught me

A quantized checkpoint has at least three separate properties: its size on disk, its runtime memory use, and the speed of a particular implementation. Here, the smaller FP8 file initially produced a slower request. The latency improvement came from optimizing the operations around GEMM and the way the encoder's GPU work was launched.

The result is a reproducible starting point: 16.10 ms on the selected batch-1 request, approximately 2.17 GiB of allocated tensor memory, and measured quality changes. Larger batches, longer inputs, and real application workloads are the next useful experiments. Their results need to be measured separately.

The main takeaway is simple: quantization gives you a smaller representation; making it faster is a separate engineering problem.

The model, source code, and report are available under Apache-2.0. Knowledgator developed the original model; this release provides an independent quantized representation, inference implementation, and evaluation.

About the author

I’m Yuri Pocepaev, a software engineer and lead developer at Neuroprem. This article is part of my public engineering log.

Elsewhere

Instagram · Telegram

Building an NVFP4 KV Cache for a Hybrid Qwen Model

Yuri Pocepaev — Sun, 27 Sep 2026 14:33:27 +0000

Packing K/V into 576 bytes per token, fixing vLLM's hybrid cache planner, and measuring the result on an RTX PRO 6000 Blackwell.

I spent a fair amount of time getting NVFP4 KV storage working in my Qwen3.8-Flash-Next serving stack. The work covered the quantizer, a packed-cache writer, a sparse attention reader, and the vLLM integration needed to make them usable together.

The most useful result from the latest comparison was capacity: at almost the same KV-cache memory budget, the server reported a pool of 439,958 tokens with NVFP4, compared with 189,904 with BF16. That is about 2.32x as many tokens in the planner's pool.

Speed was less straightforward. BF16 was faster in the sequential test; NVFP4 finished the four-request batch faster. The small quality check produced the same aggregate classification score, but two answers changed. Those details matter just as much as the memory result.

This is about quantizing the live attention cache. Both sides of the comparison use the same model checkpoint. I am not comparing newly quantized weights against an original BF16 model.

The interesting integration failure happened before inference even started. My kernels used 576 bytes per token per full-attention layer, while an earlier planning step still accounted for 1,024. In a hybrid model, that disagreement was enough to prevent the server from allocating its cache.

The code and evaluation artifacts are now public: nvfp4-qsa on GitHub.

Why the hybrid architecture matters

The inspected configuration has 48 layers: three linear-attention layers followed by one full-attention layer, repeated twelve times. That gives 36 linear-attention layers and 12 full-attention layers. The latter use QSA sparse selection to choose positions for attention reads.

The main QSA K/V arrays store information for individual tokens. The linear-attention path has recurrent state. This implementation compresses the former; recurrent state remains float32 in the observed launch configuration. Indexer state is another allocation with its own accounting.

That distinction matters before discussing memory savings. A smaller K/V representation does not shrink the model weights, draft model, graph buffers or recurrent state by the same factor.

What changed on the server

I ran both cache modes in an isolated container during a maintenance window on the RTX PRO 6000 Blackwell. Production traffic was not routed to the test endpoint. Other services remained active, so this is a controlled comparison on a shared host.

The checkpoint, MTP=3, maximum length of 169,984, twelve-sequence limit, 8,192-token scheduler budget, memory utilization 0.94 and float32 recurrent state were held constant. Compilation and CUDA graphs were enabled. The changed parameter was the cache format. NVFP4 ran first, followed by BF16; the order was not randomized or repeated.

Cache	Available KV budget	Reported planner token pool
BF16	5.85 GiB	189,904
NVFP4	5.84 GiB	439,958

That is about 2.32 times the reported token pool at nearly the same KV budget. It is smaller than the 3.56x main-K/V format ratio because the hybrid model has other state and overhead. It is an allocator observation, not a sustained full-window concurrency test.

Cache	Concurrency	Median TTFT	Aggregate E2E output
BF16	1	165.6 ms	123.48 tok/s
NVFP4	1	160.8 ms	88.91 tok/s
BF16	4	1070.9 ms	91.88 tok/s
NVFP4	4	375.8 ms	129.02 tok/s

Each condition contains twelve requests for 128 output tokens and one measured batch. Throughput uses actual token usage and complete batch wall time. Two single-request warmups were performed; the four-way condition was not separately warmed. Unique user-message prefixes prevent identical requests, while shared chat-template prefixes can still be cached.

BF16 was faster for the single-request stream in this run. NVFP4's four-way batch finished faster, but startup delays are visible and there are no independent repeated batches. Picking only the four-way result would overstate the performance evidence. The storage and planner-capacity benefit is the clearest result here.

Did the answers change?

For quality I selected sixteen examples each from pinned AG News, SIB-200 English and SIB-200 Russian test files before evaluation. Both modes received the same questions and label order. Answers were scored by exact stripped label matching.

Task	BF16	NVFP4
AG News	13/16	13/16
SIB-200 EN	11/16	12/16
SIB-200 RU	13/16	12/16
Synthetic code retrieval	9/9	9/9

Both modes scored 37/48 on classification, but their predictions were not identical. NVFP4 corrected one English example and changed the Russian version of the same example from correct to incorrect. Across all tasks, 55 of 57 answers matched. Equal aggregate accuracy is not proof that quality is unchanged.

The nine retrieval questions placed one code at three positions in three text sizes. The maximum measured input was 9,280 tokens. This is a small regression check, not 120K retrieval validation or a full-dataset benchmark. Raw answers, token usage and dataset revisions are included. The chart's Wilson intervals expose the uncertainty from small subsets; translated examples also introduce dependence between the English and Russian observations.

How 2,048 bytes became 576

Each K or V vector has 256 elements. FP4 packs two elements into each byte, giving 128 data bytes. Groups of 16 elements each have one FP8 E4M3 scale, adding another 16 bytes. With two KV heads and both K and V, the total is:

(256 / 2 + 256 / 16) x 2 x 2 = 576 bytes/token/layer

Main K/V representation	Bytes per token per layer
BF16	2,048
FP8	1,024
NVFP4, including group scales	576

At an equal main-K/V byte budget, that means 1.78 times as many tokens as FP8, or 3.56 times as many as BF16. Equivalently, the main-K/V allocation is 43.75% smaller than FP8. These are format calculations, not measured gains for the entire server.

The FP4 values use E2M1 with magnitudes 0, 0.5, 1, 1.5, 2, 3, 4, 6 and a sign bit. Element 2j goes in the low nibble; element 2j+1 goes in the high nibble. Decoding applies fp4 * group_scale / global_scale.

The global scale must remain stable while older cache entries are live. Changing it when a new token arrives would change the interpretation of existing bytes unless the old scale were retained. The integrated cache initializes both K and V global scales to 1.0; each token carries its own group scales. Some standalone numerical tests instead choose a global scale from the tensor's absolute maximum. Their results cannot be treated as direct task-quality validation of the fixed-scale serving path.

Writing and reading the packed cache

The Triton writer takes BF16 values and a token-to-physical-slot mapping. It computes group maxima, converts scales to FP8, rounds values to E2M1 and packs pairs. Invalid slots do not write.

The reader loads the selected QSA positions and their group scales and reconstructs values inside the attention kernel. It does not build a full-context BF16 staging cache.

There is an important ownership contract: valid destination slots must be unique within a write call. Duplicate slots can cause conflicting writes. Standalone checks cover duplicate rejection and overwriting a physical slot with new data. The integrated scheduler must also satisfy this contract under append, prefix reuse, cancellation and preemption.

The bug between planning and allocation

The hybrid allocator needed attention pages compatible with a recurrent-state page of 3,207,168 bytes. At an early planning step, the attention specification implied 1,024 bytes per token. A 3,136-token attention block was selected.

Later, the physical allocation used the packed 576-byte representation:

3,136 x 576 = 1,806,336 bytes

That page was too small. The format could be correct and the read kernel could pass its tests while the server still could not allocate a valid hybrid cache.

The correction belongs at the point where the backend tells the planner what it will actually allocate. The QSA backend's customize_spec() now publishes packed content bytes before platform block-size adjustment:

packed_region = spec.head_size // 2 + spec.head_size // 16
return replace(spec, state_content_bytes=2 * packed_region)

This is 288 bytes per KV head. The specification accounts for the two heads, producing 576 bytes per token. The matching page becomes:

3,207,168 / 576 = 5,568 tokens

The final cache uses physical uint8 K/V regions of 144 bytes and quantization mode NONE. QSA owns the packing and unpacking; the finalized physical width must not be transformed again. The 5,568-token page is a consequence of this configuration, not a constant to copy into another model.

Checking the implementation separately from model quality

The numerical checks below come from an earlier validation run. I used an independent codec alongside the Python reference and CUDA quantizer. A separate comparison checks fused reading against dense attention over the same decoded NVFP4 values.

Historical check	Result
Independent decode vs Python reference	cosine 0.99999726
Independent codec vs CUDA packed bytes	0.999634 match
Fused vs dense NVFP4 read	cosine 0.99999702
Fused NVFP4 vs BF16 oracle	cosine approximately 0.99079
Read boundary matrix	27/27
Write boundary matrix	28/28

The byte comparisons are not universally exact. Against the Python reference, 123 of 567 differing nibbles were exact midpoint ties. The remaining differences involved values near rounding boundaries and scale arithmetic. Calling all differences “just tie-breaking” would hide part of the observation.

The fused-versus-dense comparison checks arithmetic and layout on the quantized representation. The comparison against BF16 includes quantization error. Neither number measures downstream task accuracy.

The archived Compute Sanitizer runs report zero errors in four modes on the tested small shapes. Larger functional boundary cases are a different gate. This does not prove every long-context serving path or concurrent execution pattern safe.

What the long-context run actually tells me

Before the paired comparison, I had also exercised the integrated server with a synthetic 120,064-token input. In that earlier run, a single request reached its first text chunk in 14.569 seconds; a group of seven took 101.953 seconds to finish. Each request generated only one output token.

That established that the tested path could process a long input. The prompt repeated a phrase, so it said little about whether the model could retrieve useful information from that context. The paired retrieval check is more meaningful for correctness, but only reaches 9,280 input tokens.

The earlier run also used different settings, including MTP=2 rather than 3. I keep its raw observations in the package as a separate record; they are not a before-and-after speed comparison with the paired-comparison results.

Integration problems that remain

Two earlier BF16 startup attempts with MTP disabled failed with a CUDA illegal memory access, including in eager mode. A standalone GDN warmup with the same head dimensions passed on Blackwell and did not locate the earlier asynchronous fault. The successful comparison above uses MTP=3. FP8 was not measured in this window.

The serving stack also needed invalid/padded expert-ID handling, workspace reuse between compatible sequential MoE layers, and separation of the target's NVFP4 B12x backend from its BF16 draft backend. Embedding, output-head and index-buffer references required their own fixes.

Those changes are packaged separately. Sharing scratch buffers is appropriate only when their uses do not overlap or are explicitly synchronized. CPU tests verify route masking and shared-object identity; they do not establish CUDA stream or graph safety for arbitrary overlapping wrappers.

Testing the cache without loading the model

I separated the writer, reader and independent codec into a Python package that does not import vLLM. That makes the format and page operations testable without loading the full model.

from nvfp4_qsa import PackedCache

cache = PackedCache(blocks=4, page_size=16, device="cuda")

The checked API rejects duplicate valid write slots, non-finite inputs and incompatible dimensions. Its validation synchronizes with the host, so this wrapper is not presented as a production fast path. It explicitly rejects CUDA graph capture. Raw-kernel callers still own stream ordering and cache lifetime.

The separately installed wheel passed 16 tests on an RTX 4050 Laptop: nine CPU tests and seven GPU cases. GPU coverage includes ordinary, zero, small and saturating values; slot reuse; invalid metadata; and sparse attention over fragmented pages compared with an independent CPU calculation using decoded cache values.

What can be reproduced today

The complete server measurements came from my model-specific vLLM build. Its historical base revision has not been verified against a public upstream commit, which limits how directly someone else can reproduce that full stack.

I have also prepared a narrow port against public vLLM revision e126687a9a828d513c01a07cd69f025f27d63280, where the model implementation uses the module name qwen4_exp. The patch applies cleanly and its BF16/NVFP4 physical-spec checks pass. I have not validated a full server run of that public port. The server benchmark numbers in this article belong to the tested model-specific build.

The GitHub repository contains the installable portable source, tests, the integration patch, raw request results, dataset revisions and scripts for reconstructing the evaluation prompts. It contains no model weights. Large checkpoint tensors were not rehashed during the maintenance window; the provenance record distinguishes download metadata from verified file hashes.

The next serving experiments need repeated, independently warmed batches, a matched FP8 baseline, and a larger quality evaluation. The failed MTP-disabled startup path also needs investigation. Those are separate pieces of work from validating the cache format.

For now, the strongest result is the extra room reported by the allocator, backed by working reads and writes and a small paired evaluation. The part I would carry into another implementation is the physical-byte accounting: the quantizer, reader and cache planner all need to agree on what a page actually contains.

Code and reproduction

Elsewhere

Instagram · Telegram

Running Qwen Flash-Next NVFP4 in vLLM: PLE Loading, B12x Fixes, and Stable Inference

Yuri Pocepaev — Sun, 27 Sep 2026 14:32:29 +0000

Companion code: PLE and B12x compatibility patches, tests, and deployment notes. The repository contains extracted fixes and validation notes, not a complete runtime image.

The model loaded. The health endpoint returned a successful response. A few requests completed. Then concurrent traffic arrived, and the inference process ran out of GPU memory.

That was the most instructive part of bringing Qwen Flash-Next NVFP4 online. There was no single switch that made the deployment work. We had to get a large checkpoint onto disk, adapt a loader to its tensor layout, and find a memory configuration that survived actual generation. Each milestone removed one obstacle while exposing the next.

The eventual result was a server configured for 131,072 tokens of context, with a limit of 16 sequences, that completed three rounds of 16 concurrent requests without a recorded container restart. Getting there required giving up some of our initial context and concurrency targets—and being precise about what the successful test actually demonstrated.

Before getting into the details, a big shout-out to windowsxp811203. The quality of the work and the detailed reports deserve real credit. It is great to see someone put that much care into both the result and its documentation.

This account follows the initial bfloat16 KV-cache baseline and the additional B12x integration work needed by the developed serving stack. The measured 48-request test belongs to the initial baseline. The later B12x work is described separately, without transferring that test result to a different runtime. The loader and backend patches were required independently of KV-cache quantization.

The first practical problem was simply getting the checkpoint onto the machine. The recorded download comprised 31 files, approximately 186.4 GB in total. A single PLE shard accounted for roughly 102.4 GB. With artifacts of that size, storage preparation and download recovery were part of the deployment work rather than incidental setup.

We provisioned a dedicated volume with an ext4 filesystem and used the Hugging Face CLI with eight download workers. The transfer ran through a local proxy. At one point, progress stalled at around 14 GB. After restarting, the reported progress fell to approximately 4.7 GB before the transfer resumed at about 93 MiB/s.

The practical consequence was extra transfer time and uncertainty about completion. A progress counter was useful for watching the download, but it was not sufficient evidence that the checkpoint was ready. We waited for the final shard to finish before moving on to loading.

It is also important to keep disk size separate from GPU residency. The total checkpoint size was not a prediction of the amount of VRAM the process would consume. This deployment used PLE CPU offload, so some of the storage and loading work belonged on the host side. Later failures still depended on the memory available on the GPU during generation.

Before replacing the existing model server, we saved its container configuration and retained a rollback target. That gave the experiments a defined starting point and preserved a route back if the new model failed. It also prevented the previous deployment from becoming another variable while we were changing the new one.

The initial runtime was a model-specific vLLM container image. It provided a starting point, but it did not accept this checkpoint without local changes. Two incompatibilities surfaced before inference tuning could begin.

The first was in the attention configuration. Twelve layer entries used qwen_sparse_attention, while the runtime path we were using expected full_attention. We supplied a local configuration override that changed those entries.

That was an operational compatibility change, and it has an important limit: getting past configuration parsing does not establish that the two settings are computationally equivalent. This work did not include a controlled comparison of output quality or attention behavior before and after the override. The evidence supports the narrower claim that the override allowed this runtime to proceed with loading.

The second incompatibility involved the layout of the PLE embedding table. The loader expected checkpoint shards with numeric suffixes. The checkpoint instead exposed the complete table through a tensor name whose shard index was empty. There was also a naming mismatch: the loader path had to recognize both the longer prefixed form and the shorter form actually encountered during loading.

The distinction sounds small until the tensor behind the name is enormous. A loader cannot safely infer that an unrecognized name represents an interchangeable piece of the model. It needs to identify the layout and validate the shape before copying the data.

Our fix went through three iterations. The first recognized the empty suffix and tried to hand the tensor to the ordinary loading path. That was insufficient. The second added explicit handling for the complete table, including shape validation, but recognized only the longer prefix. The third also handled the shorter name used by the checkpoint. That was the version that loaded successfully.

The successful path checked the incoming tensor against the expected full embedding dimensions. It then used the embedding-copy helper with a checkpoint offset of zero and the relevant embedding partition boundaries. Numbered shards retained their existing handling, and other weights continued through the regular loader.

The shape check was an essential part of the change. Supporting an additional serialization layout should not mean accepting arbitrary tensor dimensions just because the name looks familiar. A mismatch should fail at loading, where the error can be explained, rather than being allowed to become an inference problem later.

We packaged the patch as a small layer over the original container image, replacing a single loader module. Keeping that change narrow made it easier to understand what differed between the starting runtime and the working one. The rest of the deployment could then be investigated without treating the entire runtime as a custom fork.

The PLE loader was only one part of the runtime work. The serving stack subsequently required a separate set of fixes at the vLLM–FlashInfer boundary to use the B12x MoE path on our Blackwell SM120 GPU. This is a separate stage from the initial concurrency test reported below; its patches should not be retroactively attributed to that earlier run.

That distinction changes how the deployment should be described. A single loader replacement explains the initial checkpoint-loading fix. The more developed serving image also carries backend integration, speculative-decoding, offload, and scheduling changes. Reproducing that stack requires preserving the complete working runtime, rather than assuming that a stock image plus the PLE patch is equivalent.

The B12x work addressed two very different problems: invalid expert routes and duplicated workspace allocations. We changed the integration around the backend. We did not rewrite the B12x CUDA kernels.

During dummy profiling and padding, the integration could pass an expert ID of -1. That value represented an inactive route on the caller's side, but it could reach a path that treated it as an expert index. Our deployment notes associate this failure with cudaErrorIllegalAddress, GPU-process failures, and Xid 31 reports.

The wrapper now validates each selected expert ID before calling the original backend. A route is valid only when its ID is nonnegative and below the expert count. An invalid route is redirected to expert zero and its routing weight is set to zero. Valid routes retain both their IDs and weights.

The essential operation is small:

valid = (expert_ids >= 0) & (expert_ids < num_experts)
safe_ids = where(valid, expert_ids, 0)
safe_weights = where(valid, routing_weights, 0)

Redirecting an invalid ID alone would be wrong: it would turn padding into a real contribution from expert zero. Clearing the routing weight is what preserves the intended zero contribution of an inactive route. The wrapper also catches IDs above the valid range, although that defensive handling is not a substitute for diagnosing an upstream routing bug if one appears outside padding or profiling.

The saved unit test makes the transformation explicit. Given IDs [-1, 7, 512] for a model with 512 experts, the wrapper passes [0, 7, 0]. Given routing weights [0.4, 0.5, 0.6], it passes [0.0, 0.5, 0.0]. That verifies the wrapper's treatment of invalid routes; it is not, by itself, a numerical validation of the complete GPU inference path.

The second problem was workspace ownership. Our deployment notes describe allocations of roughly 400 MiB per MoE layer across 48 layers. At that scale, allocating an independent workspace for every layer can consume approximately 18.75 GiB before considering the rest of the runtime. This is an estimate from the reported per-layer allocation, not a separately measured memory saving.

Those layers execute sequentially in the execution pattern for which the patch was built. Compatible wrappers can therefore reuse scratch storage instead of reserving a separate copy for every layer.

The implementation caches three buffers: the static workspace, the dynamic workspace, and the MoE output buffer. The cache key includes expert count, top-k, hidden and intermediate dimensions, maximum token count, local expert count, output dtype, device, activation, quantization mode, and source format. Wrappers with matching keys reuse the same buffers; a different key triggers a separate allocation.

Compatibility and execution order both matter. Matching dimensions alone would not make shared writable storage safe for overlapping executions. In this deployment, reuse relies on the sequential layer execution pattern. A change to scheduling or stream behavior would require checking that assumption again.

The saved workspace test verifies that two identically configured wrappers allocate once and share the same buffer objects. That is useful evidence about the Python integration. It does not replace a runtime test under CUDA graphs and real traffic.

This design is also reflected in the current FlashInfer B12x API documentation, which exposes optional shared static, dynamic, and output buffers for compatible wrappers. That describes the current upstream interface; it does not establish that the runtime version used in our deployment already provided an equivalent integration.

Why pursue B12x at all? Our deployment notes identify Marlin as the fallback and B12x as the preferred native execution path for the NVFP4 MoE workload on this GPU. This article does not include a controlled comparison between the two, so we do not attach a speedup number or a claim of identical output quality to that choice. Selecting a backend does not, by itself, requantize the checkpoint, but numerical equivalence still needs measurement.

The surrounding runtime also needed coordinated choices for CUDA graph shapes, prefill budget, and MTP fallback. The recorded constraints included the 2^31 - 1 memref limit in the affected path, autotuner allocations, and a narrow VRAM margin. These were additional integration constraints, not all manifestations of the same memory bug. The memref limit in particular should not be read as a universal limit on GPU memory or model context.

This is why the custom development runtime mattered. Sanitizing routes addressed invalid indexing. Workspace reuse addressed duplicate buffers. Graph, prefill, and speculative-decoding choices addressed other execution constraints. The PLE loader change solved a separate checkpoint-format problem. Replacing the full image with a stock runtime would discard that combination of fixes in the environment described here.

The resulting stack combines separate responsibilities:

NVFP4 MoE experts
  -> FlashInfer B12x
  -> route validation and compatible workspace reuse

PLE n-gram table
  -> CPU offload and checkpoint-layout compatibility

KV cache
  -> an independent precision and capacity decision

These patches were necessary for checkpoint compatibility and backend execution in our environment, before any decision to quantize the KV cache. A reproduction should preserve the complete working runtime and validate each change against that baseline.

With those later integration changes distinguished from the initial baseline, the earlier memory-tuning sequence is easier to read on its own terms.

Once the checkpoint loaded, memory became the main constraint. The initial goal was a context window close to 240,000 tokens with a relatively generous sequence limit. Our first configuration requested 245,760 tokens, used gpu-memory-utilization=0.948, and allowed up to 64 sequences.

It failed the KV-cache capacity check. The runtime required 6.85 GiB and had 6.79 GiB available. Its calculated maximum was 243,936 tokens, just below the requested context length.

That small difference encouraged a tempting adjustment: increase the memory-utilization setting. We tried 0.96. In this environment, that left too little physical headroom for other work during initialization, and problems appeared around autotuning and CUDA graphs. The setting increased the budget available to the runtime, but it did not create additional memory on the device.

Reducing the requested context to 240,000 while returning utilization to 0.948 got us further. The runtime reported KV capacity for 242,909 tokens, completed startup, and answered its health check.

Then the first real inference failed. The failing operation wanted another 192 MiB, but only 32.44 MiB was free.

At that point the distinction between startup success and inference success was concrete. The server had enough memory for the checks it had already performed. It did not have enough for an allocation reached during generation.

The full progression looked like this:

Maximum context	Memory utilization	Maximum sequences	Observed result
245,760	0.948	64	Failed KV-capacity validation: 6.85 GiB required, 6.79 GiB available
245,760	0.96	64	Insufficient headroom around autotuning and CUDA graphs
240,000	0.948	64	Started successfully, then failed during generation
200,000	0.94	32	With expandable segments enabled, failed during startup
200,000	0.94	32	Without that allocator setting, passed sequential requests but failed under concurrent traffic
131,072	0.92	16	Completed three rounds of 16 concurrent requests

The inference failure pointed into the GDN linear-attention path. The trace identified a temporary state-tensor allocation during forward execution. That allocation needed free memory beyond the resources already accounted for by the successful startup and KV-capacity checks.

The GPU was also shared with other workloads. Its nominal capacity was therefore not the same as the budget available to this model. The amount of free memory at a critical allocation depended on more than the model weights and configured KV cache.

Our working explanation was that the deployment had too little margin for temporary allocations during generation, and the concurrent workload exposed that margin more reliably than isolated requests. The failure locations and reported allocation sizes supported that explanation. We did not collect a full allocation-level memory profile, so we could not quantify every contributor to the peak.

The next configuration was particularly misleading. At 200,000 tokens of context, utilization of 0.94, and a sequence limit of 32, several sequential requests succeeded. Requests through the API gateway also completed. If the acceptance criterion had been “the endpoint answers,” we could have stopped there.

Concurrent traffic produced another OOM. This time the failing allocation requested 290 MiB with only 150 MiB free. The container restarted, making it clear that the earlier successful responses had not exercised the workload that mattered.

That result changed the way we evaluated subsequent attempts. A successful sequential request remained useful as a functional check, but it could no longer stand in for a concurrency test. We needed to send overlapping requests and observe the process while generation was in progress.

We also tried the allocator option expandable_segments. In this setup, it introduced a different failure mode: an OOM during memory mapping, followed by a pidfd_getfd: Operation not permitted error in the PLE offload worker. We removed the setting.

That experiment did not establish that expandable segments are generally unsuitable for inference. It established that the option did not solve this deployment's problem and made this particular startup path fail. Carrying it forward would have added another unresolved variable.

The configuration that passed our concurrent test reduced three limits together: maximum context, maximum sequences, and memory utilization. The resulting settings were:

max-model-len: 131072
kv-cache-dtype: bfloat16
gpu-memory-utilization: 0.92
max-num-seqs: 16
PLE CPU offload: enabled
MTP speculative tokens: 1
prefix caching: enabled

The runtime also used the Qwen reasoning and tool-call parsers. Multimodal request limits were configured, but the test described below used text requests; it did not validate image or video workloads.

Changing three memory-related parameters at once means we cannot attribute the success to one of them in isolation. We did not perform an exhaustive sweep that held every other factor constant. The objective at this stage was to find a usable baseline in the actual environment. The combination above supplied that baseline for the workload we ran.

We tested it with three rounds of 16 concurrently submitted chat-completion requests. Each request asked for a roughly 300-word essay about distributed systems. The generation limit was 700 tokens, temperature was 0.8, and the client timeout was 180 seconds. Each round finished before the next one began.

The script counted responses containing a finish_reason field and inspected the container restart count. It also sampled GPU memory approximately every two seconds during the beginning of each round.

The recorded results were:

Round	Responses with a finish reason	Recorded container restarts
1	16 / 16	0
2	16 / 16	0
3	16 / 16	0
Total	48 / 48	0

The highest sampled GPU memory usage was 92,712 MiB, approximately 90.54 GiB. The final memory snapshot reported 4,537 MiB free, approximately 4.43 GiB.

Those numbers need a little care. The samples describe device-wide usage, including other processes on the GPU. Sampling at intervals can miss short peaks, and the script did not continuously sample for the entire duration of every request. The free-memory reading was a final snapshot; it should not be presented as a continuously guaranteed reserve under load.

The completion count also has a specific meaning. All 48 responses passed the script's check for the presence of a finish reason. That check did not evaluate the quality of the essays or require that every response ended naturally rather than reaching its token limit. We did not derive a throughput claim, a latency percentile, or a quality score from this test.

Most importantly, a server configured for 131,072 tokens of context was not being tested with 131,072-token prompts. These were short prompts with bounded generation. The result established that the selected configuration handled this concurrent text workload without a recorded restart. Testing long-context prefill, mixed prompt lengths, sustained traffic, and multimodal requests would require additional runs.

After the concurrent test, requests through the API gateway also completed. That checked a separate part of the deployment: the model was reachable through the intended serving route after the container changes. It complemented the direct inference test, while leaving the same workload limits in place.

What made this exercise useful was the sequence of increasingly meaningful checks. Download completion established that the artifacts were present. Successful loading established that the runtime could interpret them. A health response established that the service was up. A generated response exercised inference. Concurrent requests exposed a memory problem that the earlier checks had missed.

The same distinction applied to the fixes. The loader patch addressed a checkpoint-layout mismatch. The configuration override got past a runtime compatibility issue but still needed separate quality validation. The final memory settings addressed the observed failures under a particular workload. None of those changes, on its own, justified calling every aspect of the deployment validated.

We ended with a smaller context target than we initially wanted, a lower sequence limit, and a configuration that completed the chosen concurrency check. That was a much more useful foundation for further experiments than a larger advertised context attached to a process that restarted when traffic overlapped.

Further experiments need separate baselines for the runtime and the representation being changed. Long-context and sustained-load tests should exercise the complete custom stack. Any KV-cache quantization experiment should retain the established backend integration and be evaluated independently. Keeping those changes distinguishable lets us measure memory consumption, reliability, and answer quality without attributing a result to the wrong patch.

Elsewhere

Instagram · Telegram