How I Took GLiClass FP8 from 59 ms to 16 ms on an RTX 4050

Multilingual zero-shot classification on a 6 GB laptop GPU, with native FP8, Triton, CUDA Graphs, and reproducible measurements.

I quantized Knowledgator’s GLiClass Multilang Ultra into a custom FP8 W8A8 checkpoint. Then I wanted to find out how much of that compression could translate into faster local inference.

The weights already took up a third less disk space. The first native FP8 implementation, however, needed about 59 ms per request. The original BF16 model took roughly 24 ms. Reducing precision had made the artifact smaller, but the execution path still needed work.

After fusing auxiliary operations with Triton and adding CUDA Graphs, median request latency reached 16.10 ms. BF16 with the same graph wrapper measured 23.79 ms. On this particular workload, the optimized FP8 path was 1.48× faster, with roughly a third less memory allocated to tensors.

Here is what the model does, what I changed, and what these measurements actually establish. The weights, implementation, and benchmark report are on Hugging Face.

What GLiClass does

GLiClass Multilang Ultra, developed by Knowledgator, classifies text against labels supplied at inference time. You can provide categories such as “delivery,” “refund,” “payment,” and “other” without training a new classification head for that particular list.

For example:

text = "The courier has postponed my delivery for the second day."
labels = ["delivery", "payment", "refund", "other"]

The useful output is a class and its score. This makes classification a practical component before retrieval, application logic, or a more expensive model call.

Potential applications include support-ticket routing, document categorization, intent detection, and filtering text collections. Those are examples of where this interface can help; this experiment did not validate every one of those tasks.

I used the multilingual Ultra model. My quality evaluation covers English and Russian topic classification. It does not establish performance across every language supported by the upstream model or readiness for a particular production workflow.

What is inside the FP8 checkpoint?

I started from the original GLiClass model and built the FP8 W8A8 quantization used here. A separate phase then audited the tensors, measured quality, and optimized its inference path with Triton and CUDA Graphs. Knowledgator remains the developer of the original model; the quantization and local inference work are my derivative contribution.

The checkpoint quantizes 168 matrices across 24 mT5 encoder blocks: the attention projections q, k, v, and o, plus the feed-forward projections wi_0, wi_1, and wo.

Weights use FP8 E4M3 with a separate scale for each output channel. Activations are quantized dynamically for each token. Embeddings, normalization layers, and classification components remain in BF16.

W8A8 therefore describes the selected linear projections. It does not mean every tensor in the model uses eight bits.

Weight file	Size
Original BF16	3,416,522,340 bytes
FP8 W8A8	2,259,902,516 bytes

That is a 33.85% reduction, or 1.51× compression. The shared embedding matrix alone accounts for approximately 1.02 GB of the remaining size.

I also checked the unquantized tensors against a pinned snapshot of the original model. All 80 original tensors expected to remain unchanged matched byte for byte. The exact upstream revision used during the historical quantization was not recorded, so the report states that explicitly. The BF16 snapshot used for the comparison is pinned.

Why FP8 was initially slower

The storage format does not tell you which operations the GPU will execute.

One loading configuration reconstructed the weights in BF16 while retaining activation-quantization emulation. That path took around 264 ms per request. It was useful as a diagnostic, but it was not a native FP8 performance measurement.

The next adapter used actual FP8 matrix multiplication through torch._scaled_mm and cuBLAS. A separate profiler pass recorded 168 FP8 GEMM kernels for sm89, one for each replaced projection. Yet the complete request still took 59.23 ms.

The trace showed an obvious place to investigate: 3,269 GPU kernel executions for one request. Around the matrix multiplications, separate operations computed maxima, calculated scales, converted types, padded rows, and processed outputs.

At batch size 1, the cost of running that sequence matters. Matrix multiplication is only part of request latency. The profile suggested reducing auxiliary operations and launch overhead; the subsequent measurements tested those changes at the request level.

The changes that made a difference

Fuse activation quantization

For each token, the adapter needs an absolute maximum, a scale, and an FP8 representation of the activations. The original implementation expressed these as separate PyTorch operations.

I moved them into one Triton kernel. It performs the reduction, calculates the scale, converts values to E4M3, and fills the additional rows needed for matrix alignment.

Fuse the output scaling

The native FP8 GEMM produces an FP32 result. The adapter then applies the activation scale for each token and the weight scale for each output channel before converting back to BF16.

That sequence became a second Triton kernel. The stored FP8 weights are used directly, without another round of weight quantization. GEMM continues to use use_fast_accum=False.

Together, these changes reduced median latency to 37.97 ms.

Replay the encoder with CUDA Graphs

Next, I added CUDA Graphs around the encoder. A graph replays a captured sequence of GPU work, reducing the overhead of issuing its operations separately from Python.

Text lengths vary, so the adapter uses four buckets: 64, 128, 192, and 256 tokens. Inputs are padded to the appropriate bucket, and the extra positions are masked. Before classification, the encoder output is sliced back to the original token count.

Every request copies fresh tokens and a fresh mask into the graph's input buffers. Predictions are not cached. The output is cloned before it is returned to the caller.

One compatibility issue needed attention: in the tested Transformers version, attention-mask construction copied a scalar from CPU to GPU during graph capture. Preparing the mask outside the captured region resolved it.

With these changes, median latency reached 16.10 ms. The profiled request contained 1,425 GPU kernel executions, down from 3,269, while retaining all 168 native FP8 GEMMs.

How I measured it

The machine was an NVIDIA RTX 4050 Laptop with 6 GB VRAM, running Windows. The environment used Python 3.12.9, PyTorch 2.11.0 with CUDA 12.8, GLiClass 0.1.20, Transformers 5.14.1, and Triton Windows 3.6.0.post26.

Latency was measured on one fixed AG News example with four candidate labels, at batch size 1. After warmup and quality evaluation, I ran 50 additional requests. Each request was synchronized with CUDA before and after timing. The measurement includes tokenization and postprocessing.

Model loading, Triton compilation, and graph preparation are excluded from these steady-state timings. All four graph buckets were prepared in advance. FP8 graph preparation took 2.18 seconds in this run; that is not the complete cold-start time.

Runtime	Median	p95	Peak allocated tensor memory
BF16, eager	24.18 ms	29.67 ms	3.219 GiB
BF16 + CUDA Graphs	23.79 ms	24.45 ms	3.238 GiB
FP8, initial native adapter	59.23 ms	66.36 ms	2.145 GiB
FP8 + Triton	37.97 ms	46.62 ms	2.144 GiB
FP8 + Triton + CUDA Graphs	16.10 ms	16.44 ms	2.166 GiB

For the main comparison, I applied the same CUDA Graph wrapper to the original BF16 model. Optimized FP8 was 1.48× faster than that control. The 3.68× improvement compares the optimized implementation with my initial native FP8 adapter; it describes progress in the adapter implementation.

The memory column measures allocated PyTorch tensors. It is different from total device memory shown by nvidia-smi, which also reflects allocator reservations and other overhead. The current loader temporarily reconstructs weights in BF16 before replacing the projections, so 2.17 GiB is not a loading-memory requirement.

Laptop clocks and thermals were not locked. These are measurements of this configuration, not performance confidence intervals across independent sessions. I did not measure sustained throughput at larger batch sizes.

What happened to quality?

The paired evaluation used 664 identical examples: a seeded subset of 256 AG News test examples and the full English and Russian SIB-200 test splits, with 204 examples each. SIB-200 candidate labels were in English for both languages.

Dataset revisions, example indices, and hashes are recorded in the repository. The original and quantized tokenizers produced identical token IDs. No example was truncated at the 256-token limit.

Dataset	BF16 macro-F1	Optimized FP8 macro-F1	Difference
AG News	79.08%	79.49%	+0.41 pp
SIB-200 English	84.57%	84.04%	−0.53 pp
SIB-200 Russian	84.09%	83.42%	−0.67 pp

Optimized FP8 changed 5 of 664 top-1 predictions relative to BF16, for 99.25% agreement. Agreement measures whether the two variants choose the same label; it is not classification accuracy. BF16 with and without the graph wrapper produced identical top-1 predictions on all 664 examples.

I would not interpret the small positive AG News difference as an improvement in model quality. The other two datasets show declines, and this evaluation is too narrow to establish general quality preservation. A support-ticket router, for example, would need its own evaluation on representative tickets and candidate labels.

The single-label pipeline retained only the winning label and its probability. Full score distributions were not saved, so this report makes no claims about changes across all logits or probability calibration.

Try it locally

The Hugging Face repository contains the weights, tokenizer, dependencies, adapter, and an inference.py entry point.

After downloading the repository and installing the environment described in the model card, run:

python inference.py --text "NASA launched a spacecraft to study Mars." --labels space politics sports business

Or use the Python API:

import torch
from inference import load_pipeline

classifier = load_pipeline()

with torch.inference_mode():
    result = classifier(
        "NASA launched a spacecraft to study Mars.",
        ["space", "politics", "sports", "business"],
        batch_size=1,
        threshold=0.0,
    )

print(result)

The tested adapter supports sequential inference, batch size 1, and at most 256 input tokens, including labels and prompt formatting. The tokenizer truncates longer inputs. Concurrent requests to one instance require synchronization or separate runners.

This is currently a specialized inference path for the measured workload. Other GPUs, Linux execution, and installation in a fresh environment were not independently validated for this release. The measured configuration is Windows on an RTX 4050 Laptop.

What this experiment taught me

A quantized checkpoint has at least three separate properties: its size on disk, its runtime memory use, and the speed of a particular implementation. Here, the smaller FP8 file initially produced a slower request. The latency improvement came from optimizing the operations around GEMM and the way the encoder's GPU work was launched.

The result is a reproducible starting point: 16.10 ms on the selected batch-1 request, approximately 2.17 GiB of allocated tensor memory, and measured quality changes. Larger batches, longer inputs, and real application workloads are the next useful experiments. Their results need to be measured separately.

The main takeaway is simple: quantization gives you a smaller representation; making it faster is a separate engineering problem.

The model, source code, and report are available under Apache-2.0. Knowledgator developed the original model; this release provides an independent quantized representation, inference implementation, and evaluation.

About the author

I’m Yuri Pocepaev, a software engineer and lead developer at Neuroprem. This article is part of my public engineering log.