DEV Community: Benito August

Under 3ms Latency Building a High-Performance LLM Gateway in Rust

Benito August — Wed, 17 Jun 2026 21:10:32 +0000

Scaling Large Language Model (LLM) integrations in production introduces three critical bottlenecks: high cloud API token costs, latency overheads, and data privacy leaks. When you use Retrieval-Augmented Generation (RAG) or feed long chat histories back to models like Gemini, Claude, or GPT-4, you are not only paying for redundant context, but you are also waiting for the network round-trip.

Traditionally, developers turn to Python-based microservices for LLM orchestration and preprocessing. However, Python’s interpreter overhead, lack of true concurrency, and high memory footprint make it a bottleneck when running local deep-learning models (like BERT-based NER or MiniLM sentence embedders) at high throughput.

To solve this, we designed NLProxy, a local-first, high-performance LLM gateway written in Rust. In this post, we’ll dive into how we leveraged the Rust systems programming ecosystem (ort, candle, tokio, regex, and linfa-clustering) to build a gateway capable of handling 5000+ requests per second with sub-3ms latency.

The Core Concept: Local-First Hybrid LLM Gateway

A local hybrid gateway sits between your internal application threads and public cloud APIs. Before any prompt egresses your network, the gateway intercepts it to perform:

PII Masking (Shielding): Local Named Entity Recognition (NER) to redact names, emails, and IPs.
Semantic Prompt Compression: Clustering sentences and discarding redundancies.
Prompt Firewall Check: Detecting jailbreaks, SQL injections, or ReDoS payloads in $O(n)$ linear time.
Post-Generation Verification: Using local Natural Language Inference (NLI) to flag hallucinations before returning the answer.

Doing all of this locally under 3 milliseconds is impossible in Python. Here is how Rust makes it achievable.

1. Zero-Copy PII Shielding with `candle`

Redacting Personally Identifiable Information (PII) before it reaches third-party servers is a strict compliance requirement under GDPR and HIPAA. In NLProxy, we use a hybrid approach: deterministic regular expressions combined with a local Named Entity Recognition (NER) model running on CPU.

For the NER engine, we leverage Hugging Face's candle, a minimalist machine learning framework for Rust. Unlike Python’s heavy PyTorch bindings, candle lets us run lightweight BERT models natively without virtual environments or massive dynamically linked libraries.

Here is a look at how NLProxy's NerEngine loads a safe tensor BERT model and predicts entity spans:

use candle_core::{Device, Tensor};
use candle_transformers::models::bert::{BertModel, Config};
use tokenizers::Tokenizer;
use candle_nn::VarBuilder;

pub struct NerEngine {
    model: BertModel,
    tokenizer: Tokenizer,
    classifier_weight: Tensor,
    classifier_bias: Tensor,
    device: Device,
}

impl NerEngine {
    pub fn load(models_dir: &std::path::Path) -> anyhow::Result<Self> {
        let device = Device::Cpu;
        let folder = models_dir.join("bert-base-NER");
        let config_str = std::fs::read_to_string(folder.join("config.json"))?;
        let config: Config = serde_json::from_str(&config_str)?;
        let tokenizer = Tokenizer::from_file(folder.join("tokenizer.json"))?;

        let vb = unsafe {
            VarBuilder::from_mmaped_safetensors(&[folder.join("model.safetensors")], candle_core::DType::F32, &device)?
        };

        let model = BertModel::load(vb.pp("bert"), &config)?;
        let classifier_weight = vb.get((9, 768), "classifier.weight")?;
        let classifier_bias = vb.get(9, "classifier.bias")?;

        Ok(NerEngine { model, tokenizer, classifier_weight, classifier_bias, device })
    }
}

By memory-mapping safetensors (VarBuilder::from_mmaped_safetensors), we eliminate the startup overhead and minimize RAM usage. The system redacts PII into random token hashes (e.g. __PROT_82739182__) stored locally in memory, restoring them instantly when the LLM returns the output.

2. Semantic Prompt Compression with `linfa` and `kodama`

To reduce input token fees, we compress prompts by clustering similar sentences and keeping only the most representative ones. We run a sentence-embedding model local ONNX via the ort crate (ONNX Runtime bindings) to obtain dense vector arrays (Array2<f32> from the ndarray crate).

Once embeddings are computed, we can choose between:

KMeans Clustering (using the linfa-clustering crate) for massive contexts.
Hierarchical Ward Linkage (using the kodama crate) for precise structural grouping.

Here is a snippet showing how NLProxy's semantic compressor handles KMeans sentence reduction locally in Rust:

use linfa_clustering::KMeans;
use linfa::traits::{Fit, Predict};
use ndarray::Array2;

fn cluster_kmeans(
    sentences: Vec<String>,
    embeddings: &Array2<f32>,
    original_indices: Vec<usize>,
    aggressiveness: f32,
) -> (Vec<String>, Vec<usize>) {
    let n = sentences.len();
    let n_clusters = (n as f32 * (1.0 - aggressiveness)).max(1.0).min(n as f32) as usize;

    if n_clusters >= n {
        return (sentences, original_indices);
    }

    let kmeans = KMeans::params(n_clusters)
        .max_n_iterations(100)
        .tolerance(1e-4);

    let dataset = linfa::Dataset::from(embeddings.clone());
    let model = kmeans.fit(&dataset).expect("KMeans clustering failed");
    let labels = model.predict(&dataset);

    let mut selected_sentences = Vec::new();
    let mut selected_indices = Vec::new();

    for cluster_idx in 0..n_clusters {
        let mut cluster_member_indices = Vec::new();
        for (i, &label) in labels.iter().enumerate() {
            if label == cluster_idx {
                cluster_member_indices.push(i);
            }
        }

        if cluster_member_indices.is_empty() { continue; }

        let centroid = model.centroids().index_axis(ndarray::Axis(0), cluster_idx);
        let mut best_idx = cluster_member_indices[0];
        let mut min_dist = f32::MAX;

        for &idx in &cluster_member_indices {
            let emb = embeddings.index_axis(ndarray::Axis(0), idx);
            let dist = emb.iter().zip(centroid.iter())
                .map(|(a, b)| (a - b).powi(2))
                .sum::<f32>()
                .sqrt();

            if dist < min_dist {
                min_dist = dist;
                best_idx = idx;
            }
        }

        selected_sentences.push(sentences[best_idx].clone());
        selected_indices.push(original_indices[best_idx]);
    }

    // Sort selected sentences back to their original document order
    let mut combined: Vec<_> = selected_sentences.into_iter().zip(selected_indices.into_iter()).collect();
    combined.sort_by_key(|&(_, idx)| idx);
    combined.into_iter().unzip()
}

Executing this in Rust takes a fraction of a millisecond. We get a 40% to 55% reduction in token count without losing the system instructions or core semantic context.

3. ReDoS-Immune Firewall Engine

LLM application firewalls must defend against prompt injection attacks (like jailbreaks) and system denial-of-service attempts. Python and JavaScript regular expression engines are prone to ReDoS (Regular Expression Denial of Service) due to backtracking. An attacker can construct a complex query that causes the CPU to lock up, taking down your gateway.

Rust’s regex crate solves this fundamentally. It guarantees linear-time search $O(m \times n)$ with respect to the input length by using Finite Automata (DFA/NFA) representation, avoiding backtracking entirely.

NLProxy wraps these rules into deterministic DFA checks, and overlays semantic classification using cosine similarity against an attack vector database. The CPU overhead is negligible (measured in micro-seconds), keeping the firewall execution invisible to user request threads.

4. Hallucination Guardrails with Local NLI

Before sending the LLM output back to the user, we evaluate it against the original prompt or source document to check for logical contradictions. Running another cloud LLM call to evaluate the answer is slow and expensive.

Instead, NLProxy embeds a lightweight cross-lingual XLM-RoBERTa model set up for Natural Language Inference (NLI) classification (Entailment vs Contradiction). Running this locally via candle classification layer allows us to identify contradictions on the fly:

use candle_transformers::models::xlm_roberta::XLMRobertaForSequenceClassification;

pub fn predict_contradiction(&self, premise: &str, hypothesis: &str) -> anyhow::Result<f32> {
    let tokens = self.tokenizer.encode((premise, hypothesis), true).map_err(anyhow::Error::msg)?;
    let token_ids = tokens.get_ids();
    let input_ids = Tensor::new(token_ids, &self.device)?.unsqueeze(0)?;
    let attention_mask = Tensor::new(vec![1u32; token_ids.len()], &self.device)?.unsqueeze(0)?;
    let token_type_ids = Tensor::new(vec![0u32; token_ids.len()], &self.device)?.unsqueeze(0)?;

    let logits = self.model.forward(&input_ids, &attention_mask, &token_type_ids)?;
    let logits_vec = logits.squeeze(0)?.to_vec1::<f32>()?;

    let max_val = logits_vec.iter().cloned().fold(f32::NEG_INFINITY, f32::max);
    let sum_exp: f32 = logits_vec.iter().map(|&val| (val - max_val).exp()).sum();
    let contradiction_prob = (logits_vec[0] - max_val).exp() / sum_exp; // index 0 matches Contradiction

    Ok(contradiction_prob)
}

If the contradiction probability exceeds a threshold (e.g. 70%), the gateway intercepts the response, blocks the output, or corrects placeholders natively before the user ever sees it.

Real-World Performance & Benchmarks

Running the entire unified pipeline (PII Shielding, Semantic Compression, Firewall, LLM Call, and Post-Verification) locally in Rust vs. Python yields massive efficiency gains:

Metric	Python (FastAPI + PyTorch)	Rust Core (NLProxy)	Improvement
Max Throughput	~180 req/sec	5000+ req/sec	27.7x
Pre-LLM Latency	~65 ms	< 3 ms	21.6x
RAM Footprint	~4.2 GB	~1.2 GB	3.5x
ReDoS Immunity	Vulnerable (PCRE)	100% Immune	Architectural

Conclusion

By writing the core gateway logic in Rust, NLProxy shows that systems programming is no longer just for operating systems or databases. For AI engineering, building compiled, native middleware on top of crates like candle and ort lets you run local models, guarantee absolute privacy, and slash cloud token bills without degrading your API response times.

If you are looking to build secure, low-latency, and cost-effective AI features, it’s time to move your integration middleware from Python to Rust.

NLProxy is an open-core project. Check out the repository on GitHub and start building high-performance AI middleware.

Stop Measuring Keystrokes: How Vanity Metrics are Fueling a "Productivity Theater" Crisis

Benito August — Mon, 30 Jun 2025 04:46:42 +0000

Your team might be "busy," but are they actually shipping quality code? Let's talk about the difference between activity and impact, and why it's crucial for your team's health and performance.

My frustration as a manager hit its peak when I faced a harsh reality: my project board was a fantasy.

I saw tickets in the "Done" column, but when it came to validation, the deliverables were broken, incomplete, or far from reality. My team was active, but they weren't effective.

Without realizing it, I was the director of a full-blown "Productivity Theater," and the script was being written by a set of deeply flawed metrics. This is a trap many leaders and teams are falling into in the remote era.

The Rise of "Productivity Theater"
"Productivity Theater" is the phenomenon where employees, feeling the pressure to appear busy, dedicate significant time and energy to simulating activity rather than producing effective work.

And it's not a minor issue. A startling report from Visier (2023) found that 43% of employees spend more than 10 hours per week on this kind of performative work.

Why does this happen? The social psychologist Dr. Devon Price, in his book "Laziness Does Not Exist," argues that our culture has ingrained in us a "Laziness Lie"—a belief system that equates our self-worth with our productivity. This pushes developers and knowledge workers to work until they're sick and feel immense guilt for not "doing enough," making it essential to look busy at all times.

Vanity Metrics: The Fuel for the Fire
This "theater" is fueled by a leader's obsession with Vanity Metrics. These are metrics that are easy to measure but offer little to no real insight into performance or impact.

In a dev team, this looks like:

Hours logged online

Number of commits

Lines of code written

Frequency of Slack messages

This approach is fundamentally at odds with the nature of deep, valuable work. As Cal Newport explains in his book "Deep Work," knowledge work is divided into two types:

Deep Work: "Activities performed in a state of distraction-free concentration that push your cognitive capabilities to their limit. These efforts create new value, improve your skill, and are hard to replicate."

Shallow Work: "Non-cognitively demanding, logistical-style tasks, often performed while distracted. These efforts tend not to create much new value in the world and are easy to replicate."

When you measure 'keystrokes' instead of quality, you are actively encouraging and rewarding Shallow Work, leading directly to burnout and poor outcomes.

The Solution: Shift from Activity to Impact
The change begins when we start asking different questions and measuring what truly matters. We need to shift our focus from Input (activity) to Output (results) and Outcome (impact).

Here's a simple framework contrasting these approaches:

Metric Type	Example	Why It's Dangerous / Useful	Leader's Focus
Input/Vanity	Hours Online	Dangerous: Fosters digital presenteeism, doesn't reflect real focus, leads to burnout.	Trust your team with flexibility.
Input/Vanity	# of Commits/Messages	Dangerous: Promotes quantity over quality. A single, well-thought-out commit is worth more than 10 minor ones.	Encourage effective, asynchronous communication and meaningful code contributions.
Output/Outcome	Sprint Velocity / Cycle Time	Useful: Measures the actual pace of delivery and helps identify process bottlenecks.	Set clear goals (SMART), realistic deadlines, and track progress.
Output/Outcome	Code Quality (e.g., bug rate, test coverage)	Useful: Reflects the effectiveness and craftsmanship of the work, directly impacting user satisfaction and maintainability.	Define quality standards, implement code reviews, and offer constructive feedback.
Output/Outcome	Impact on Business OKRs	Useful: Directly links the team's technical work to strategic business results.	Ensure team goals are aligned with company OKRs.

What's Your Experience?
This isn't just a management theory; it's a reality in our terminals and IDEs every day. The push for performative work is one of the biggest drains on developer morale and a direct path to burnout.

So, I'm asking you, the developers and tech leads in the trenches:

What's the most absurd vanity metric you've been judged by? Lines of code? Number of PRs?

Or, on the flip side, what's the most effective impact-based metric your team uses that you actually find valuable?

Share your stories (the good, the bad, and the ugly) in the comments below!

Closing Thought
Fighting against vanity metrics isn't about working less; it's about reclaiming our ability to work on what matters. It's about creating an environment where deep, meaningful work is not only possible but is the only thing that's truly valued.

References
Newport, C. (2016). Deep work: Rules for focused success in a distracted world.

Price, D. (2021). Laziness does not exist.

Tableau. (n.d.). What are vanity metrics?

Visier. (2023, March 1). Productivity survey shows performative work.