Using RLM Cut's Token Costs by 96% for LLM

#rust #ai #llm #opensource

As someone who is constantly exploring ways to make AI applications faster and cheaper, I found myself looking for a solution to a problem that kept slowing me down: processing 100,000+ token context windows without burning through API budgets or waiting through long network delays. That's when I came across the research on Recursive Language Models (RLM).

This blog is a summary of my personal experience building RLM-Rust: how I got started, why I moved away from Python, the challenges I faced, what I built, and the lessons I learned. If you're a developer curious about handling massive context payloads more efficiently, this might help you decide if RLM is worth your time. (Spoiler: it absolutely is.)

A Personal Start: How I Came Across RLM,

I wanted to reduce the cost of running my agents, since more context on every call meant more tokens and more money. While exploring options, I found a repo implementing RLM in Python. I tested it out, but it was super slow — noticeable latency on every query. So I decided to switch it over to Rust, which improved things a lot which includes very less latency which was 37.6% faster than Python.

Starting Out : My First Working Prototype

My first real step was replacing Python subprocesses with Rhai, an embedded, sandboxed scripting engine written in pure Rust:

use rlm::core::rlm::{Rlm, RlmConfig};
use rlm::types::ClientBackend;

#[tokio::main]
async fn main() -> Result<(), Box<dyn std::error::Error>> {
    let config = RlmConfig {
        backend: ClientBackend::Gemini,
        backend_kwargs: serde_json::json!({
            "model_name": "gemini-flash-latest",
            "api_key": std::env::var("GEMINI_API_KEY")?,
        }),
        max_iterations: 10,
        ..Default::default()
    };

    let mut rlm = Rlm::new(config, None);
    let result = rlm.completion(&prompt, None).await?;
    println!("Answer: {}", result.response);
    Ok(())
}

What surprised me was how smooth and fast it felt. Running code inside Rust's local memory space executed in under 0.001 milliseconds, which just isn't possible with Python process spawning. That small win was enough to make me want to keep going.

What I Built Using RLM-Rust

🔹 Token Reduction Benchmark
I tested RLM-Rust against a raw direct LLM call using a 5,000-line log dataset (~120,000 tokens, 348KB). The raw call billed 83,917 input tokens. RLM-Rust billed 3,288 — a 96.1% reduction, with the same correct answer both times.

🔹 In-Memory Query Engine
Instead of sending the full text over the network every turn, RLM-Rust keeps the raw data in local memory and lets Rhai filter through it in sub-microsecond time, passing only the relevant tokens back to the API.

🔹 Multi-Provider Support
I added support for Gemini, OpenAI, Anthropic Claude, OpenRouter, Vercel AI Gateway, vLLM, and Azure OpenAI, so it's not tied to one provider.

Things I Found Out

Local memory beats network transfer. Keeping large context in RAM and querying it with code is a lot cheaper and faster than sending raw text over HTTP every time.

Embedded engines cut out a lot of overhead. Rhai beat subprocess spawning in every test — no IPC, no Python dependency.

Final Thoughts

RLM changed how I think about context processing in AI applications. It's not just a wrapper around an API call — it turns the model into something closer to an orchestrator that decides what it actually needs to look at, instead of parsing everything at once.

If you're dealing with large context windows and want to cut down your token costs, I'd recommend giving this approach a try. It still needs more testing and refinement in places. If you are interested you can give it try repo link is below.

📦 GitHub Repository: https://github.com/mohitjoer/RLM-Rust