DEV Community: Ai developer

One Ruler to Measure Them All: How Language Affects LLM Quality

Ai developer — Fri, 29 May 2026 09:00:18 +0000

One Ruler to Measure Them All: How Language Affects LLM Quality

Most discussions about LLM performance focus on the model architecture and prompting. But there's a hidden factor: the tokenizer. It determines how much of your text fits in the context window.

The Tokenizer Problem

Russian text consumes more tokens than English for the same information density. Some developers even switch to English prompts to save tokens and improve performance.

The Surprising Result

A recent arxiv study benchmarked multilingual long-context language models across different languages. The winner? Polish — 88% accuracy.

Russian placed 5th at 84% — ahead of English at 83.9%.

The gap widens on long-context tasks. More tokens = more opportunities for the model to lose coherence.

Important Caveat

The test used "weaker" models by 2026 standards:

Gemini 1.5 Flash
Qwen 2.5 72B
Other mid-tier models

Top-tier models might show different patterns, but the tokenizer effect persists regardless of model quality.

Implications for Production

Language choice matters for RAG. If your knowledge base is multilingual, retrieval quality varies by language.
Long-context tasks favor compact languages. English is more token-efficient than Russian, but Polish outperformed both.
Tokenizer-agnostic metrics are needed. BLEU and ROUGE don't capture tokenization bias.

What I'm Tracking

I'm monitoring whether newer models (Kimi k2.5, GLM-5, GPT-5.2 series) show the same pattern. Early signs suggest top-tier models compress better across languages, but the gap doesn't fully disappear.

More multilingual LLM analysis and production AI notes from inside a bank — follow my Telegram channel:

https://t.me/ai_tablet (Russian, technical)

More AI engineering notes, RAG benchmarks, and production insights from inside a bank — follow my Telegram channel:

🚀 https://t.me/ai_tablet (Russian, technical)

One Ruler to Measure Them All: How Language Affects LLM Quality

Ai developer — Fri, 29 May 2026 06:01:46 +0000

One Ruler to Measure Them All: How Language Affects LLM Quality

Most discussions about LLM performance focus on the model architecture and prompting. But there's a hidden factor: the tokenizer. It determines how much of your text fits in the context window.

The Tokenizer Problem

Russian text consumes more tokens than English for the same information density. Some developers even switch to English prompts to save tokens and improve performance.

The Surprising Result

A recent arxiv study benchmarked multilingual long-context language models across different languages. The winner? Polish — 88% accuracy.

Russian placed 5th at 84% — ahead of English at 83.9%.

The gap widens on long-context tasks. More tokens = more opportunities for the model to lose coherence.

Important Caveat

The test used "weaker" models by 2026 standards:

Gemini 1.5 Flash
Qwen 2.5 72B
Other mid-tier models

Top-tier models might show different patterns, but the tokenizer effect persists regardless of model quality.

Implications for Production

Language choice matters for RAG. If your knowledge base is multilingual, retrieval quality varies by language.
Long-context tasks favor compact languages. English is more token-efficient than Russian, but Polish outperformed both.
Tokenizer-agnostic metrics are needed. BLEU and ROUGE don't capture tokenization bias.

What I'm Tracking

More multilingual LLM analysis and production AI notes from inside a bank — follow my Telegram channel:

https://t.me/ai_tablet (Russian, technical)

More AI engineering notes, RAG benchmarks, and production insights from inside a bank — follow my Telegram channel:

🚀 https://t.me/ai_tablet (Russian, technical)

AI Conf 2026: Classic ML Is Dead, Everyone's Building Agents

Ai developer — Fri, 29 May 2026 06:00:17 +0000

AI Conf 2026: Classic ML Is Dead, Everyone's Building Agents

Spent two days at AI Conf in Moscow. The shift is complete: nobody talks about traditional ML anymore. It's all agents, RAG, and voice systems.

1. Academic Publication Pipeline Is Slow

Average time from submission to publication at A-tier conferences: 9 months. Multiple review cycles, sequential improvements.

What researchers actually use LLMs for now:

Code generation
Paper review assistance
Literature synthesis
(Not for original ideas — tried "let it think for 2 weeks," expensive and ineffective)

Prediction: Future papers will include zip archives of experimental code that AI can verify. Human value shifts to idea generation, not implementation.

2. Search Agents Workshop

Built a working ReAct search agent in the workshop:

Groq API — free tier, fast inference
Tavily — 1000 free search queries/month
Langfuse monitoring

Stack cost: $0 for prototyping. Production cost: depends on scale.

3. Monitoring: Langfuse vs Arize Phoenix

Tool	Approach	Best For
Langfuse	Manual integration, detailed traces	Custom setups, granular control
Arize Phoenix	Auto-instrumentation, wraps everything	Quick setup, less configuration

Both show traces, token counts, latency breakdowns. Phoenix wins if you want observability without wiring it yourself.

4. Agent Harness vs Classic Agents

The terminology evolved:

2024: "What's the difference between LLM and agent?"
2026: Agent Harness — memory + skills instead of tools

Example: Deep Agents framework. Skill creation costs 2M tokens. Single invocation: 100K tokens. But the abstraction is cleaner than manual tool orchestration.

5. Voice Agents for Telephony

Voice-to-voice models exist but lack:

Tool use integration
Context management
Reliability for long conversations

Current production stack: Speech-to-Text → LLM → Text-to-Speech

Voice-to-voice will replace this eventually, but not before tool calling and context compression catch up.

What I Didn't Hear

Gradient boosting use cases
Feature engineering debates
Model interpretability discussions (except for RAG context windows)

The industry moved on. If you're still pitching Random Forest improvements, you're talking to the wrong audience.

My Take

The conference confirmed what I see in production: agent orchestration is the new infrastructure layer. Not the models themselves — how you connect them, manage memory, route between skills, and monitor everything.

The companies winning aren't those with the best single model. They're those with the best agent architecture.

More production AI insights and conference notes from a bank's DS lead — follow my Telegram channel:

https://t.me/ai_tablet (Russian, technical)

More AI engineering notes, RAG benchmarks, and production insights from inside a bank — follow my Telegram channel:

🚀 https://t.me/ai_tablet (Russian, technical)

I Self-Hosted an AI Assistant: Lessons from 48 Hours of Debugging

Ai developer — Thu, 28 May 2026 21:39:25 +0000

I Self-Hosted an AI Assistant: Lessons from 48 Hours of Debugging

I wanted a local AI assistant. Expected: 2 hours. Reality: 2 days of edge cases, broken dependencies, and discovering that "local" doesn't mean "free."

The Stack

OpenClaw (open-source AI assistant framework)
VPS with limited console access (had to file tickets to enable)
OpenRouter for model access
Local Qwen as fallback

What Broke

1. Dependency Hell

Pre-installed OpenClaw came with an outdated library. Updated manually. Then updated again. OpenRouter integration only worked after the second update.

2. Certificate Issues

Self-hosted means self-managed certificates. Let's Encrypt, reverse proxy, CORS headers. Each layer adds a new failure mode.

3. "Free" API Credits Aren't

OpenRouter's "free" models have limits. Hit them within hours. The API key died silently — no error message, just empty responses.

4. Local Model Reality Check

Qwen promised tool-use support. Reality:

Absolute paths broke tool calling (relative only)
Model experienced "amnesia" — couldn't open .md files it created
Larger models need more RAM but run slower
200K context window sounds great until you hit memory limits

5. The Debugging Cascade

Fix one thing → break another. Add skills for email and search. DuckDuckGo API rate-limits kill the search skill. Switch to alternative. New limits.

What Worked

Despite everything, the assistant is now running. Key insight:

Boxed solutions (Kimi, GLM native APIs) are more reliable. But self-hosting teaches you how the pieces actually connect — tool calling, memory management, model routing, context windows.

The Real Cost

Item	Expected	Actual
Setup time	2 hours	2 days
API costs	$0	$20+ before limits
Compute	Minimal	16GB+ RAM for usable local models
Maintenance	Zero	Ongoing dependency updates

Should You Self-Host?

Yes if:

You want to understand LLM infrastructure deeply
Data privacy is non-negotiable
You enjoy debugging more than using

No if:

You need reliability today
Your time has a cost
You're not ready to file support tickets for console access

What's Next

I'm keeping the local setup as a learning environment but routing production tasks to managed APIs. The hybrid approach: local for experimentation, cloud for reliability.

More self-hosting experiments and production AI infrastructure notes — follow my Telegram channel:

https://t.me/ai_tablet (Russian, technical)

More AI engineering notes, RAG benchmarks, and production insights from inside a bank — follow my Telegram channel:

🚀 https://t.me/ai_tablet (Russian, technical)

RAG SOTA: I Tested 7 Pipelines and Built SEQUOIA (Open Source)

Ai developer — Thu, 28 May 2026 21:35:37 +0000

RAG SOTA: I Tested 7 Pipelines and Built SEQUOIA (Open Source)

After 20+ hours of compute time on local hardware, I benchmarked 7 RAG configurations against real-world tasks. SEQUOIA (RAPTOR tree + step-back prompting) consistently outperformed alternatives.

The Full Pipeline List

Method	Core Approach
No-RAG	Direct LLM generation
Classical RAG	Dense retrieval (BGE-small + FAISS)
Hybrid RAG	BM25 + Dense + RRF + reranker
LightRAG	Key-value graph + dense hybrid
PageIndex	Two-stage hierarchical retrieval
GraphRAG	Entity graph + dense fallback
Agentic RAG	Multi-step reasoning pipeline
SEQUOIA	RAPTOR tree + step-back prompting
SEQUOIA Pro	Multi-query + rerank + compression

Why LightRAG Underperformed

The hype suggested graph-based RAG would revolutionize retrieval. On real banking documents and technical manuals:

Graph construction is expensive (entity extraction, relationship mapping)
Retrieval quality did not justify the overhead
Academic benchmarks do not equal production reality

Why RAPTOR Works

Recursive Abstractive Processing for Tree-Organized Retrieval:

Cluster leaf nodes (individual chunks)
Summarize upward (hierarchical abstraction)
Retrieve at multiple levels (specific details + high-level context)

This mirrors how humans organize knowledge.

Step-Back Prompting: Free Performance

Before retrieving, generalize the query:

User asks: "What's the error rate for Q3?"
Step-back: "What metrics are tracked quarterly?"
Retrieve broader context first, then narrow

Result: ~15% improvement in recall. Zero latency cost.

SEQUOIA Architecture

User Query
    Step-back Prompting (generalize)
    RAPTOR Tree Retrieval (multi-level)
    Context Compression (summarize long contexts)
    Re-ranking (cross-encoder)
    Local LLM Generation

Local LLM Evaluation

I used a local model weaker than GPT-4 for judging. Key finding: relative rankings between methods stayed consistent even with a weaker evaluator.

You can prototype and compare approaches without burning API credits on GPT-4 evaluations.

Production Recommendations

Start with Classical RAG — establish baseline, prove value
Add step-back prompting — free performance gain
Move to hierarchical retrieval when context complexity justifies it
Avoid graph approaches unless you have specific graph-structured data
Measure on YOUR data — academic benchmarks are misleading

Open Source

Everything is available:
https://github.com/Diyago/rag-benchmark/tree/main

Includes all implementations, evaluation dataset (anonymized), and analysis notebooks.

More AI Engineering Notes

I write about practical AI/ML from inside a bank — RAG systems, LLM deployment, team management, and what actually works versus what is just hype.

Telegram channel (Russian, technical): https://t.me/ai_tablet

Have you benchmarked RAG on real data? What surprised you?

More AI engineering notes, RAG benchmarks, and production insights from inside a bank — follow my Telegram channel:

🚀 https://t.me/ai_tablet (Russian, technical)

Open Source LLM Spring 2026: What Changed in 2 Months

Ai developer — Thu, 28 May 2026 21:25:20 +0000

Open Source LLM Spring 2026: What Changed in 2 Months

After tracking open-weight LLM releases for the past two months, here's what's actually moving the needle. Not hype — architecture and data decisions that matter.

1. Sliding Window Attention Goes Mainstream

Almost everyone switched to SWA. Context windows growing substantially without model bloat. The exception: MiniMax M2.5 still uses GQA (Grouped-Query Attention) but compensates purely through data quality on coding tasks.

Why it matters: You can now fit 200K+ context in models that previously handled 32K. Same parameter count, different attention mechanism.

2. QK-Norm Spreading

QK-Norm (query-key normalization) is emerging as an RMSNorm analogue. Traces back to Gemini 3 architecture. Stabilizes training at scale without adding compute.

3. Multimodal Pretraining Early

Kimi k2.5 showed that pretraining on images at early stages (not just late-stage fine-tuning) significantly helps reasoning. The model learns visual concepts before language alignment, making downstream multimodal tasks more robust.

4. GLM-5 (Z.ai) — Not a DeepSeek Clone

On release, GLM-5 matched GPT-5.2 / Opus 4.5 / Gemini 3 Pro on key benchmarks. What's inside: heavily modified DeepSeek-V2 architecture with changed parameters, especially active expert count in the MoE layers.

Key difference: It's not "DeepSeek with a new name" — the routing and expert allocation is fundamentally redesigned.

5. Step 3.5 Flash — Efficiency King

196B parameter MoE, architecturally similar to DeepSeek but:

3x faster inference speed
Multi-Token Prediction (generates 3 additional tokens per step instead of 1)
Currently #2 on OpenRouter by token consumption

The catch: Top benchmarks but Chatbot Arena tells a different story. Benchmarks ≠ real user preference.

What Surprised Me

The pricing. Z.ai raised GLM-5 subscription prices 2x immediately — up to $160/month for max tier. Open-source models aren't free to run at scale, and providers are pricing accordingly.

Production Implications

Model	Best For	Watch Out
GLM-5	General reasoning, coding	Cost
Step 3.5 Flash	High-throughput APIs	Arena scores
MiniMax M2.5	Coding tasks	GQA limits context
Kimi k2.5	Multimodal apps	Early pretraining specifics

What I'm Watching Next

Streaming KV-cache compression for longer contexts
Whether anyone replicates the Kimi early-multimodal pretraining approach
If Step 3.5's multi-token prediction becomes standard

More architecture deep-dives and production AI notes from inside a bank — follow my Telegram channel:

https://t.me/ai_tablet (Russian, technical)

More AI engineering notes, RAG benchmarks, and production insights from inside a bank — follow my Telegram channel:

🚀 https://t.me/ai_tablet (Russian, technical)

RAG SOTA: I Tested 7 Pipelines and Built SEQUOIA (Open Source)

Ai developer — Thu, 28 May 2026 21:13:57 +0000

RAG SOTA: I Tested 7 Pipelines and Built SEQUOIA (Open Source)

After 20+ hours of compute time on local hardware, I benchmarked 7 RAG configurations against real-world tasks. The results surprised me — and changed how I think about retrieval architecture.

Why This Matters

RAG is everywhere in 2026. Everyone claims their pipeline is "SOTA," but most benchmarks use toy datasets. I wanted to see what actually works when you have:

Messy real documents (not clean academic corpora)
A local LLM (slightly weaker than GPT-4)
Production constraints (latency, cost, accuracy tradeoffs)

The 7 Configurations Tested

Method	Approach	Score
No-RAG	Direct LLM generation	Baseline
Classical RAG	Dense retrieval (BGE-small + FAISS)	Poor
Hybrid RAG	BM25 + Dense + RRF fusion + cross-encoder reranker	Moderate
LightRAG	Key-value extraction graph + dense hybrid	Disappointing
PageIndex	Two-stage hierarchical retrieval	Okay
GraphRAG	Entity graph + dense fallback	Complex
Agentic RAG	Multi-step reasoning pipeline	Slow, expensive
SEQUOIA	RAPTOR tree + step-back prompting	Best
SEQUOIA Pro	Multi-query + rerank + compression	SOTA

What Surprised Me

LightRAG underperformed

The Twitter-hyped "graph RAG revolution" didn't hold up on real data. LightRAG produced what I call "procedural warming" — it looks sophisticated but retrieval quality was mediocre. Academic benchmarks ≠ production reality.

Step-back prompting is underrated

Most RAG systems fail because they retrieve on the literal query. Step-back prompting (rewriting the query into a more general form before retrieval) improved recall by ~15% across the board. Combined with RAPTOR tree clustering, it creates a retrieval hierarchy that actually makes sense.

Local LLMs can evaluate

I used a local model for summarization and judging. Slightly weaker than GPT-4, yes, but the relative rankings between methods stayed consistent. This means you can prototype and benchmark without burning API credits.

SEQUOIA Architecture

User Query
    ↓
Step-back Prompting (generalize)
    ↓
RAPTOR Tree Retrieval (hierarchical clusters)
    ↓
Rerank + Context Compression
    ↓
Local LLM Generation

RAPTOR = Recursive Abstractive Processing for Tree-Organized Retrieval. Cluster leaf nodes, summarize upward, retrieve at multiple levels of abstraction.

Step-back = Before searching, ask: "What is the general principle behind this specific question?"

Results

On my test set (banking documents, technical manuals, internal wikis):

Method	Precision	Recall	Latency
Classical RAG	0.62	0.58	120ms
Hybrid RAG	0.71	0.65	340ms
LightRAG	0.59	0.61	890ms
SEQUOIA	0.84	0.79	450ms
SEQUOIA Pro	0.87	0.82	680ms

SEQUOIA Pro trades some latency for accuracy. SEQUOIA (basic) is the sweet spot for production.

Code & Reproducibility

Everything is open source:

🔗 github.com/Diyago/rag-benchmark

All 7 implementations
Evaluation dataset (anonymized)
Configs for local LLM setup
Notebooks for analysis

Lessons for Production

Don't trust academic benchmarks blindly. Test on YOUR data.
Hierarchical retrieval beats flat. RAPTOR's tree structure matches how humans actually organize knowledge.
Query rewriting is free performance. Step-back prompting costs nothing in latency but improves retrieval significantly.
Local evaluation is viable. You don't need GPT-4 to compare methods relatively.

What's Next

I'm extending SEQUOIA with:

Multi-modal retrieval (images + text)
Streaming context compression
Adaptive depth (shallow for simple queries, deep for complex)

More AI Engineering Notes

I write about practical AI/ML from inside a bank — RAG systems, LLM deployment, team management, and what actually works vs. what's just hype.

Telegram channel (Russian, technical): AI.Insaf

Have you benchmarked RAG on real data? What surprised you? Drop a comment or reach out on Telegram.

Эта статья также опубликована в Telegram-канале AI.Insaf — про AI/ML из банковской практики, бенчмарки и управление DS-командами.

Подписывайтесь на канал для оперативных разборов и практических кейсов: https://t.me/ai_tablet