Over the last 18 months our ML team has been doing some very interesting things: building AI agents on top of PostgreSQL, while the infrastructure evolves, the industry matures, and quality expectations keep rising. We started with a single A100 in a managed cloud and fairly modest tasks like "let's plug in RAG." Today we simultaneously support a production assistant, build analyst agents, run experiments, create our own benchmarks for agents, and get ready to move to a server with 8x H200, where you can already run Qwen3-235B and do serious fine-tuning work on smaller models.
This article is about the tasks we solve, how our hardware, models, and frameworks evolved, and why we ended up caring much less about which model is "best" and much more about context management and agent architecture. Everything described is current as of late 2025 to early 2026.
The tasks that shaped our stack
Around a single LLM infrastructure, a whole suite of applications has grown. To make this concrete, here are the main directions:
Classic RAG Fusion over documentation and other sources
We index internal documentation, articles, Q&A pairs, SQL examples, books, and other text into pgvector. We use BGE-M3 for embeddings and work not with BM25, but with sparse vectors: we build sparse text representations, compute dense vectors, and combine everything via multiplication, ranking, discounting, and additional heuristics. Some unstructured-source processing is delegated to the LLM.Evolving RAG into an MCP agent with web search and SQL executor
A classic docs chat stopped being enough: users need a full assistant with a clear role model and many tools. So RAG grew into a ReAct agent that, via Model Context Protocol (MCP), can talk to PostgreSQL, call web search, and combine sources in one reasoning chain.An AI agent for analytics
This is an interface over analytics data: the agent understands questions like "show revenue by region for the last year," generates SQL, explains results, can return to previous steps, build charts, and gradually increase query complexity.Graph-RAG agent over a large C codebase (AI copilot)
The PostgreSQL core and related projects are millions of lines of C. We build a knowledge graph in Apache AGE: vertices are directories, files, functions, structs, macros, variables; edges are dependencies and calls. The agent answers questions about where behavior is implemented, what changes might break, and helps navigate code as a knowledge base instead of raw text search.SQL generator
A full text-to-SQL pipeline that we train and evaluate with EX/EM metrics and benchmarks like Spider/BIRD plus our own schemas. For training we used Qwen-0.6B with GRPO/SFT; we also experimented with QLoRA fine-tuning of models around Qwen2.5-14B on pure SFT.Hint set generator for PostgreSQL
The center here is the pg_hint_plan extension. The model sees a query, stats, and indexes, and proposes hints in hint-set format. We train it with GRPO and constrain the output with a formal grammar so pg_hint_plan can parse and apply it.DB schema and business logic generator from DSL
We use a DSL (domain-specific language) that describes not only entities but also a transaction schema for schema mutations. The schema is represented as aggregates with a hierarchical data structure in JTD (JSON Typedef). The agent generates tables, relationships, transaction wrappers, and business-logic fragments while following the JTD specification.Test data generator
Under the hood it is a "simple" INSERT generator with nuance. The agent pulls plausible value lists (cities, names, addresses), accounts for relationships between tables, and constrains output with a formal grammar for easy validation and replay.Support-ticket summarization and structuring
The goal is not just to retell a conversation but to structure it: who did what, which hypotheses were tested, what commands were run, and what the final outcome was. The agent also classifies tickets by existing labels.Scanning large codebases for suspicious fragments
The LLM sifts large code corpora and flags fragments that look suspicious by predefined criteria. It is a tool for speeding up manual audits, not replacing them.Our own agent benchmarks
We arrived at a scheme: "tested agent + testing agent + validator." The first solves the task as it would in production, using tools. The second plays the user, asking and clarifying based on a limited subset of context. The third (validator) analyzes the dialogue and decides whether the goal is achieved and the test conditions are met. This tests system behavior, not model IQ.Memory and formal constraint research
In parallel we explore short-term and long-term agent memory, context management (from a simple window to summarization, masking, and graphs), and how formal grammars and strict structured output affect quality, speed, and stability.
Throughout the article I will keep referring back to these tasks — this is what drove our hardware, models, and frameworks.
First stack iteration: 1x A100, RAG, LangChain, and Ollama → vLLM
Our resource story is quite classic. We started with one A100 80 GB in a managed cloud, and it had to cover experiments and product demos. It was enough to run 30-70B models in Q6_K_M quantization via Ollama, spin up simple RAG cases, and build early SQL-generator prototypes.
Later we got an internal server with two A100s. It made life much easier: production runs there, multiple environments exist, we keep stable pipelines and run experiments in parallel. At that point it became clear that Ollama is great for a fast start, but for production we needed a more controllable and efficient inference engine. That's how we ended up with vLLM.
But why vLLM instead of TensorRT-LLM or SGLang, which seem to have higher peak performance?
Not necessarily in your case. The internet is full of charts and tables that either contradict each other (because they are built on different hardware, different models, at different times) or show relative parity on average:
In short, it is the usual benchmark wars. So the choice depends not on a number in a chart, but on the framework's applicability to our specific case.
What did we need? Support for structured output and context-free grammars — all three engines support that. Support for many different models from Hugging Face — here TensorRT-LLM starts to lose because it supports a limited set of families and requires compiling the model when switching, which slows and complicates R&D. We also wanted CPU inference, which exists in vLLM and SGLang but not in TensorRT-LLM. In the end, the final choice between vLLM and SGLang came down to community size — vLLM won.
We are now expecting a server with 8x H200, which can be considered a real platform for the heaviest models. It becomes realistic to run Qwen3-235B and its neighbors, and to launch systematic training activity for small models for specific tasks: SQL/Cypher/DSL generators, semantic classifiers, vectorizers, rerankers, summarizers, log analyzers, and so on. We could do this on A100 too, but at the cost of aggressive compromises in quantization, model size, and training time.
As for frameworks, we started with the simplest set: LangChain, LangGraph, LangFuse, RAGAS, and several Qwen2.5 models. We needed Russian and English, so we quickly dropped parts of the LLaMA and Gemma families: for our tasks and corpora they noticeably lost to Qwen in Russian, and prompting/instruction tuning was harder. At that stage our scheme was: LangChain for RAG, LangGraph for early agent graphs on Qwen. With that stack we built a prototype assistant, tested baseline RAG, SQL generator, and early experimental "generator + critic + validator" pipelines. We used RAGAS to evaluate iterative changes in RAG - both pipeline/prompt/data changes and base model changes. Target metrics were answer_relevancy and answer_correctness; based on them we decided whether a change helped or hurt:
In parallel, our retrieval approach matured: instead of the usual BM25 + dense vectors combo, we bet on BGE-M3 with its dense + sparse representations and started treating sparse vectors as the full-text part of search. For our texts this was more convenient and easier to operate.
SQL generator and hint sets: where fine-tuning is truly justified
"We fine-tuned the model" sounds proud. But it is easy to forget that for most applied tasks, fine-tuning large models is more luxury than necessity. Our practical experience is pretty down to earth.
For the 80B class (and especially for models like Qwen3-235B) the typical task set — RAG, SQL, code, tools — is already solved out of the box. About 90% of the problems are fixed by:
- solid context management;
- decent RAG (BGE-M3, good indexing, clean parsing);
- pipeline architecture (generator, critic, validator, SGR);
- strict structured output.
Fine-tuning is justified for us in two cases. First, when we need a small model for a specific scenario, for example on-prem for a client with no powerful GPUs. Second, when we need a very specific output format that is hard to get reliably with prompts alone.
Technically we use the classic toolbox: we tried PEFT/LoRA/QLoRA, LLaMA-Factory and LMPO, but we ended up with a de facto standard — TRL for experiments with SFT and RL approaches (GRPO, GSPO, and similar). For complex tasks like the SQL generator we prefer to test methodology on small models (for example, Qwen-0.6B) and only then transfer selected trajectories and data to larger models.
The main filter is simple: if we do not have compute, a proper benchmark with metrics, or data for the task, we do not fine-tune for that task. First tune through context, then fine-tune.
The SQL generator was the first place where we thought: "Okay, here fine-tuning is worth it". We wanted the model not only to produce valid SQL, but to behave predictably, understand PostgreSQL dialect nuances, and do some schema linking during reasoning on typical user queries.
As a test we used Qwen-0.6B: we ran SFT and GRPO with LoRA and learned how to design rewards. This improved quality on complex queries and showed that such pipelines are justified on small models. We also trained Qwen2.5-14B with QLoRA on pure SFT. The result improved format stability, but did not add new knowledge, which showed up clearly on benchmarks.
The hint-set generator grew from the same logic, but with stricter constraints. Here pg_hint_plan is central, and any hint format mistake can make the planner ignore the hint. So we immediately constrained output with a formal grammar and trained with GRPO, carefully selecting hints that actually speed up queries.
It is worth noting that there are many RL algorithms, each with pros and cons. For example, we tried the relatively new GSPO and found it fits MoE architectures better, while GRPO converged faster on small dense models.
DSL, JTD, and test data generation
A separate branch is work with DSL. We needed a description language that declares not only a data schema but also a transaction scheme for mutations: what operations are possible, what invariants must hold, what aggregates exist. We described the structure in JTD (JSON Typedef) terms: that linked hierarchical aggregates, business invariants, and transaction logic. The agent working with this DSL acts as a translator from business language to a formal language, can ask clarifying questions, and generate required mutations if needed. You can read more about the DSL itself on Habr.
Another more down-to-earth pipeline is a test data generator. It receives a schema and requirements, and outputs a set of INSERTs. If needed, the agent pulls plausible values (lists of cities, names, domains, and so on), tracks relationships and edge cases. Everything it outputs is constrained by a formal grammar, so generation is always syntactically correct and valid for later stages of testing.
From a "zoo" of pipelines to agents on MCP
While all these components lived separately, life was not easy. RAG had its own pipeline, SQL had its own, Graph-RAG over code had another. Each had its own DB connections, its own log formats, and its own idea of what context means.
The move to MCP brought long-awaited order to infrastructure and minds. It became much easier to explain how agents work with tools, what abstraction layers exist, and how to extend them. We started describing access to data and tools in terms of MCP servers, and built ReAct agents on top of that. As a result we got a unified layer where:
- PostgreSQL, the file system, web search, graph store, and internal APIs all look the same as a set of tools with clear contracts;
- Different agents can reuse the same set of MCP servers without copy-paste and a "zoo" of integrations.
Against this backdrop, model behavior as an agent became especially important: how carefully it calls tools, how it reacts to errors, whether it can honestly admit missing data. For a start you can use leaderboards like this, but you also need to evaluate on your own tasks. That pushed us to create our own benchmark with the trio "tested agent — testing agent — validator". We needed more than a "good model"; we needed a system that behaves predictably in a long reasoning trajectory.
Current stack: vLLM, Qwen3-Next-80B, and our own agent layer
Today the stack looks roughly like this.
At the bottom sits vLLM as the main inference engine for open-source models. Next is an OpenAI-compatible MCP client that handles streaming, connects to MCP servers, manages the agent loop, and ensures all needed tools are called and all context is passed to the LLM as intended. We deliberately moved away from building complex scenarios entirely inside LangChain/LangGraph, keeping them for quick prototyping. There is a reason: they are still rough and have many bugs in production scenarios. We also built our own logging and analytics on PostgreSQL. We previously used LangFuse, but only until we found a memory leak that took down production several times a week.
It is worth noting that we share resources between teams via LiteLLM: a convenient gateway with an OpenAI-compatible API that can route requests to different models - local or external - as needed.
Tools and data sources live in a separate layer of MCP servers. This lets us connect and disconnect servers without rewriting agent logic and share them across products/teams.
In this configuration, load testing in a multi-user mode gives an average of 18 seconds per response when using the docs search tool, with p95 around 60 seconds, which is more than good enough for our context.
Quantization and its character
In practice, quantization boiled down to a fairly simple picture.
While we lived on A100, AWQ-4bit became the working norm for large models. In this form, Qwen3-Next-80B-A3B fits on the card, shows solid quality on our tasks, and allows building multi-stage agents without inflating latency to ugly levels.
We tried more aggressive modes, but quickly saw the cost of errors go up: the model more often drifts into the wrong language, loops on template phrases, and breaks tool-call formats (this still happens sometimes). Some of this is fixed by prompts and sampling parameters (temperature, top-p, top-k), but the truly reliable combo appears only when you wrap the model with formal grammars and strict output specs.
With H200 we will look more actively at FP8 quantization and larger models, but the philosophy will not change: quantization is a trade-off between quality and speed. We will choose based on benchmarking and measurable criteria, not excluding latency.
What we learned about model selection
If we compress all the experience into one thesis, it is this: the model matters, but context and architecture matter more.
Yes, if you take gpt-5.2 or Gemini Pro 3.0, they will be objectively smarter than any open analog. But if an agent cannot manage context, has no memory, and lacks solid tool orchestration, no "mega-trillion-parameter model" will save the situation.
We went from Qwen2.5-14B/32B in Q6_K_M via Ollama to Qwen3-Next-80B-A3B AWQ-4bit on vLLM, and now we are looking at Qwen3-235B on H200. At every step it was possible to build a useful agent, provided the rest of the system was there: RAG, MCP, an agent layer, memory, grammars, and benchmarks.
So now, when choosing an LLM for a new task, we first ask not which model is in its prime. We care about:
- which data and context sources we need;
- how we will manage that context;
- which tools the agent needs, and in what form;
- how we will measure behavior quality.
And the model is just a component embedded into this system. Even if it is gpt-5.2, without context and architecture nothing will move.
Top comments (0)