DEV Community: Szilard Galambos

I Built a RAG Assistant for My Own Field, So I Could Tell When It Was Wrong

Szilard Galambos — Fri, 28 Aug 2026 16:49:17 +0000

I wanted to understand RAG properly, and reading about a technology is not the same as building one. Building one means choosing a document to build it on, and that choice turned out to matter more than I expected.

I took a document from my own field: a roughly 170-page design guide for engineering plastics and injection moulding, the kind of reference I spent twenty years working alongside. That did two jobs at once. The result is useful to the people I come from, rather than a demonstration over a corpus nobody needed searched. More importantly, I could judge whether an answer was actually correct, which is exactly what you cannot do on material you do not know.

That second point turned the project into an exercise in measurement rather than a demo, and it is why the things the measurements turned up are the most useful part of the story.

First, what RAG is

One definition, because everything below depends on it. RAG, retrieval-augmented generation, means the language model does not answer from memory. Each question first triggers a search across your own documents, and the passages that come back are handed to the language model together with the question, with an instruction to answer from those and nothing else.

Two things follow. Every answer can point at the part of the guide it came from, and anything the guide does not cover produces a refusal instead of an invention. The second consequence matters more: once the language model may only use what the search hands it, the search decides the quality of the answer. A capable language model given the wrong three paragraphs will not notice; it will simply write a more fluent wrong answer than a weaker one would. That is why most of what follows concerns the search rather than the language model.

How the search works, and what the system is made of

Searching by meaning rather than by wording requires both sides of the comparison to exist in the same form. An embedding model provides that form: it converts a piece of text into a list of numbers arranged so that passages about the same topic land close to each other. The question goes through the same conversion, and the search then reduces to finding the stored pieces nearest to it.

Because the guide has to be converted before anyone can ask anything, the system falls into two phases: once per document the guide is cut into pieces, each is embedded and stored, and then on every question the question is embedded and matched against that store. The same embedding model must serve both phases, since vectors from different embedding models cannot be compared.

Those pieces are called chunks: 223 from the body text, plus 185 figure entries and 17 table entries, held in three collections in Qdrant, a database built for nearest-neighbour lookup. Keeping figures and tables apart mattered later, when the chat model had to be told which kind of content it was looking at. That chat model, Qwen3.5-9B, is the language model that writes the answers, and it runs alongside the embedding model on llama.cpp on a single GPU. Everything stays on that machine, which is not a technical footnote: a licensed design guide is not something you upload to a third party to make it searchable.

One component sits in front of all this, because not every message is a design question and not every design question is covered by the guide. A classifier separates greetings and help requests from content questions, then a similarity threshold decides whether the nearest chunks are close enough to answer from.

What an engineer gets out of it

None of that machinery is visible to the person using it. A designer types a concrete question, such as how thick a wall can be before the surface sinks, and the assistant answers from the guide while naming the section, figure or table it used. That citation is the point rather than a nicety, because an engineer about to commit a wall thickness to a steel mould has to check where the number came from. The refusal works the same way from the other direction: a confident answer to something the guide never addresses would be worse than none.

So the value is not that the assistant saves you from reading 170 pages. It is that it finds the right two of those pages in seconds and shows you which two.

Round 1: where the guide gets cut

The first link in that chain is the cutting, and it is easy to underestimate. A chunk spanning two unrelated subsections produces a vector that represents neither properly, and no amount of search quality afterwards recovers what the cut destroyed.

Hierarchy-aware chunking follows the guide's own chapters and subheadings, so a boundary lands where the guide itself changes subject. The naive alternative cuts every N characters regardless. Which of the two wins was never really in doubt, so I did not build the naive pipeline to find that out. I built it to see the size of the difference, because "structure helps" is an intuition, and a number is something you can act on.

The naive run produced 137 chunks against 223, and inspection showed why that mattered: 86.1% bridged two or more unrelated subsections, 77.4% began mid-sentence, and 31.4% exceeded the embedding model's token limit. These are not cosmetic flaws but chunks that do not correspond to a single coherent idea, which is exactly what the vector is meant to capture.

The retrieval scores followed. One definition first, since these numbers recur: retrieval works like asking a librarian for the right page, so Recall@1 measures how often the first suggestion is correct, and Recall@5 counts a hit anywhere in the top five.

Recall@1: 0.843 for hierarchy-aware chunking against 0.708 for naive chunking, a gap of 13.5 percentage points.

Round 2: which chat model follows the rules

Putting the right chunks in front of the chat model does not guarantee that it stays inside them. Twice it did not: once inventing a formula for a table that exists in the guide only as a drawn image, and once claiming a figure was absent when it was plainly there. Both were settled with prompt rules that tell the model what to do when the context is incomplete instead of letting it improvise.

Those rules are only worth writing if the chat model follows them consistently, which turned the next question into one about the model itself. My assumption was that the larger Qwen3-14B would be the safer default and the newer, smaller Qwen3.5-9B the speed option.

Repeated runs said otherwise. At one prompt configuration the 14B model fell back into the already-fixed hallucination on 5 out of 5 attempts, while the 9B held steady across both of its checked runs. The deciding axis was therefore not raw capability but reliability under repetition, and the smaller model shipped.

Round 3: the questions I had not been asking

Every improvement so far was measured with questions written in the guide's own vocabulary, which is a flattering test, because that is the one condition the system was tuned for. Real users do not write that way.

Rephrased as a non-expert would type them, using "why do parts get weird dents" instead of the formal defect name, Recall@5 fell from 1.000 to 0.400. The content was unambiguously in the guide, so the failure lay entirely in the matching, and widening the search did not recover it. The problem is therefore not how many candidates are considered but how the question is represented in the first place.

That leaves a documented limitation rather than a solved one. The obvious next steps stop relying on meaning-matching alone: rewriting the question into formal terms before searching, adding keyword matching to catch a specialist's exact wording, or re-ranking the closest candidates in a second pass.

How all of this was measured

None of those findings would have surfaced without something to measure against, so the evaluation ran on three levels.

Retrieval was scored across 425 synthetic questions, giving Recall@1 of 0.899, Recall@5 of 0.995 and an MRR of 0.943, where MRR rewards the correct answer for landing near the top rather than merely appearing somewhere.

Answer quality was scored by an external LLM-as-Judge, deliberately not the chat model that writes the answers, since a language model grading its own output is not evidence.

Faithfulness at 4.44 and conciseness at 4.08 sit lowest, consistent with the hallucination work above, while relevance at 4.96 sits near the ceiling, which is what good retrieval should produce.

End to end, 10 of 10 scripted user journeys passed at roughly 18 seconds per answer, a figure that tracks answer length rather than slowness. Three simulated personas of rising expertise were then scored for satisfaction.

The most expert persona was the most satisfied, reassuring precisely because that was the expected direction. Several of these figures rest on small samples, which is worth naming: 5 runs against 2 in the chat-model comparison, three personas here. They are directional evidence, enough to act on but not proof at any rigorous confidence level.

Metric	Result
Recall@1, hierarchy-aware against naive chunking (body-text subset)	0.843 against 0.708
Recall@1 / Recall@5 / MRR (all 425 questions)	0.899 / 0.995 / 0.943
Recall@5, formal against colloquial phrasing	1.000 against 0.400
LLM-as-Judge, 1 to 5: faithfulness / relevance / completeness / conciseness	4.44 / 4.96 / 4.64 / 4.08
User journeys passed, average response time	10 of 10, roughly 18 s
Persona satisfaction, 1 to 5 (novice / intermediate / expert)	3.50 / 3.83 / 4.67
Runs that fell back into the fixed hallucination, one prompt setup	Qwen3-14B 5 of 5, Qwen3.5-9B 0 of 2

Key takeaways

Where a document gets cut decides how well it can ever be searched. The 13.5-point gap came from nothing more than respecting the guide's own outline.

A bigger language model is not automatically a safer one. Reliability under repetition proved worth more than raw capability.

A system tested only in its own vocabulary looks better than it is. The 60-point collapse on plain-language phrasing is invisible to any demo, because demos are written by whoever built the thing.

Conclusion

Building on ground I already knew is what made the measurement possible. On an unfamiliar document I could have collected exactly the same numbers and still not known which answers were right, and every finding above came from being able to tell the difference.

The colloquial-phrasing gap is still open, and it is the first thread I would pull next.

Side note: a multilingual variant exists, but it is not the subject here. It depends on a cloud translation service, which breaks the fully local property everything above rests on, and it has not been through this evaluation. Every number here refers to the local, English-language system.

About the Author

Szilárd Galambos spent 20 years as a mechanical engineering group lead at Robert Bosch, and is currently on a deliberate career break to build expertise in data science and AI. With a background in engineering mathematics and hands-on experience in n8n workflow automation, Linux server administration, and AI integration, he bridges the gap between traditional engineering thinking and modern data-driven approaches.

Connect on LinkedIn.

I Made Local AI Faster Than the Cloud - A Complete Home Automation Voice Control Journey

Szilard Galambos — Thu, 28 May 2026 08:59:21 +0000

What if your home could understand you, without sending a single word to the cloud?

That question started this project. I wanted to control my smart home with voice commands in Hungarian, a language that sits far outside the English-centric comfort zone of most voice assistants. I wanted context awareness: the system should know which lights are already on, what time of day it is. And I wanted it to be private: no audio recordings uploaded to someone else's servers, no device state telemetry leaving my network.

What I did not expect was that the journey from cloud to local AI would end with my local setup outperforming the cloud version. This is the full story, with the raw numbers to prove it.

The Problem and Motivation

The cloud version worked. Groq's Whisper API transcribed Hungarian speech reliably, OpenAI's GPT interpreted the commands, and my lights responded in about four seconds. But four seconds is actually the good news. The bad news is in the variance: the same system took anywhere from 2.7 to 9.2 seconds depending on cloud load and network conditions. On a bad day, it felt slow. On a very bad day - like the one data point at 9.2 seconds - it felt broken.

More fundamentally, I was uncomfortable with what was being sent out. Every voice command I spoke, along with the full list of my smart home devices (names, locations, current states), went to Groq and OpenAI. That is not a privacy disaster, but it is a privacy trade-off I did not need to make.

The other motivation was simply learning. I worked as a mechanical engineering group lead and I am using a career break to build hands-on AI and data science skills. Running local LLMs and STT models myself, understanding where the bottlenecks are, benchmarking performance, this was exactly the kind of project that teaches things you cannot learn from tutorials alone.

System Architecture

The setup spans two machines on a wired home LAN.

The Home Server is a passive-cooled Intel Celeron N3150 box running Debian 12. It has no GPU, runs 24/7, and hosts the orchestration layer: n8n for workflow automation, Domoticz as the smart home controller, and a Mosquitto MQTT broker. Think of it as the brain that coordinates but never does heavy computation.

The Desktop PC is an Intel Core i7-4770 machine running Ubuntu 22.04. This is the AI inference machine. Its GPU changed over the course of the project, first a GTX 1050 Ti with 4 GB VRAM, later an RTX 4060 Ti with 16 GB, and that GPU upgrade is the turning point of the story.

Here is what happens when I press record on my phone:

I speak a Hungarian command into the Webhook Audio Recorder app, which sends the audio file via HTTP POST to n8n
n8n sends the audio to faster-whisper for speech-to-text transcription
In parallel, n8n queries Domoticz for the current device list and their states
The transcribed text, device list, and current time are passed to Ollama (Qwen2.5:7b), which interprets the command and returns a JSON control payload
n8n publishes that JSON to the MQTT broker
Domoticz receives the MQTT message and executes the command, lights go on, blinds move

The AI models I used throughout: Qwen2.5:7b (Q4_K_M quantization, 4.7 GB) for language understanding and JSON generation, and Systran/faster-whisper-small (~500 MB) for Hungarian speech recognition.

Version 1 - Cloud Baseline

The cloud version was straightforward to set up. In n8n, an HTTP Request node calls the Groq Whisper API with the audio file, and an OpenAI Chat Model node handles the LLM side. Domoticz provides the device list, the workflow builds a system prompt, and the AI returns a JSON array of commands.

It worked well. Both the STT and the LLM coped with Hungarian syntax and device names without special tuning, better than I expected. The median end-to-end latency across 21 test runs was 4.0 seconds.

The catch: that 4.0 seconds is the median, not the ceiling. The cloud had a wide spread. OpenAI's response time ranged from 1.6 to 8.2 seconds in my measurements, dragging the total anywhere from 2.7 to 9.2 seconds. Cloud services have their own load and queuing behavior, and my home automation latency was subject to it.

The other catches: cost (paid API subscriptions), internet dependency (no voice control during outages), and the privacy trade-off described above.

Version 2 - Going Local with GTX 1050 Ti

The GTX 1050 Ti has 4 GB of VRAM. That sounds like enough, the Qwen2.5:7b model is 4.7 GB in Q4_K_M quantization. It is not enough.

Ollama loaded approximately 24 of the model's 29 layers into VRAM (~3,500 MiB used). The remaining 5 layers ran on CPU and RAM. This hybrid mode works, but it means every inference cycle crosses the VRAM/RAM boundary repeatedly. The LLM ran at about 3,100 ms per request in warm state, measurable, but acceptable.

The real problem was faster-whisper. After Ollama took 3,500 of the 4,096 MiB available, there was only ~535 MiB of free VRAM left, not enough for the faster-whisper model. I tried the CUDA image anyway and got an immediate "CUDA out of memory" error. There was no other option: faster-whisper ran on CPU.

On this machine, CPU-mode STT took about 2,800–3,500 ms per request. That single constraint, no room in VRAM for the second model, doubled the latency of every request.

The first measurement run with both models running showed a median end-to-end time of 13.3 seconds. Usable, but not satisfying.

The KEEP_ALIVE Discovery

Then I found the single configuration change that cut the response time nearly in half.

By default, Ollama loads the model into VRAM on the first request and unloads it after 5 minutes of inactivity. Every "cold" request, the first one after a quiet period, paid a ~12 second loading penalty. Setting OLLAMA_KEEP_ALIVE=-1 keeps the model permanently resident in VRAM.

With static loading, the median end-to-end latency dropped to 6.9 seconds. Same hardware, same models, one environment variable. The lesson: configuration matters as much as hardware.

The trade-off is that VRAM stays permanently occupied. On the GTX 1050 Ti, that meant zero headroom for any other GPU workload. On a 16 GB card, it would not be a concern.

Version 3 - GPU Upgrade, RTX 4060 Ti

The GTX 1050 Ti taught me that the bottleneck was VRAM, not the CPU. The RTX 4060 Ti has 16 GB. That changes everything.

With 16 GB available, both models fit comfortably on the GPU simultaneously:

The LLM loaded all 29/29 layers into VRAM, confirmed in the Ollama logs:

load_tensors: offloading 28 repeating layers to GPU
load_tensors: offloading output layer to GPU
load_tensors: offloaded 29/29 layers to GPU
load_tensors:        CUDA0 model buffer size =  4168.09 MiB

faster-whisper moved from the CPU image to the CUDA image, and VRAM allocation after both models are loaded: Ollama at 4,892 MiB, faster-whisper at 754 MiB, total 5,654 MiB, leaving 10,426 MiB free. The card is barely breaking a sweat.

The results were immediate. GPU-mode STT dropped from ~2,800 ms to 279 ms (static mode, from standalone benchmark), a 10x speedup. LLM inference dropped from ~3,100 ms to 586 ms (static mode), a 5x speedup. With static loading enabled, the median end-to-end latency from the n8n measurements was 1.6 seconds.

The cloud baseline was 4.0 seconds. Local AI, on hardware I already owned plus a mid-range GPU upgrade, is now 2.4× faster.

Benchmark Results

All measurements come from real n8n workflow runs, not synthetic benchmarks. The workflow measured the actual time between sending the audio file and receiving the JSON command back, including all network hops between Home Server and Desktop PC.

Full statistics from the raw JSONL data:

Configuration	n	STT median	LLM median	Total median	Total range
Cloud (Groq + OpenAI)	21	0.44 s	2.98 s	4.0 s	2.7 – 9.2 s
GTX 1050 Ti · dynamic LLM	17	3.54 s	9.26 s	13.3 s	13.2 – 14.3 s
GTX 1050 Ti · static LLM	16	2.76 s	3.48 s	6.9 s	6.7 – 7.5 s
RTX 4060 Ti · dynamic LLM	16	0.86 s	2.97 s	4.4 s	4.2 – 4.9 s
RTX 4060 Ti · static LLM	58	0.34 s	0.82 s	1.6 s	1.5 – 2.1 s

A few things stand out:

Cloud variance is real. The local GTX configurations had extremely tight variance, the GTX dynamic spread was only 1.1 seconds across 17 measurements. The cloud had a 6.5-second spread. A home automation command that might take 3 seconds or 9 seconds is a different user experience than one that reliably takes 6–7 seconds.

The RTX dynamic mode is interesting. With the RTX 4060 Ti but without static loading, the LLM median was 2.97 seconds, nearly identical to the cloud's 2.98 seconds. The GPU is fast enough that even with model loading overhead amortized across a few requests, you are in the same ballpark as cloud. Enable static loading and you leave cloud performance far behind.

The ~0.5 second overhead is consistent. Across all five configurations, the difference between (STT + LLM) and the total end-to-end time was 0.47–0.63 seconds. That is the n8n workflow overhead plus the local network round-trip. It does not scale with model speed, it is a fixed cost.

Component-Level Numbers

Metric	GTX 1050 Ti	RTX 4060 Ti	Speedup
STT (faster-whisper-small)	2,957 ms (CPU)	279 ms (GPU)	~10.6×
LLM static (Qwen2.5:7b)	3,079 ms (hybrid)	586 ms (full GPU)	~5.3×
VRAM used (both models)	~3,500 MiB / 4,096 total	5,654 MiB / 16,380 total	—

Component times from direct benchmark scripts; end-to-end totals from n8n measurement JSONL files.

Key Takeaways

VRAM is the main bottleneck, not the model. The same Qwen2.5:7b model ran in 3,100 ms on GTX (hybrid mode) and 586 ms on RTX (full GPU). The model did not change. The hardware headroom did.

Configuration matters as much as hardware. The single OLLAMA_KEEP_ALIVE=-1 setting cut response time from 13.3 to 6.9 seconds on the GTX, without any hardware change. If you are running Ollama and wondering why it feels slow, check this setting first.

Local AI can beat cloud with the right setup. The RTX 4060 Ti with static loading achieves 1.6 seconds median end-to-end. Cloud median was 4.0 seconds. Local is 2.4× faster, and far more consistent.

Privacy is not a trade-off here. Every voice command, every device state query, every AI inference step stays on the local network. Nothing leaves the house. This is not "good enough for a home project" privacy, it is architecturally private by design.

Open-source models handle minority languages better than expected. Qwen2.5:7b correctly interpreted Hungarian voice commands and in most cases generated valid JSON control payloads across all test configurations. faster-whisper-small transcribed Hungarian speech accurately enough for a smart home context. Neither model was fine-tuned for Hungarian, they work out of the box.

Conclusion

This started as a learning project with modest ambitions: replace cloud APIs with local models, see how the numbers compare, write it up. It ended with a home automation system that responds to Hungarian voice commands in 1.6 seconds, runs entirely offline, and costs nothing per query.

The hardware path matters. A 4 GB GPU creates forced trade-offs; a 16 GB GPU removes them. But the path from 4 GB to 16 GB taught me more about bottlenecks, configuration, and the gap between "it runs" and "it runs well" than any tutorial could.

If you are thinking about building something similar: start with whatever hardware you have. The constraints will teach you something. Then upgrade only what the data tells you to.

If you have questions, suggestions, or a similar build of your own, I would love to hear about it in the comments.

About the Author

Interests: home automation, AI-powered workflows, and making technology work in the real world.