DEV Community: Savindu Thathsara

I Tried to Build an Offline AI Assistant for an 8GB Laptop, and the CPU Said No

Savindu Thathsara — Sun, 25 Oct 2026 11:44:28 +0000

I Tried to Build an Offline AI Assistant for an 8GB Laptop, and the CPU Said No

I kept waiting for the memory to fill up. A question sat there for three seconds, and 13 GB was still free.

I am Savindu Thathsara. I live in London, and this is a Hearthlen note. Hearthlen is an offline AI assistant on a lightweight Linux laptop: you say “hey Hearthlen,” ask something, and a model on that machine answers. Nothing in the question goes to a cloud API. The plan was written for an 8GB laptop. The ThinkPad I am actually building on has 16 GB. That extra room did not save the reply.

What I thought would break

The early budget was a pile of RAM. Leave most of an 8 GB machine for the person and the browser. Keep a wake word always on. Borrow one heavy model while a request is in progress, then let it go. One model at a time, because two of them would spend the same memory twice.

That story is about fitting. The failure I hit was about waiting.

On 25 October 2026 the dashboard read 13.1 GB free, 2.4 GB used, 15.5 GB total. The models on disk were 3.5 GB: Qwen2.5-3B at 2.0 GB, a 1.5B coder at 1.0 GB, SmolLM2 at 0.3 GB, and the Lessac voice at 0.1 GB. The language model was not even loaded until someone spoke. The listener, which is the process holding the microphone and the speech model, was 1507 MB. After I dropped a second language and restarted it, that process was 1331 MB, and 13.0 GB was still available. The machine had room. The answer was still late.

The fast model could not use the sentence

The first voice model was SmolLM2, 360 million parameters. It was quick. “How are you?” came back in 437 ms. “What should we do today?” came back in 356 ms, as a picnic. A joke took 346 ms. A later pass, “Hi, how about you?”, took 857 ms to generate and still sounded like a line from a list.

A model that size can finish a short sentence. Give it “I have two hours free and it is raining” and it does not keep the rain. It picks a pleasant errand and stops. I wrote that down in the small model was the stock answer. Speed was fine. The answer was not.

The model that could think made the CPU do the work

The voice model is now Qwen2.5-3B-Instruct, 4-bit, 2,104,932,768 bytes, running through llama.cpp on the CPU. There is no discrete GPU on this laptop.

I sent the sentence the small model had been failing: “I have two hours free and it is raining. What should I do?” The reply used both facts. It suggested reading a book, because it was raining and there were two free hours. That was 39 tokens in 3017 ms. About thirteen tokens a second. You feel that gap. The face sits on Thinking the whole time, which at least shows the wait is the model and not a frozen window.

The same CPU shows up when the assistant writes a page. Qwen2.5-Coder-1.5B produced a small HTML animation in 8220 ms, inside a turn of 19233 ms. A circle animation took 14347 ms to write, and the whole turn was 26338 ms. A form from one sentence took 48881 ms. None of those runs reported a full memory. They reported a processor chewing through a local model while the person watched.

Speech itself is cheap when the line is short. “How are you?” was transcribed in 373 ms. A short YouTube command was 364 ms. The spoken reply from Piper, including playback of “Hi, Savindu,” was 1939 ms. The long pause is the language model, after the words are already text.

A second language made the same wall taller

I added Sinhala beside English. The speech model had to carry both, and the 3B model had to answer in Sinhala. One question, “ආයුබෝවන්. ඔයා කොහොමද?”, came back in Sinhala in 9564 ms. The grammar was rough, and nine seconds is too long for a greeting. Memory was still the same 13 GB free. I took Sinhala out. English speech is the smaller English-only model again. The note on that measurement is the slow reply was CPU time.

Taking the second language out removes one bill. It does not make the 3B model instant. A question still waits on those tokens. I would rather wait three seconds for an answer that uses the sentence than get a picnic in a third of a second.

What I am keeping

The 8 GB budget is still the target, because this ThinkPad is luckier than the laptop the project is for. Weights of 3.5 GB and a listener over a gigabyte would sit much closer to the edge on 8 GB, and the rule stays: one heavy model at a time, then it leaves. That rule is about room.

The rule the CPU added is about the clock. A local assistant feels intelligent when the model is big enough to hold the sentence, and it feels broken when that model takes long enough that you wonder if it heard you. On this machine those two things arrived together. More RAM was already there. It did not move the 3017 ms.

Next I stay on Qwen2.5-3B for questions and on the small English speech model for the microphone. The budget those numbers still have to fit is an 8GB laptop running YouTube and a local assistant.

Savindu Thathsara builds practical AI and automation in London. This series is the public log of an offline AI operating system he is writing and learning at the same time.

The Slow Reply Was CPU Time

Savindu Thathsara — Sun, 25 Oct 2026 11:34:18 +0000

The Slow Reply Was CPU Time

Thirteen gigabytes of memory were still free, and the answer was still slow.

Savindu Thathsara, savithathsara.me. This is a Hearthlen note from the laptop.

Sinhala had just been added beside English, and a question took long enough to feel stuck. The check was /proc/meminfo: 13.0 GB available out of 15.5 GB. The listener was 1507 MB. The language model was not loaded until someone spoke. So this was not a machine that had run out of memory.

The time was the models doing work. The bilingual speech model spent 2407 ms on a short clip, because it also had to guess the language. A Sinhala reply from Qwen2.5-3B took 9564 ms. English is the only speech path again, on the smaller English model. After that restart the listener was 1331 MB, and 13.0 GB was still free. The 3B model is still what thinks, so a reply is still a few seconds. It is no longer also paying for a second language. The bilingual note is English and Sinhala.

Savindu Thathsara builds practical AI and automation in London. This series is the public log of an offline AI operating system he is writing and learning at the same time.

English and Sinhala

Savindu Thathsara — Sun, 25 Oct 2026 11:17:06 +0000

English and Sinhala

The Sinhala voice said "ආයුබෝවන්. මම හර්ත්ලන්." Inference took 112 milliseconds. The greeting itself was 2.04 seconds of audio.

Savindu Thathsara, savithathsara.me. This is a Hearthlen note from the laptop.

English was the only filled-in language. Sinhala was an empty row in the same manifest. That row now points at a real voice, si_LK-sinhala-medium, and both languages share one multilingual speech model. The wake phrase is still "hey Hearthlen". After it, a sentence in Sinhala script is answered in Sinhala and spoken with that voice. English still uses the Lessac voice.

Asked "ආයුබෝවන්. ඔයා කොහොමද?", Qwen2.5-3B answered in Sinhala script in 9564 ms. The sentence is rough. It is Sinhala, which is what the voice speaks. The dashboard shows EN and සිංහල, and the one just heard is the bright label. The window those labels sit on is the dashboard.

Savindu Thathsara builds practical AI and automation in London. This series is the public log of an offline AI operating system he is writing and learning at the same time.

The Dashboard

Savindu Thathsara — Sun, 25 Oct 2026 11:08:42 +0000

The Dashboard

At 11:07 the Hearthlen window stopped being a text box. The mark was on the left, and the rail next to it said 13.1 GB of memory was free.

Savindu Thathsara, savithathsara.me. This is a Hearthlen note from the laptop.

The mark is the same one that sits in the corner while the voice is open. Rings come in when it is listening, the wire sphere turns while Qwen is generating, and the rings go out when it speaks. Under that, a path names the real stages: the microphone, the mel spectrogram, a 96-number wake embedding, the wake head, speech, Qwen 3B, and the voice. Those 96 numbers are the wake head on disk. They are not a store of your documents. The conversation it can recall is the last few turns.

The numbers are read on the machine. Memory was 13.1 GB free, 2.4 GB used, 15.5 GB total. The model files were 3.5 GB, with 193.9 GB still free on the disk. Qwen2.5-3B was 2.0 GB of that. The listener, the face, the hotkey, and the dashboard were running. The language-model process was not, until the next sentence. The earlier voice is in the small model was the stock answer.

Savindu Thathsara builds practical AI and automation in London. This series is the public log of an offline AI operating system he is writing and learning at the same time.

The Small Model Was the Stock Answer

Savindu Thathsara — Sun, 25 Oct 2026 11:03:36 +0000

The Small Model Was the Stock Answer

"What should we do today?" came back as "go to the market." The words were generated, and they still sounded like a line that had been saved in advance.

Savindu Thathsara, savithathsara.me. This is a Hearthlen note from the laptop.

The voice model was SmolLM2 at 360 million parameters. A model that size can finish a short sentence. It cannot hold the detail you just said and do something with it, so the sentence comes out generic. The voice agent now keeps Qwen2.5-3B-Instruct loaded, 4-bit, 2,104,932,768 bytes. Asked "I have two hours free and it is raining. What should I do?" it answered: "Since it's raining and you have two hours free, why not take this opportunity to read a book?" That was 39 tokens in 3017 ms. It used the rain and the two hours. The smaller model never did.

The face shows Thinking while those tokens are generated. A plain command still goes to the desk. The conversation this replaces is in the reply comes from the model.

Savindu Thathsara builds practical AI and automation in London. This series is the public log of an offline AI operating system he is writing and learning at the same time.

The Reply Comes From the Model

Savindu Thathsara — Sun, 25 Oct 2026 10:46:08 +0000

The Reply Comes From the Model

"What should we do today?" The ThinkPad answered, in the model's own words, "i should probably go to the market today." That sentence took 667 milliseconds to generate.

Savindu Thathsara, savithathsara.me. This is a Hearthlen note from the laptop.

The greeting used to be a stored line. "How are you?" always came back as the same menu. The voice agent now keeps SmolLM2-360M loaded and sends the latest sentence to it, with the last few turns in front. Speech-to-text on this question was 408 ms, and the whole turn was 3840 ms. The log marks it agent=voice. An earlier "Hi, how about you?" came back as "hi, heh, I think I'm ready now." in 857 ms.

A plain request such as "open the browser" still goes to the desk agent, which uses the keyboard and mouse. A sentence that is also a question stays in the conversation. The words show up in the chat while you talk, from the live line.

Savindu Thathsara builds practical AI and automation in London. This series is the public log of an offline AI operating system he is writing and learning at the same time.

The Words Appear While You Talk

Savindu Thathsara — Sun, 25 Oct 2026 10:37:50 +0000

The Words Appear While You Talk

The ThinkPad microphone was already on. A two-second recording peaked at 4976, and the listener's first frames peaked at 2852.

Savindu Thathsara, savithathsara.me. This is a Hearthlen note from the laptop.

The card is the Intel HDA device, codec ALC257. The internal mic boost is at 30 dB, the top of that control. What was missing was the picture. The voice session now opens the Hearthlen chat, and while you are still talking the words land in a "You" line and in the box at the bottom. A spoken "how are you" from this desk became caption 'How are you?' and then heard='How are you?' in 373 ms. At 10:37 the window showed that line, under "Hi, Savindu.", with the reply beneath it.

Room noise can make the transcriber repeat one short phrase. A line that does that is thrown away, and a phrase only starts once it is louder than the noise floor measured at startup, which was 833 this time.

The session those words belong to opens from Win+V.

Savindu Thathsara builds practical AI and automation in London. This series is the public log of an offline AI operating system he is writing and learning at the same time.

Win+V Opens the Voice

Savindu Thathsara — Sun, 25 Oct 2026 10:29:07 +0000

Win+V Opens the Voice

Super+V on the ThinkPad, and the voice agent opened in that same second.

Savindu Thathsara, savithathsara.me. This is a Hearthlen note from the laptop.

Win+H was already the chat window, from Win+H opens the chat. Win+V now asks the listener to start the spoken session, the same one "hey Hearthlen" starts. It says hello and then keeps the microphone. The key went out at 10:28:14. The log line is voice: Win+V at 10:28:14. session open is 10:28:16.

The conversation those keys open is in a conversation, then the work.

Savindu Thathsara builds practical AI and automation in London. This series is the public log of an offline AI operating system he is writing and learning at the same time.

A Conversation, Then the Work

Savindu Thathsara — Sun, 25 Oct 2026 10:23:21 +0000

A Conversation, Then the Work

The wake word answered once and then went quiet. A person does not talk to a command line.

Savindu Thathsara, savithathsara.me. This is a Hearthlen note from the ThinkPad.

After "hey Hearthlen" the microphone stays open. Three agents share the sentence. The voice agent talks, and the small model stays loaded so the next reply does not pay for another startup. The device agent handles the volume, the brightness, files, reminders, and Wi-Fi. The desk agent takes the sentences that start work, create, open, write, and the rest, and it uses the keyboard and mouse on a background thread. The voice says it will do that, and it keeps listening. Eight quiet seconds ends the session.

The voice agent, already loaded, answered three different sentences on the laptop. "how are you" came back in 437 ms. "what should we do today" came back in 356 ms. "tell me a short joke" came back in 346 ms. A recording of the wake phrase from this desk scored 0.25, so that pass never opened the session.

The previous ear is in the wake word trained on this Mac.

Savindu Thathsara builds practical AI and automation in London. This series is the public log of an offline AI operating system he is writing and learning at the same time.

The Wake Word Trained on This Mac

Savindu Thathsara — Sun, 25 Oct 2026 10:11:31 +0000

The Wake Word Trained on This Mac

I said “hey Hearthlen” from the next machine. The ThinkPad logged score 0.99. The sentence before it, “open the browser”, did not wake it.

Savindu Thathsara, savithathsara.me. This is a Hearthlen note from the two computers sitting together.

The earlier listener, in the name it hears, transcribed a burst and then guessed. The same phrase came back as Hertzland, Herthlin, and once as something that was not the name at all. There is no published model for this name. openWakeWord already publishes the mel network and the speech-embedding network, and those stay frozen. I trained only the small phrase head, on this Mac, from offline speech, and copied the three files to the ThinkPad. The listener scores each 80 ms of microphone audio on the laptop. It does not download a model when someone says the name.

The holdout from that training run was positives 0.96, negatives 0.04, recall 0.91, false accepts 0.00. The laptop uses threshold 0.97. The phrase finished at 10:06:48.494 and the log line is that same second, score 0.98. A later pass scored 0.99, said hello, and heard the next sentence: Turn the volume up. Speech-to-text took 367 ms. The turn took 1914 ms. With the speaker turned down to 46, the same phrase scored 0.91 and stayed quiet.

Savindu Thathsara builds practical AI and automation in London. This series is the public log of an offline AI operating system he is writing and learning at the same time.

The Name It Hears

Savindu Thathsara — Sun, 25 Oct 2026 09:19:52 +0000

The Name It Hears

I said “hey Hearthlen” from the next machine. The ThinkPad microphone recorded it, and the wake listener did nothing.

Savindu Thathsara, savithathsara.me. This is a Hearthlen note from the two computers sitting together.

The old listener was PocketSphinx, waiting for “hey hearth” and “hard one”, because that is what an English dictionary makes of a made-up name. On this recording Whisper heard “Hey, Hertzland.” The keyword list, at its most sensitive setting, scored 0 hits.

The listener now transcribes a short burst and accepts the name as it actually comes out: hearthlin, herthlin, hertzland. The burst that woke it took 340 ms. The log line was wake: hey hearthlen, one second after the phrase ended, and it gave the greeting.

Savindu Thathsara builds practical AI and automation in London. This series is the public log of an offline AI operating system he is writing and learning at the same time.

He Asked for an Animation and Got a Form

Savindu Thathsara — Sun, 25 Oct 2026 00:50:46 +0000

He Asked for an Animation and Got a Form

“create html animation” came back as another form, because every HTML sentence was pushed through a form fence, and the animation sentence had been saved as the answer to “what should I type?”

Savindu Thathsara, savithathsara.me. This is a Hearthlen note from the ThinkPad.

The fence is the one in six inputs means six boxes. A 360M model, asked for a page, copies the instruction and invents tags. The fence stops that, and it also means the only page it can produce is a form. An animation does not fit in it.

Two things changed. A sentence that starts a new task is no longer glued onto the previous question. And a page is written by Qwen2.5-Coder 1.5B, on the laptop, then pasted into the editor, saved, and opened. The sentence is the brief. create html animation became a red square that slides, in 19233 ms, of which the coder spent 8220 ms. create a circle animation in html became a red circle that moves, in 26338 ms, of which the coder spent 14347 ms. Chromium is showing /home/savindu/page-3.html.

Savindu Thathsara builds practical AI and automation in London. This series is the public log of an offline AI operating system he is writing and learning at the same time.