Built a local AI pipeline that captures system audio, transcribes it, structures the result with a local LLM, and sends reviewed notes to Obsidian. I also wrote about what broke when real users started testing it.

Ivan — Tue, 29 Sep 2026 18:20:58 +0000

Sep 29

Building Memplua: A Local AI Pipeline for Turning System Audio Into Obsidian Notes

#ai #opensource #education #softwareengineering

5 min read

Building Memplua: A Local AI Pipeline for Turning System Audio Into Obsidian Notes

Ivan — Tue, 29 Sep 2026 11:25:05 +0000

I started building Memplua because I wanted my knowledge base to fill itself. Not literally, of course. The real problem was the gap between consuming information and turning it into something structured. I could watch a technical video, listen to a podcast, or hear something useful during work, but if I wanted that information in Obsidian, I still had to stop, extract the important parts, rewrite them, and organize everything manually.

That was the part I wanted to automate.
The idea behind Memplua is simple: information already passes through your computer, so instead of asking the user to upload every source manually, the application should capture that stream, process it locally, and prepare structured notes for review.
The project is open source and currently focused on Windows.

The pipeline

The current audio pipeline looks roughly like this:

system audio
voice activity detection
transcription
text chunking
local LLM
structured notes
manual review
Obsidian

The main design goal was to keep the system generic. I did not want to build separate integrations for YouTube, browsers, Discord, Telegram, and every other possible source. If the information is already being played through the operating system, system audio gives me one universal input layer.

For the same reason, I currently see Memplua more as a capture-and-interpretation pipeline than as a collection of service-specific connectors.

Capturing audio on Windows

I started with audio because it seemed like the most difficult and useful input type.

System audio capture is highly OS-specific, so I chose Windows first. It is also the platform I use myself, which made debugging much easier.
The capture layer produces raw audio samples. These samples then go through preprocessing and are eventually passed to transcription.

I use Whisper for speech-to-text because sending raw audio directly into a general-purpose LLM would be both inefficient and unnecessary.
Once transcription is working, the problem changes from audio processing to context management.

The chunking problem

Speech transcription arrives as chunks, but language does not care about chunk boundaries.
A phrase like: Docker is a tool may be separated from: for containerization. If the LLM processes both independently, the second fragment becomes almost meaningless.

My first implementation used a floating window: the next group of text reused part of the previous group. This helped preserve context, but it created another problem. The model started seeing the same information several times and generated duplicated notes.

This became especially annoying after I added manual review, because users had to approve multiple versions of the same term or idea.

I eventually changed the strategy and separated the input into two parts:

focus — text from which the model is allowed to extract new information;
context — previous text that can only help interpret the focus section. This sounds like a small change, but it made the pipeline much cleaner. The model can still understand incomplete phrases using previous context without continuously generating entities from the same old text.

Turning text into structured knowledge

The LLM does not return free-form prose. I ask it to produce a structured JSON response that can be parsed by the application. Internally, I currently work with a few simple concepts.
A term is a self-contained entity with a name and definition. A thought connects multiple terms. A group of extracted terms and thoughts becomes a draft note set that can later be reviewed by the user.
For local inference I use llama.cpp, and one of the models I have been testing is Qwen3-4B-Q4.

The important part for me is that inference happens locally. Memplua may process conversations, technical content, personal audio, or other information that I do not want to send to external APIs.
Local inference also keeps the project usable without subscriptions, although it introduces another problem: setup becomes more complicated.

Why I stopped fully automatic writing

My first mental model was very simple:
audio → LLM → Obsidian

In practice, I quickly realized that letting an LLM write directly into a knowledge base is a terrible default.The model can misunderstand something, duplicate information, create bad relationships, or simply produce something that is technically correct but useless.

So I added a mandatory review step. The application prepares a draft, but the user can edit or reject it before anything is saved. This makes the system less magical, but much more trustworthy.

It also changes the purpose of the tool. Memplua is not supposed to replace thinking. It is supposed to remove the repetitive work required to turn consumed information into a usable draft.

VAD became more difficult than expected

I added Silero VAD to remove silence and noise before transcription.
In theory, this should be straightforward: detect whether a chunk contains speech and avoid wasting resources on empty audio.

In practice, I started getting unexpectedly low probabilities even on obvious human speech.

One external tester later reproduced the same behaviour. Their microphone signal had perfectly reasonable amplitude values:
peak=0.55 rms=0.24

but Silero returned:
max_vad_probability=0.0013

The audio was then dropped as if no voice was present. When VAD was disabled, transcription and note generation started working normally.
That narrowed the issue down significantly. The problem is likely somewhere in the VAD input pipeline: sample rate, normalization, window size, tensor shape, state handling, or a combination of those.

For now, this is one of the parts I still consider unfinished.

Then real users started testing it

After I published Memplua, external users immediately found problems that I had missed.

One build was missing a runtime DLL because my own development machine already had the dependency installed. Another bug caused the process to launch while the UI itself remained invisible.

Both problems are now fixed, and I rebuilt the installer so dependencies are handled more reliably. I also added both online and offline installation paths.

The more interesting part was not the bugs themselves, but the quality of the feedback. People started sending logs, testing different configurations, opening GitHub issues, and even helping each other inside issue discussions.

For a small project, that was probably the first moment when Memplua started to feel like an actual open-source project instead of just my personal repository.

Current state

Memplua is still alpha software, but the main pipeline already works.
At the moment, the project has:

Windows system audio capture;
transcription;
local LLM inference;
structured note generation;
manual review and editing;
Obsidian export;
online and offline installers;
fixed UI startup behaviour;
early external testers and active issues.

The next work is mostly about reliability and usability rather than adding more AI features. I want first-run setup to be easier, VAD behaviour to be more predictable, model configuration to be clearer, and local inference presets to require less manual tuning. That is less exciting than adding another model, but probably much more important.

Final thoughts

The most useful thing about releasing Memplua was not getting repository views or stars. It was watching people use the application differently from how I use it. They found bugs I had stopped noticing, questioned UI decisions that seemed obvious to me, and forced me to think more carefully about the boundaries of the project.

That feedback has already changed the application more than another week of development in isolation probably would have.
If you are interested in local AI, Windows audio processing, Obsidian, or local inference, feedback is welcome.
GitHub: https://github.com/Ijne/memplua
Website: https://memplua.space

DEV Community: Ivan

Built a local AI pipeline that captures system audio, transcribes it, structures the result with a local LLM, and sends reviewed notes to Obsidian. I also wrote about what broke when real users started testing it.

Building Memplua: A Local AI Pipeline for Turning System Audio Into Obsidian Notes

Building Memplua: A Local AI Pipeline for Turning System Audio Into Obsidian Notes

The pipeline

Capturing audio on Windows

The chunking problem

Turning text into structured knowledge

Why I stopped fully automatic writing

VAD became more difficult than expected

Then real users started testing it

Current state

Final thoughts