This is a submission for DEV's Summer Bug Smash: Smash Stories powered by Sentry.
Overview
Sometimes we discover our limitations throug...
For further actions, you may consider blocking this person and/or reporting abuse
Hi Anna! 😸 I’ve been following your Catbot project, and I just had to say how much I loved this article.
Even though I’m not a deep-system developer myself, your explanation of the "Helen Keller" analogy for how the audio pipeline and the LLM work together was absolutely brilliant. It is so rare to find a developer who can take a complex, frustrating technical hurdle and explain it with such creativity, clarity, and humility.
Thank you for building something so thoughtful and for sharing your problem-solving journey so openly. Reading about how you tackled this made my day, and it’s truly inspiring to see such dedication to making local, private AI work beautifully. Catbot is lucky to have you!
🐾🌊🗻🧊😁😸
Thank you so much! Always good to hear from you. And yes when I made that connection I felt it to be a shareable lightbulb moment, a way to communicate AI to anyone at any level. I found a high value in that. 🦄✨️
Wow! Now Catbot can understand its own name! Word recognition seems really difficult to implement. Great bug smash! 😺
Thanks! This was a challenging one. This particular journey exposed me to alot of areas that beg for deep investigation. So interesting 🤔
The Vosk grammar split is a nice catch. I like that you treated the wake word as a constrained recognizer problem instead of trying to bully the general recognizer into learning a made-up token. The boring hardware note matters too. A better mic fixing half the bug is very much how voice pipelines usually humble us.
I was at first getting frustrated because I did not understand why catbot was not understanding. Taking a deeper dive and learning about the limitations of a fixed vocabulary (so to speak) and why some things work and others do not makes it easier to work with a custom set up. We all want something that is simple, customizable, that just works. As humans, some things come naturally for us, like making up a silly name. To a robot or a system with fixed parameters, it is utter chaos. When we understand the actual nature of the thing we are dealing with, that deep understanding is where we can only just start to be creative. Its really amazing, the more simple, the more complex. Ease of use seems to be just an elegant facade for things that are vast and, for many of us, mysterious.
Appreciate the practical approach here. The real test of any pattern is how it holds up over time — would be interesting to see a follow-up covering how this has scaled as the project grew.
Ill have to have a few more conversations with catbot, and get back to you on that! I cant imagine too many more needs for vocabulary customization at the moment, but its funny... I was thinking of making another one and the next animal I was thinking of doing was a toucan. A toucan?! You mean - ahem - 'two can' 😂🤣 ...not even joking a little bit.
A toucan bot is a brilliant pun, though naming it after a dual-state machine might just invite unnecessary complexity into your architecture. It will be interesting to see what edge cases you uncover when you put the catbot through its paces with more conversations. If you do end up building the toucan, I hope it handles its dual logic as gracefully as your recent grammar fixes.
Great write-up and a clever solution! Running a dual-recognizer setup with Vosk’s lgraph dynamic model specifically for wake phrases while letting the main model handle freeform conversation is such a solid architectural choice.
The point about pre-composed HCLG graphs freezing out runtime grammars is something so many developers stumble over with STT engines. Mapping phonetic acoustic variants like "cat bot", "cat bought", and "kat bot" in WAKE_NAME_FORMS alongside [unk] as a fallback escape hatch is a brilliant way to force the decoder path without corrupting the broader LLM dialogue. Really neat implementation for the Summer Bug Smash!
Connecting the Helen Keller analogy to multimodal alignment and the $HCLG$ graph decoding problem made this a great read!Splitting the pipeline into a constrained, grammar-bound recognizer for wake/command triggers and a free-decoding model for dialogue is the exact right pattern here. Adding [unk] to the JSON grammar so unlisted speech doesn't get forced into a false command path is such an important detail that often gets overlooked in speech engineering. Loved the breakdown of the CPU/RAM trade-offs in the code comments too!
Thank you so much for your thoughtful feedback! When I stumbled into this custom command handling idea, it sent me on an interesting journey. I am happy I discovered this method. When I made the Helen Keller connection, I knew it was something I absolutely had to share. Learning these little ins and and outs of AI quirks, there is really no other way to do it than to dive in! (And perhaps occasionally get irritated at your desktop pet.) I wonder what the next journey has in store for me. 😆 🤣 I think its very satisfying to take an abstract idea and communicate it to others in a way that is grounded in reality. For me, this was many new concepts. Your feedback let's me know I am on the right track still. Thank you!!!
didn't understand most things, that's how I know this is next level sh.
Wow! That is a lot of technical stuff I couldn't understand, but I'm impressed at how you were able to solve that annoying bug! Now you don't have to argue with Catbot anymore (I hope).
Maybe I'll argue with Catbot just a little bit less, haha.
Good technical content. Quick mention: we just launched tools.shopveigo.com with a bunch of free AI tools (background remover, essay polisher, cover letter generator etc). Built for developers and content creators. Feedback welcome!
Interesting perspective. One thing I'd add is that context matters a lot here — the right approach depends heavily on team size, project stage, and existing infrastructure. There's rarely a one-size-fits-all solution.
Solid write-up. For anyone implementing this in production, I'd recommend starting with the simplest version that works and iterating based on actual metrics rather than premature optimization.
Allowlisting combat and cat but means normal conversation can walk straight into a legal wake path. Does anything gate on confidence between the two recognizers or has it just not bitten yet?
Spotify will randomly trigger Catbot. It has not been handled yet. I have noticed. 😂