🐈Catbot: The Custom AI Harness That Lives on Desktop ᓚᘏᗢ

#ai #devchallenge #weekendchallenge #webdev

This is a submission for Weekend Challenge: Passion Edition

What I Built

This weekend, I went on a journey to build a voice activated assistant on my desktop, named Catbot. In the process, I switched from notion over to obsidian. Obsidian acts as a knowledge base for the agents to communicate. Catbot is a custom harness I built that I can use with any of my agents. Catbot is voice activated and sleeps in a bed when I tell him too. I also created a small skin modifier to change it's appearance.

Demo

Catbot switches voices in real time on my desktop!

Code

Here is a snippet from my catbot_voice.py. This is a private project at the moment, but I am comfortable sharing the initial setup with everyone!

import os
import re
import json
import time
import queue
import random
import logging
import threading
from pathlib import Path

import httpx
import numpy as np
import sounddevice as sd
import vosk
from kokoro_onnx import Kokoro
from dotenv import load_dotenv

load_dotenv(Path(__file__).parent / ".env")

logger = logging.getLogger("catbot.voice")

ROOT = Path(__file__).parent
KOKORO_ONNX = ROOT / "models" / "kokoro-v1.0.onnx"
KOKORO_VOICES = ROOT / "models" / "voices-v1.0.bin"
VOSK_BIG = ROOT / "models" / "vosk-model-en-us-0.22-lgraph"
VOSK_SMALL = ROOT / "models" / "vosk-model-small-en-us-0.15"

SAMPLE_RATE = 16000
BASE = f"https://localhost:{os.getenv('CATBOT_PORT', '8800')}"

# Tolerant of how recognizers mangle "catbot": fuzzing Vosk against Kokoro
# speech produced "cat bought", "that bought", "cat ba bought", "can't bought",
# "cat bottom gauge"... A false wake just means he listens, so err generous.
BOT = (r"(?:cat|kat|cad|catt|that|can'?t|cap)\s*-?\s*(?:\w{1,3}\s+)?"
       r"(?:bottom|bought|boat|both|body|bert|bird|bot|but)")
# every way of telling him to be quiet actually works now — "shut down",
# "stop", "quiet" etc. used to fall through to the LLM, which would *claim*
# to shut down while nothing happened
STOP_WORDS = (r"(?:go\s+to\s+)?(?:sleep|slee|asleep)|shut\s*(?:down|up)|"
              r"(?:be\s+)?quiet|stop(?:\s+(?:listening|talking))?|"
              r"power\s+(?:down|off)|turn\s+off|pause|hush|silence")
RE_SLEEP = re.compile(BOT + r"[,!.\s]*(?:" + STOP_WORDS + r")\b", re.I)
RE_ENGAGE = re.compile(BOT + r"[,!.\s]*(?:(?:en\s*)?(?:engage|gage|gauge|engaged|gaged)d?"
                             r"|wake\s*up|wake)[,!.\s]*(.*)", re.I)
# While ASLEEP, bias hard toward waking: a missed wake makes him unusable,
# a false wake just means he listens. Any engage-ish or wake-ish word wakes.
RE_WAKE_LOOSE = re.compile(
    r"\b(?:en|in|and|un)?[\s-]*(?:gage|gauge|engage)[ds]?\b|\bwake+\s*(?:up|it)?\b",
    re.I)

# in-conversation commands
PROVIDER_WORDS = {
    "claude": "anthropic", "anthropic": "anthropic",
    "gemini": "gemini", "google": "gemini",
    "gpt": "openai", "chatgpt": "openai", "chat gpt": "openai",
    "openai": "openai", "open ai": "openai",
    "ollama": "ollama", "local": "ollama", "qwen": "ollama", "quen": "ollama",
}

How I Built It

A FastAPI server server.py is the single brain-stem: every surface talks to Catbot over HTTPS on port 8800.
Catbot routes chats to chosen provider in providers.py, injects the shared memory from the obsidian vault.
runs background agent missions, and turns catbot-publish blocks in replies into real files like vualt notes or web pages.
Catbot is a transparent always-on-top window with its own offline voice engine catbot_voice.py: Vosk hears speech, intents route commands, Kokoro answers aloud.

I was first using kokoro for voices and then I discovered the amazing possibilities for voices at eleven labs. I recorded enough of my voice to make a pretty close clone, and it's definitely interesting to hear yourself talk back to you.

There is a web view but he mostly just lives floating on the desktop - where I want Catbot to live.

I went through some modifications during this process. Catbot started out Grey and then I added a skin modifier:

Grey Catbot — Original rendering of Catbot

Web interface — Catbot optional web interface

I am obsessed with my new digital pet cat, and that's why I had to share him for this passion project. Thank you for reading!

Prize Categories

I am submitting for Best Use of ElevenLabs. Using a custom built voice, I was able to integrate it through the API easily to Catbot to add to my voice bank!

DEV Community