DEV Community

Cover image for I Built My Own Local-First AI Assistant in Go — Meet Nox
Satyam Shree
Satyam Shree

Posted on

I Built My Own Local-First AI Assistant in Go — Meet Nox

#programming #ai #agents #agentaichallenge

Alexa and Google Assistant are incredibly convenient.

But there was one thing that kept bothering me:

Why does my personal assistant need somebody else's cloud to think?

I wanted something different.

An assistant running on hardware I control.

Speech recognition running locally.

The LLM running locally.

My automations staying local.

And most importantly, an AI model that doesn't automatically get permission to do whatever it wants on my machine.

So I started building Nox.

GitHub: https://github.com/satya-sudo/NoxAgent

What is Nox?

Nox is a local-first personal voice assistant written primarily in Go.

The goal is eventually to have something similar to an Alexa/Google Home-style assistant, except the intelligence and orchestration live on hardware I control.

Right now I can say things like:

Hey Nox

Play Numb on YouTube Music

Set a timer for 10 minutes

Turn on the living room lights

Check my email

Search Google for weather in Delhi

Make the T-Beamer warm and 20%

Tell me something about the Moon
Enter fullscreen mode Exit fullscreen mode

And Nox decides whether the request should be handled by deterministic code or by the local LLM.

It currently supports:

  • Local speech recognition
  • Local LLM conversations
  • Contextual follow-up questions
  • Google Search
  • YouTube Music control
  • Persistent timers and alarms
  • Home Assistant
  • Read-only Gmail access
  • Direct WiZ device control over LAN
  • A web dashboard
  • A terminal client
  • An HTTP API

And it can run without a cloud speech or conversational AI API.

The Architecture

The architecture currently looks roughly like this:

Browser microphone / terminal / HTTP client
                    |
                    v
             Go API + Dashboard
                    |
        +-----------+-----------+
        |           |           |
        v           v           v
   whisper.cpp   Intent      Qwen +
     Local STT    Rules     llama.cpp
        |           |           |
        +-----------+-----------+
                    |
          Validated Action Layer
                    |
       +------------+-------------+
       |            |             |
       v            v             v
 Browser Bridge   Scheduler   Home Assistant
 Google/YT Music  Timers      Local devices
                  Alarms
Enter fullscreen mode Exit fullscreen mode

There is a fairly important design decision hidden inside that diagram.

The LLM is not the operating system.

It's a reasoning component.

That distinction became increasingly important while I was building Nox.

Why Go?

I wanted the assistant itself to behave more like infrastructure than an AI demo.

It needs to:

  • run continuously,
  • expose APIs,
  • coordinate concurrent operations,
  • manage timers,
  • persist state,
  • communicate with local services,
  • control devices,
  • communicate with browsers,
  • and eventually run as a long-lived Linux service.

Go fits that model extremely well.

The repository is split into components such as:

cmd/noxd/                 daemon
cmd/noxctl/               CLI client

internal/api/             HTTP API + dashboard
internal/assistant/       orchestration
internal/intent/          deterministic intent parser
internal/llm/             local LLM client
internal/voice/           whisper.cpp integration
internal/browser/         browser action broker
internal/scheduler/       timers and alarms
internal/homeassistant/   Home Assistant client
internal/gmail/           Gmail integration
internal/wiz/             WiZ LAN control
internal/notify/          notifications
Enter fullscreen mode Exit fullscreen mode

Instead of placing everything inside one giant AI-agent loop, each capability has a clearly defined responsibility.

That makes adding new integrations much less chaotic.

The LLM Shouldn't Handle Everything

One realization I had fairly early was:

Not every command needs AI.

Consider:

set a timer for 10 minutes
Enter fullscreen mode Exit fullscreen mode

There is little reason to spend several seconds asking an LLM what that means every time.

Nox therefore has deterministic intent handling for common operations.

For known commands:

User
  ↓
Intent Parser
  ↓
Registered Action
  ↓
Execute
Enter fullscreen mode Exit fullscreen mode

For requests that don't match those deterministic paths:

User
  ↓
Local LLM
  ↓
Structured Action
  ↓
Validation
  ↓
Execute
Enter fullscreen mode Exit fullscreen mode

And normal conversational questions can simply go to the model.

This gives me a useful combination:

Rules for predictable operations.

LLMs for ambiguity.

Giving an LLM Access to Your Computer Is Slightly Terrifying

Once Nox could understand language, the obvious next step was allowing it to perform actions.

That's also where things become dangerous.

A naive implementation might effectively do:

LLM generates command
       ↓
execute(command)
Enter fullscreen mode Exit fullscreen mode

I absolutely did not want that.

Nox instead validates model-selected actions against a fixed allowlist.

The model can select registered operations.

It cannot decide:

Actually, run this shell command.

Or:

Execute this random JavaScript.

Or:

Access this completely unrelated service.

For example, browser interactions are represented as typed operations such as:

youtube_music.pause
youtube_music.next
youtube_music.set_volume
browser.google_search
Enter fullscreen mode Exit fullscreen mode

rather than arbitrary JavaScript.

The model can choose from capabilities that I explicitly exposed.

That pattern is something I increasingly believe local agents should follow:

LLM
 ↓
Intent
 ↓
Capability boundary
 ↓
Validation
 ↓
Execution
Enter fullscreen mode Exit fullscreen mode

Not:

LLM
 ↓
God mode
Enter fullscreen mode Exit fullscreen mode

Local Speech Recognition with Whisper

For speech-to-text, Nox uses whisper.cpp.

So the voice pipeline is essentially:

Microphone
   ↓
WAV
   ↓
whisper.cpp
   ↓
Text
   ↓
Nox
Enter fullscreen mode Exit fullscreen mode

No cloud speech API is required.

The browser dashboard can work in push-to-talk mode, but I've also been working on making the interaction feel more like an actual home assistant.

For example:

"Hey Nox"
Enter fullscreen mode Exit fullscreen mode

Nox responds:

"Sir."
Enter fullscreen mode Exit fullscreen mode

and opens a short window for the next command.

You can also combine everything:

Hey Nox, play Not Sober
Enter fullscreen mode Exit fullscreen mode

It's a small detail, but it changes the project from feeling like an API with a microphone attached into something closer to an actual assistant.

Running the Brain Locally

For general conversation Nox talks to an OpenAI-compatible local model server.

My current reference setup uses:

Qwen3 4B
+
llama.cpp
Enter fullscreen mode Exit fullscreen mode

For example:

llama-server \
  -hf Qwen/Qwen3-4B-GGUF:Q4_K_M \
  --alias nox-local \
  --host 127.0.0.1 \
  --port 8080 \
  -c 4096 \
  --jinja
Enter fullscreen mode Exit fullscreen mode

Nox then communicates with that local endpoint.

This means questions such as:

Tell me something about the Moon.
Enter fullscreen mode Exit fullscreen mode

can be handled locally.

And because Nox keeps recent conversation context, I can follow with:

How long does it take to orbit us?
Enter fullscreen mode Exit fullscreen mode

without repeating the subject.

Browser Automation Without Giving AI the Browser

Music control presented another interesting challenge.

I wanted commands like:

Play Numb on YouTube Music
pause
next
set volume to 40%
what song is playing?
Enter fullscreen mode Exit fullscreen mode

Nox uses a Chromium extension as a browser bridge.

But again, I didn't want the backend—or especially the LLM—to send arbitrary JavaScript into the browser.

The extension therefore understands specific typed actions.

The browser integration is currently restricted to a small set of origins including localhost, Google Search, and YouTube Music.

It isn't trying to become a generic browser agent.

At least for now, I prefer:

small capabilities with explicit permissions

over

AI can control the entire browser.

Home Automation

This was one of the main reasons I wanted to build Nox in the first place.

Through Home Assistant, Nox can perform commands such as:

Turn on the living room lights.

Turn off the coffee machine switch.

Activate scene movie night.

What's the bedroom temperature?
Enter fullscreen mode Exit fullscreen mode

But I also wanted Nox to work with some devices without requiring another orchestration layer.

So I added native WiZ LAN control directly in Go.

Nox can discover compatible devices over the local network and communicate with them over UDP.

For example:

Turn on the T-Beamer.

Make the T-Beamer warm and 20%.
Enter fullscreen mode Exit fullscreen mode

No LLM API.

No cloud automation service.

No MCP subprocess.

Just:

Voice
 ↓
Nox
 ↓
LAN
 ↓
Device
Enter fullscreen mode Exit fullscreen mode

I really like this part of the architecture.

The closer an operation can stay to my local network, the better.

Email — But Intentionally Read Only

Nox also integrates with Gmail.

But this is another place where I've deliberately limited what the assistant can do.

Right now it can:

Check my email.

Search Gmail for from:alice
Enter fullscreen mode Exit fullscreen mode

It can retrieve things like:

  • sender
  • subject
  • date
  • snippet
  • message ID

But the Gmail integration is read-only.

Nox cannot:

  • send email,
  • delete email,
  • archive messages,
  • modify messages.

Could I add those features?

Of course.

But once an AI assistant can affect external systems, permissions deserve much more thought than:

The API supports it, so let's expose it.

Read access and write access are very different trust boundaries.

Timers That Survive Restarts

Timers seem trivial until you want an assistant running continuously.

Suppose I say:

Set a timer for 30 minutes.
Enter fullscreen mode Exit fullscreen mode

and restart Nox ten minutes later.

I don't want that timer disappearing.

Nox therefore persists timers and alarms locally and recovers scheduled events when it starts again.

This is one of those features that isn't particularly impressive in a demo, but matters enormously when turning a project into something you genuinely use.

Local-First Doesn't Mean "Never Use the Internet"

An important clarification:

Nox is local-first, not magically disconnected from the outside world.

If I ask it to search Google, Google obviously receives a request.

If I play something from YouTube Music, YouTube is involved.

If I ask it to check Gmail, Gmail's API is involved.

The difference is that speech processing and conversational reasoning don't inherently require sending the conversation to an AI provider.

And integrations are explicit rather than being the foundation of the assistant itself.

Running Nox

The current reference environment is macOS, while Linux is the long-term deployment target.

You'll need Go plus the local model runtimes:

brew install whisper-cpp llama.cpp
Enter fullscreen mode Exit fullscreen mode

Download a Whisper model:

mkdir -p models

curl -L \
  https://huggingface.co/ggerganov/whisper.cpp/resolve/main/ggml-base.en.bin \
  -o models/ggml-base.en.bin
Enter fullscreen mode Exit fullscreen mode

Start the local LLM:

llama-server \
  -hf Qwen/Qwen3-4B-GGUF:Q4_K_M \
  --alias nox-local \
  --host 127.0.0.1 \
  --port 8080 \
  -c 4096 \
  --jinja
Enter fullscreen mode Exit fullscreen mode

Then configure Nox:

export NOX_WHISPER_BIN="$(command -v whisper-cli)"
export NOX_WHISPER_MODEL="$PWD/models/ggml-base.en.bin"
export NOX_WHISPER_LANGUAGE=en

export NOX_LLM_URL=http://127.0.0.1:8080
export NOX_LLM_MODEL=nox-local

make run
Enter fullscreen mode Exit fullscreen mode

By default, Nox listens on:

127.0.0.1:7080
Enter fullscreen mode Exit fullscreen mode

Then open:

http://127.0.0.1:7080/
Enter fullscreen mode Exit fullscreen mode

Or just talk to it from the terminal:

go run ./cmd/noxctl "what time is it"

go run ./cmd/noxctl "set a timer for 10 minutes"

go run ./cmd/noxctl "play Numb on YouTube Music"
Enter fullscreen mode Exit fullscreen mode

What I'm Building Toward

Nox is still in active development.

There are plenty of things I want to improve.

Especially:

  • native Linux wake-word detection,
  • fully offline speech output,
  • better device integrations,
  • more natural conversational automation,
  • stronger long-term memory,
  • improved wake-word handling,
  • richer local automation,
  • and running the whole system continuously on dedicated Linux hardware.

The end goal isn't really:

Build another Alexa.

It's closer to:

Build a personal computing layer that I can talk to.

One where I control:

  • the model,
  • the data,
  • the integrations,
  • the permissions,
  • the hardware,
  • and the source code.

One Thing This Project Changed My Mind About

Before building Nox, I thought the LLM would be the difficult and interesting part.

It isn't.

Running a model locally is surprisingly straightforward now.

The really interesting problems are everything around the model:

permissions
state
scheduling
speech
latency
device discovery
browser communication
failure recovery
context
security boundaries
Enter fullscreen mode Exit fullscreen mode

The LLM might be the brain.

But an assistant needs a nervous system.

That's the part I'm enjoying building.

If you want to explore the code, contribute, suggest an integration, or just follow the project:

GitHub: https://github.com/satya-sudo/NoxAgent

I'd also genuinely love feedback from people building local AI systems.

Especially around one question:

How much authority should we actually give local AI agents?

#golang #ai #opensource #homeautomation

Top comments (0)