DEV Community: GarySto

How to Actually Build Your Own Local AI Assistant

GarySto — Thu, 29 Oct 2026 21:35:47 +0000

Originally published on garystow.co.uk.

A few weeks ago I had nothing, no local AI setup, no idea where to even start, just a genuine question: could I build a proper voice assistant using only free tools. Here's the actual path, assuming you're starting from exactly nowhere.

Start with your hardware, before you install anything

This is the step everyone skips and it's the one that decides everything else. Local AI models don't run in the cloud, they run on your actual graphics card, and two numbers matter more than anything else.

VRAM, the dedicated memory on your graphics card. This is what actually holds the model while it's thinking. More VRAM means bigger, smarter models running at full speed. Less VRAM means the model either doesn't fit at all, or spills over into your regular system memory, which is dramatically slower.
System RAM, your computer's general memory. Less critical than VRAM for speed, but you want headroom here too, 32GB is comfortable.

Open Task Manager, go to Performance, click your GPU, and you'll see your dedicated VRAM figure right there. Then check whether you're on Nvidia or AMD. This matters because almost every tutorial online assumes Nvidia, using something called CUDA. If you're on AMD like me, that path isn't available, but it's not a dead end, it just means a different backend called Vulkan, which works properly and is built into the software you're about to install.

Installing LM Studio

This is the actual engine everything else sits on top of.

Go to lmstudio.ai and download it. Make sure you get the plain version, there's a separate account-gated product on the same site called Bionic, that's not what you want.
Install it and open it up.
Load any small model to start (more on picking one below), ask it a basic question, and check Task Manager, Performance, your GPU, while it's replying. If you see GPU usage spike, it's actually using your graphics card properly rather than falling back to your CPU.

Picking your first model

This is where people freeze, so here's the actual rule of thumb I use. Stick to models published by the model's original maker, so names like Qwen or openai, or by a group called lmstudio-community, who package models properly for this exact use case. Before downloading, check for a green badge saying full GPU offload possible, that confirms it'll actually fit on your card. And on quantisation, the technical-sounding number and letters after the model name, Q4_K_M is the standard, sensible default, a solid balance of speed, quality, and file size, don't overthink it beyond that at the start.

On my own machine right now I've got a small model for quick general chat, a 14B all-rounder, a dedicated coding model, and a larger 20B model that's become my actual daily driver, all running through this same setup. Start with something smaller than that, get it working end to end, then upgrade once you trust the process.

Turning it on so other things can actually talk to it

Right now your model only talks to you inside LM Studio's own chat window. To let something like a voice assistant actually reach it, you need the Local Server.

In LM Studio, find the Developer tab.
Switch Local Server to running. You'll see an address, something like a local web address ending in colon 1234, that's the door everything else knocks on.
Leave Just-in-Time model loading switched on, this loads a model only when something actually asks for it, rather than eating your VRAM constantly.
Set an auto-unload time if you plan on using more than one model, mine's set to an hour of inactivity.

Worth knowing honestly, if you switch between two different models regularly like I do, one for chat and one for code, only one properly fits in memory at a time on a 16GB card, so there's a genuine ten to thirty second wait the first time you switch. Not a bug, just physics.

Giving it a personality, the fun bit

Inside LM Studio there's a System Prompt field, this is a set of instructions the model reads before every single reply, and it's how a generic model becomes something that actually feels like your assistant rather than a chatbot. Mine's built in two modes, one for casual back and forth with real personality, and a completely separate one that drops that personality entirely and writes in my own voice instead whenever it's producing something meant to go out under my name. Worth deciding early which of those you actually need.

What to actually type once it's running

Start simple. Ask it to introduce itself, see if the personality you set actually comes through. Ask it something you already know the answer to, to confirm it's genuinely working. Then, and this is the one that matters most for trust, ask it something it has no way of actually knowing, and watch what it does. If it confidently makes something up rather than telling you it doesn't know, that's the very first thing worth fixing before you build anything else on top of it.

On LinkedIn I showed a live demo of this in action, worth a look if you want to see how it actually responds in real time.

Teaching it to actually listen, without a click

Everything so far still needs you to type. Getting it to genuinely listen, the way Wade does, needs something called a wake word, a tool called openWakeWord handles the actual detection. Here's a real snag worth knowing before you get your hopes up, there's no pre-trained wake phrase for whatever name you've picked for your own assistant. You get a short, fixed list, Alexa, Hey Mycroft, Hey Jarvis, that's genuinely it unless you train your own from scratch. Mine ended up answering to Hey Jarvis, still makes me laugh given the name I actually gave him.

Worth knowing too, once it's wired up, if wake detection feels unreliable, check your actual microphone before you touch a line of code. Mine looked completely normal in every settings menu and was quietly too quiet to be picked up properly, swapping to a proper USB headset mic fixed detection completely, no amount of code tuning would have solved it.

Giving it ears and a voice

Faster-Whisper handles turning your actual speech into text, running fully on your own machine, no cloud involved. Piper handles the reverse, turning replies back into a spoken voice, also entirely local. Worth setting expectations honestly here, free local voices have a real ceiling on how expressive and natural they sound compared to paid cloud options, it's a genuine trade-off for staying completely free, not a flaw you've caused.

Tying it all together

This is the part where it stops being configuration and becomes real software engineering. A Python backend, using something called Flask, has to actually manage the whole loop, listening for the wake word, capturing your voice, sending it to Whisper, passing the result to your model, sending the reply to Piper, and displaying all of it somewhere you can see, in my case a HUD I built from scratch using HTML, CSS and JavaScript.

And this is honestly where I have to stop holding your hand.

Everyone's setup from here genuinely differs, your hardware, your microphone, your exact file paths, your chosen wake phrase, enough that writing out generic code steps would do you more harm than good, you'd end up debugging my assumptions instead of building your own thing. This bit takes real trial and error, proper coding, and patience with things breaking in specific, findable ways rather than a big leap forward. I'm genuinely still learning as I go here myself, so I can't pretend I've got a clean, universal path from this point through to a finished voice loop.

If you've built something like this yourself, or you're further along than me, I'd genuinely love to hear how you approached it. If you're just getting started, you're already closer than you think. Good luck.

If getting a build like this actually found by anyone matters to you once it's live, I wrote separately about what makes a site readable to search engines and AI crawlers, worth a look once you're at the publishing stage.

Built something similar, or stuck on a step? Email me at garystow.hq@gmail.com — I read everything myself.

I write about practical AI for marketers and building with free tools at garystow.co.uk. This piece first appeared there.

How to Build a Live Village Dashboard for Your AI Agents

GarySto — Fri, 09 Oct 2026 21:50:12 +0000

Originally published on garystow.co.uk. The free starter code is on GitHub.

Written by Gary Stow with help from AI tools. Every fact is checked and every article is approved by Gary before it goes live.

I run a lot of automations. One team writes and checks articles for this site, another runs three demo trading strategies, and a supervisor sits over both deciding what needs me. For months the only way to see any of it was a dashboard full of tables. It worked for me, but when my wife or my children asked what I was actually building every evening, a table of job names meant nothing to them.

So I turned the whole thing into a village. I built it to help explain my work to my family, and the visual approach made it far easier for them to understand than anything I could say. The style is a cross between RimWorld and South Park: I love playing RimWorld, and I have always been a fan of South Park.

Every page on this site is a house. Every trading strategy is a field that grows or wilts with its results. Every request walks to the Town Hall, and builders are always working on the next house. You can see the live version in the Village.

Open the live Village ↗ How the village works

This guide shows you how to build your own, one step at a time. You do not need a game engine, and everything here is free. If you can run a Python script, you can do this.

What you need before you start

Python 3 on your computer
A text editor (I use VS Code)
A web browser
Something that already runs on its own, such as a scheduled script, an AI agent or a bot, that you want to watch
Optional: a free GitHub Pages website, if you want a public version people can visit

How the village works

There are only three moving parts, and keeping them separate is what makes this manageable.

Your automations write a line to a log file every time they do something.
A small Python script reads those logs and writes one JSON file describing the village.
One HTML page reads that JSON file and draws the village on a canvas.

The drawing never talks to your automations directly. That means a bug in the village can never break the real work, and you can rebuild the picture as often as you like.

Step 1: Make every automation leave a trail

The village can only show what your automations record. Give every script one shared function that adds a line to an activity log. One line of JSON per event, called JSON Lines, is the simplest format there is, because you only ever add to the end of the file.


    import json
    from datetime import datetime

    def log_activity(who, text, outcome="ok"):
        row = {"ts": datetime.now().strftime("%Y-%m-%d %H:%M"),
               "who": who, "text": text, "outcome": outcome}
        with open("activity_log.jsonl", "a", encoding="utf-8") as f:
            f.write(json.dumps(row) + "\n")

    log_activity("writer", "Drafted: how to build a village")

Call it at the important moments: a job starting, finishing, failing, or waiting for a human.

Step 2: Decide what every building means

Before drawing anything, write down what each building stands for. This is the part that makes the village make sense to someone who has never seen your system. Mine looks like this:

The Town Hall is the supervisor. Every request goes through it.
Each team has its own district: website on one side, trading on the other. Work from one never walks into the other.
Houses are web pages. Bigger houses get more visits.
Fields are trading strategies. They all start the same size, then grow or shrink with their profit or loss.
People standing outside the Town Hall are requests waiting for my decision.

Keep a simple lookup from the names in your log to the building where that work happens.

Step 3: Turn the logs into one data file

Save this as village_data.py. It reads the last 48 hours of the log and writes the file the page will draw from. Run it on a schedule, or every time the page asks for it (Step 9 does that).


    import json
    from datetime import datetime, timedelta
    from pathlib import Path

    LOG = Path("activity_log.jsonl")
    OUT = Path("village/village_data.json")
    BUILDING_FOR = {"writer": "workshop", "checker": "hall",
                    "publisher": "site", "trader": "field"}

    def recent_events(hours=48):
        cutoff = datetime.now() - timedelta(hours=hours)
        events = []
        for line in LOG.read_text(encoding="utf-8").splitlines():
            e = json.loads(line)
            if datetime.strptime(e["ts"], "%Y-%m-%d %H:%M") >= cutoff:
                events.append({"time": e["ts"][11:], "from": "hall",   # every job starts at the Town Hall
                               "at": BUILDING_FOR.get(e["who"], "hall"), "text": e["text"]})
        return events[-30:]

    def build(pages, results):
        data = {"generated": datetime.now().isoformat(timespec="minutes"),
                "pages": pages, "events": recent_events(), "fields": results}
        OUT.parent.mkdir(exist_ok=True)
        OUT.write_text(json.dumps(data), encoding="utf-8")

    def main():
        build(pages=[{"title": "Home page", "url": "/", "views": 84}],
              results={"strategy_a": 120, "strategy_b": -40})

    if __name__ == "__main__":
        main()

The page views can come from any analytics export you already have. Start with made-up numbers and swap in real ones once the picture works.

Step 4: Draw the ground with isometric tiles

The RimWorld style look comes from isometric tiles: diamonds that are twice as wide as they are tall. One small function turns a grid square into a position on the screen, and everything else is built on top of it.


    const TW = 64, TH = 32;                 // one tile: 64 wide, 32 tall

    function toScreen(x, y) {               // grid square -> pixel position
      return { sx: (x - y) * TW / 2, sy: (x + y) * TH / 2 };
    }

    function tile(ctx, x, y, colour) {
      const { sx, sy } = toScreen(x, y);
      ctx.beginPath();
      ctx.moveTo(sx, sy);
      ctx.lineTo(sx + TW / 2, sy + TH / 2);
      ctx.lineTo(sx, sy + TH);
      ctx.lineTo(sx - TW / 2, sy + TH / 2);
      ctx.closePath();
      ctx.fillStyle = colour;
      ctx.fill();
    }

Loop over a grid and call tile for every square. Use one colour for grass, another for roads, and blue for a river between districts.

Step 5: Draw buildings in a paper cut-out style

A building is three shapes: a darker left wall, a lighter right wall and a roof. Thick black outlines give it the cut-out look. Draw from the back of the map to the front, sorted by x + y, so nearer buildings cover the ones behind them.


    function poly(ctx, pts, fill) {
      ctx.beginPath();
      pts.forEach(([px, py], i) => i ? ctx.lineTo(px, py) : ctx.moveTo(px, py));
      ctx.closePath();
      ctx.fillStyle = fill;
      ctx.fill();
      ctx.lineWidth = 3;
      ctx.strokeStyle = "#000";
      ctx.stroke();
    }

    function shade(hex, pct) {              // lighten or darken a colour
      const n = parseInt(hex.slice(1), 16);
      const f = c => Math.max(0, Math.min(255, c + Math.round(2.55 * pct)));
      return "#" + [f(n >> 16), f((n >> 8) & 255), f(n & 255)]
        .map(v => v.toString(16).padStart(2, "0")).join("");
    }

    function box(ctx, x, y, h, colour) {    // h = height in pixels
      const { sx, sy } = toScreen(x, y);
      const top = sy - h;
      poly(ctx, [[sx - TW/2, top + TH/2], [sx, top + TH], [sx, sy + TH], [sx - TW/2, sy + TH/2]], shade(colour, -25));
      poly(ctx, [[sx, top + TH], [sx + TW/2, top + TH/2], [sx + TW/2, sy + TH/2], [sx, sy + TH]], shade(colour, -10));
      poly(ctx, [[sx, top], [sx + TW/2, top + TH/2], [sx, top + TH], [sx - TW/2, top + TH/2]], colour);
    }

Add windows, doors and signs once the boxes look right. Resist the urge to make it pretty too early. Get the layout working with plain boxes first.

Step 6: Make the people walk the real work

This is the part that brings it to life. Every event in the data file sends a little person from one building to another. When the supervisor approves something, someone walks from the Town Hall to the executor. When an article goes live, a builder walks to the housing district.


    const walkers = [];

    function playEvent(ev) {                // ev.from and ev.at are building names
      walkers.push({ from: BUILDINGS[ev.from], to: BUILDINGS[ev.at], t: 0 });
    }

    function moveWalkers(dt) {              // dt = milliseconds since last frame
      for (const w of walkers) w.t = Math.min(1, w.t + dt / 4000);  // 4 seconds a trip
      for (let i = walkers.length - 1; i >= 0; i--) {
        if (walkers[i].t >= 1) walkers.splice(i, 1);
      }
    }

    function walkerPos(w) {
      return { x: w.from.x + (w.to.x - w.from.x) * w.t,
               y: w.from.y + (w.to.y - w.from.y) * w.t };
    }

Play one event every few seconds and loop back to the start when you run out. Draw a coloured path for each team's flow, so you can see at a glance that website work and trading work never cross.

Step 7: Let the data change the world

Now connect the picture to real numbers. These two rules do most of the work in mine.


    function fieldScale(net) {              // net = profit or loss in pounds
      return Math.max(0.6, Math.min(1.45, 1 + net / 400));
    }

    function houseFloors(views) {
      return views > 200 ? 3 : views > 50 ? 2 : 1;
    }

Clamping matters. Without the limits, one great or terrible week would make a field fill the screen or vanish. Pick limits that keep the village readable.

Step 8: Run the drawing loop

Save the page as village/index.html, next to the data file, with a <canvas id="village" width="1200" height="750"> in it. Fetch the data once, then redraw about sixty times a second.


    const canvas = document.getElementById("village");
    const ctx = canvas.getContext("2d");
    let DATA = null, last = performance.now();

    function frame(now) {
      const dt = now - last;
      last = now;
      ctx.setTransform(1, 0, 0, 1, 0, 0);
      ctx.clearRect(0, 0, canvas.width, canvas.height);
      ctx.setTransform(1, 0, 0, 1, canvas.width / 2, 80);  // grid square 0,0 sits top centre
      drawGround();
      drawBuildings();
      moveWalkers(dt);
      drawWalkers();
      requestAnimationFrame(frame);
    }

    fetch("village_data.json")
      .then(r => r.json())
      .then(d => { DATA = d; requestAnimationFrame(frame); });

drawGround, drawBuildings and drawWalkers are your own functions built from the pieces above.

Step 9: Serve a private version from your own computer

My private village runs from the same small Flask app as my local dashboard, so it can show everything, including the actual requests waiting for me. Clicking a person outside the Town Hall opens the decision.


    from flask import Flask, send_from_directory
    import village_data          # the Step 3 script

    app = Flask(__name__)

    @app.route("/village/")
    def village():
        return send_from_directory("village", "index.html")

    @app.route("/village/village_data.json")
    def data():
        village_data.main()      # rebuild from the latest logs every time the page asks
        return send_from_directory("village", "village_data.json")

    if __name__ == "__main__":
        app.run(port=5007)

Install Flask once with pip install flask, run the script, then open localhost:5007/village/ in your browser and you are watching your own system.

Step 10: Publish a safe public version

The public version uses the same page with a different data file. Before anything leaves my computer, a function strips out anything private. Requests become a generic title, and money is rounded.


    def make_public(data):
        pub = dict(data)
        pub["requests"] = [{"id": r["id"], "from": r["from"], "title": "A decision for Gary"}
                           for r in data.get("requests", [])]
        pub["fields"] = {k: round(v / 50) * 50 for k, v in data["fields"].items()}
        return pub

A scheduled task then copies the page and the public data file into my website on a new branch, opens a pull request, and only merges it after an automatic layout check in a headless browser and an SEO checklist both pass. If either check fails, nothing goes live. I wrote about why I make automation check its own work in the independent audit.

Step 11: Make it readable for everyone

Small pixel text looks great and is impossible to read. Use a large, clear font for labels (I use Atkinson Hyperlegible, which was designed for low vision), strong contrast, and hide the smaller labels when someone zooms out so they never pile on top of each other. Test it on a phone before you share it.

What went wrong the first time

Three things caught me out, so you can skip them.

Labels overlapped as soon as the village grew. Showing department names only when zoomed in fixed it.
My first version let trading work walk into the housing district. Separate districts and separate colours for each team's flow made the picture honest.
My scheduled jobs run without a window, but every git command they started popped up its own black console box. On Windows, start those commands with the CREATE_NO_WINDOW flag from Python's subprocess module, or you will get a screen full of flashing windows.

Do I need to know JavaScript to build this?

No. I am self-taught and built mine with help from AI tools. You need to be able to copy code, run it, read the error when it breaks and ask for help with that error. The canvas drawing above is the only JavaScript you need.

Is it safe to show my automations in public?

Yes, if the public version gets its own data file. Never point the public page at your private data. Build a function like make_public that removes names, request details and exact money figures, and check its output before every publish.

Can I build this without paying for anything?

Yes. Python, Flask, a browser canvas, GitHub Pages and Windows Task Scheduler are all free. If you want an AI assistant in the village too, you can run one at home with no token bill, which I explain in how to build your own local AI assistant.

Why not use a game engine?

You could, but a game engine adds a lot to learn before you see anything. A single HTML page with a canvas loads anywhere, can be hosted for free and is easy for an AI assistant to help you change. Start simple and move to an engine only if you outgrow it.

Start small

You do not need eleven steps on day one. Get one building and one walking person working from a real log file. Once you see your own automation move across the screen for the first time, you will want to build the rest.

See the finished village, live ↗

If you build your own village, I would love to see it. The full version of mine runs live at garystow.co.uk/village, and I write about practical AI for marketers at garystow.co.uk.