DEV Community: Matúš Námešný

Building an AI Homelab

Matúš Námešný — Wed, 24 Jun 2026 12:57:46 +0000

As part of my job over the past year, I've been deploying and evaluating LLMs locally on various hardware setups. Naturally, I became well versed in the open source tools and models available and various hardware options. So I decided to build my own homelab for local inference.

My goal was to have a playground to try out different models, tools, and agent frameworks. Out of the hardware options available, I chose a Mac Mini M4 Pro for its balance of performance and power efficiency. It also fits nicely under my TV and can double as a media center.

All configuration files are available in my homelab repo.

Note: This article was originally published on my personal website in February 2026 and updated in June 2026.

Homelab Overview

Here's a high-level overview of my homelab setup. I'll go into more detail on each component later.

The Homelab is connected to my cloud VPS via Tailscale. On the cloud VPS, Traefik acts as a reverse proxy, forwarding requests to my Mac Mini over the Tailscale network. On the Mac Mini, Nginx proxies requests to various services running locally. Additionally, if a service requires authentication, it's handled by Authelia on the VPS using auth_request.

The inference backbone is handled by llamactl, which manages multiple model instances using llama-server and MLX. The entrypoint for user interaction is Mikoshi, previously called AgentKit. There's also an audio service for ASR and TTS using MLX models, and Glances for basic system monitoring.

How It Evolved

The first iteration of my homelab started with using Ollama and OpenWebUI. However, I quickly ran into limitations with Ollama, which led me to develop llamactl, which I detailed in my Llamactl article. Ollama is also not a great open source citizen as they build on top of llama.cpp without proper acknowledgment. More recently, they implemented their own engine, which still copies heavily from llama.cpp without attribution (even bugs).

OpenWebUI provided a decent interface but it felt quite opinionated and opaque. For example, they didn't support model context protocol (MCP) servers and forced users to use their own MCP proxy MCPo. Their functions and valves system is interesting and could be powerful, but it's opaque. Settings are duplicated and scattered, some for admin, some for user, which gets confusing for my single user instance. Their custom license is just the final nail in the coffin for me even if it makes no difference for my personal use case.

But, ultimately, the main motivation for me was to get a hands-on experience with LLM clients, tools and agents not hidden behind a shiny UI.

Smolagents Experiment

To understand how agents work, I started with the smolagents framework. It is quite simple and easy to understand by diving into the code. I started by creating a simple ToolCallingAgent and wrapped it in a basic chat loop. As I extended the loop with tool calls and more complex logic, I realized I was basically re-implementing the core logic of smolagents. For now, I kept the smolagents integration as is and went on to add MCPs.

Mikoshi

The basic chat loop I built on top of smolagents grew into its own thing that I named Mikoshi (originally named AgentKit, but renamed to Mikoshi. Bonus points if you recognize the origin of the name). Over time I added more features like flexible AI providers, conversation memory, tool management, and plugins. When I started integrating MCPs, I ran into conflicts with how smolagents managed them. Since I had already built most of the core agent logic myself, I decided to remove the smolagents dependency entirely and complete my own simple agent framework.

MCPs

The Model Context Protocol (MCP) is an open standard that allows LLMs to integrate and share data with external tools and sources. Since it seems like it's becoming a de facto standard, I wanted to add MCP support to my setup using the official Python SDK. All the guides focus on writing servers, which explains the literal thousands of vibe coded and abandoned MCP servers on GitHub.

The documentation on writing clients is comparably sparse. The recommended way seems to be initializing an MCP client and session per message, which seems quite heavy. After a few experiments I went with keeping a global session open for each MCP server and reusing it for multiple messages.

Audio Services

To add audio capabilities, I set up a Whisper ASR and TTS service using mlx-audio library. I created a simple FastAPI app that exposes OpenAI compatible endpoints for transcription and text-to-speech synthesis. The app loads the models on demand and unloads them after a timeout to save memory the same way as llamactl.

Monitoring

For monitoring, I set up Glances which provides a web interface to monitor system resources like CPU, memory, disk and network usage. It's lightweight and easy to set up.

Tool Examples

Here are some example tools I've integrated into Mikoshi:

Task Tool: Uses CalDAV to manage tasks and todos.
Notes Tool: Integrates with a git repository to manage markdown notes using Gitea MCP.
Anki Tool: Create flashcards in Anki
Dictionary Tool: Lookup word definitions in a dictionary

macOS Side Notes

Coming from a Linux background, setting up a homelab on macOS was an interesting experience. Here are some side notes:

Homebrew is great for package management, but is nowhere near as comprehensive as apt or pacman
Launchd is useful for managing services, but the capabilities are quite limited compared to systemd. For example, no easy way to set resource limits or dependencies between services. Starting and stopping services is also more cumbersome.
Given that macOS is a Unix system, I was surprised to learn that Docker needs to run inside a Linux VM. This prevents me from using GPU acceleration for containerized workloads.

What's Next

For now, I mostly focus on adding more plugins and tools to Mikoshi and refining the existing ones. I want to get the most out of local LLMs and specific tools and agents seem to be the way to go. I also want to expand monitoring and possibly add tracing, prompt management, and memory capabilities.

I've also got Reachy Mini, a small open source robot. It seems like a perfect platform for building a voice assistant and I've recently started working on the integration.

Resources

Llama-Server is All You Need (Plus a Management Layer)

Matúš Námešný — Thu, 04 Sep 2025 21:06:52 +0000

If you're running LLMs locally, you've probably used Ollama or LM Studio. They're both excellent tools, but I hit some limitations. LM Studio is primarily a desktop app that can't run truly headless, while Ollama requires SSH-ing into your server every time you want to switch models or adjust parameters.

For more control, there's llama-server from llama.cpp. It's powerful, lightweight, supports virtually every model format, offers extensive configuration options, provides OpenAI-compatible APIs, and, in my opinion, is noticeably faster than Ollama. But it's CLI-only - want to switch models? SSH in and restart.

I wanted to manage my home LLM server from anywhere without constantly SSH-ing just to switch models. So I built what was missing: a management layer on top of llama-server.

Enter Llamactl

So I built what was missing: a management layer on top of llama-server. Meet llamactl - a management server and proxy that gives you the power of llama-server with the convenience of remote management.

You get a modern React web dashboard for visual management, REST APIs for programmatic control, and the ability to create, start, and stop instances with just a few clicks. Need a 7B model for quick responses and a 70B model for complex reasoning? Run both. Want to switch between them based on the task? Just change the model name in your API request.

It's also OpenAI API compatible, so your existing tools, scripts, and integrations work without modification - just point them to your llamactl server instead of OpenAI's endpoints. Want to use Open WebUI for a ChatGPT-like interface? Just configure it to use your llamactl server as the OpenAI API base URL, and you're instantly chatting with any of your local models.

What Llamactl Brings to the Table

Multiple Model Serving: Run different models simultaneously — a 7B model for speed, a 70B model for quality, or a vision model for image analysis. Switch between them by simply changing the model name in your API requests.
Web Dashboard: A modern React UI that beats SSH-ing into servers. Create instances, monitor health, view logs, and manage everything from your browser.
Smart Resource Management: Idle timeout automatically stops unused instances to save resources. LRU eviction ensures your most-used models stay available. Configurable instance limits prevent resource exhaustion.
On-Demand Starting: Point your application at a model that isn't running? Llamactl automatically starts it for you. No more "is the server running?" guesswork.
API Key Authentication: Separate keys for management operations vs inference requests.
State Persistence: Server restarts don't kill your carefully configured instances. Everything comes back exactly as you left it.

My Setup: LLMs Anywhere, Anytime

Here's how I've configured my home LLM infrastructure to work seamlessly from anywhere:

I run my LLMs on a Mac Mini M4 Pro at home. The 48GB of unified memory gives me enough room to run larger models like Gemma 3 27B or Qwen 3 Coder 32B and switch between them as needed.

Both my home Mac Mini and a cloud VPS are connected via Tailscale. This creates a secure, private network that lets them communicate as if they were on the same LAN, regardless of where I am.

The setup is straightforward:

Llamactl runs on the Mac Mini, managing my llama-server instances
Open WebUI also runs locally, providing a ChatGPT-like interface
Traefik runs on my VPS as a reverse proxy

Traefik on the VPS proxies requests through the Tailscale network to my home setup, giving me a clean public URL (like llm.mydomain.com) that securely tunnels to my home lab.

This setup lets me deploy any model, switch between them, and chat with my LLMs from anywhere with internet access. No VPN client needed, no SSH required, and Tailscale handles the security with zero-trust networking.

What's Next

Llamactl is still evolving, and I have several ideas for future improvements:

Enhanced Admin Dashboard: While the current web UI covers the essentials, I'd like to add proper user authentication with usernames and passwords, and better management of inference API keys.
Multiple Backend Support: Right now it's focused on llama-server, but adding support for other inference engines like vLLM or mlx_lm.server could make it even more versatile.
Built-in Chat Interface: Open WebUI is fantastic, but having a simple chat UI built directly into llamactl could reduce the setup complexity for basic use cases.
Better Resource Scheduling: Smart load balancing and automatic model placement based on hardware capabilities.

Get Involved

Llamactl is open source and available on GitHub. You can find complete documentation and guides at llamactl.org. Whether you're interested in trying it out, reporting bugs, suggesting features, or contributing code, I welcome all feedback and contributions.

If you're tired of SSH-ing into servers just to switch LLM models, give llamactl a try. It might just be the management layer you didn't know you needed.