Run GGUF Models Locally on Linux with Local AI Server

Running AI models locally gives you more privacy, control, and independence from cloud services.

However, setting up llama.cpp, configuring GPU acceleration, managing multiple GGUF models, and running everything as a reliable background service can take time.

That is why I created Local AI Server.

GitHub repository:

https://github.com/hossbit/local-ai-server

What is Local AI Server?

Local AI Server is an open-source project that helps Linux users run GGUF language models locally using:

• llama.cpp
• llama-swap
• CPU or GPU acceleration
• An OpenAI-compatible API
• Automatic model discovery
• A systemd user service

It is designed for developers, Linux users, home labs, private AI environments, and people who want to use AI without depending on a permanent internet connection.

The project is lightweight and transparent. You maintain control over your models, configuration, API keys, and hardware.

Main Features

Local AI Server includes:

• Automatic discovery of .gguf model files
• On-demand model loading
• Switching between multiple models
• OpenAI-compatible API endpoints
• CPU inference
• NVIDIA CUDA support
• Vulkan support
• ROCm support
• OpenVINO support
• SYCL support
• Automatic hardware detection
• Model and runtime recommendations
• Multimodal model support
• Speculative decoding support
• Prometheus metrics
• API-key management
• User-level systemd integration
• Split GGUF model support
• Update and uninstall commands

Installation

Install Local AI Server with one command:

bash
curl -fsSL https://hossbit.github.io/localai/install.sh | bash