DEV Community

Noor Latif
Noor Latif

Posted on • Edited on

Running Local AI with Flox and Ollama

#ollama #ai #python #devops

Flox released a guide on how and why you should use Flox to run Ollama and how it creates reproducible development environments that work the same on any machine. Without the overhead of containers and a Dockerfile build process. This guide is slightly different in that it simplifies the setup process to run a local AI model. Flox runs on Nix underneath. Some people say Nix is so efficient at isolated environments that they think Docker would never have been created if the inventors knew about Nix.

Install Flox

The easy way with my community maintained installer

curl -sSf https://raw.githubusercontent.com/noor-latif/get-flox/refs/heads/main/install.sh | bash
Enter fullscreen mode Exit fullscreen mode

Set Up Your Environment

Initialize Flox in a folder of your choice:

flox init
Enter fullscreen mode Exit fullscreen mode

Install Python and Ollama:

flox install python3 ollama
Enter fullscreen mode Exit fullscreen mode

Configure Ollama as a Service

Edit your Flox environment:

flox edit
Enter fullscreen mode Exit fullscreen mode

Add these configurations:

Under [services]:

ollama.command = "ollama serve"
Enter fullscreen mode Exit fullscreen mode

Under [vars] (to make the API accessible on your network):

OLLAMA_HOST = "0.0.0.0:11434"
Enter fullscreen mode Exit fullscreen mode

Start Everything

Activate Flox with services:

flox activate -s
Enter fullscreen mode Exit fullscreen mode

The -s flag automatically starts services (like Ollama) when you enter the environment.

Run an AI Model

Deploy a lightweight model:

ollama run tinyllama
Enter fullscreen mode Exit fullscreen mode

This starts an interactive chat. Press Ctrl+D to exit. The model API remains available at http://localhost:11434 while the service is running.

Verify Service Status

Check if Ollama is running:

flox services status
Enter fullscreen mode Exit fullscreen mode

If you need to manually stop/start:

flox services stop
flox services start
Enter fullscreen mode Exit fullscreen mode

Why This Works Better

Flox handles the complexity automatically. When you exit the environment with exit, Flox cleanly stops all services. No orphaned background processes to track down.

Bonus: Auto-Activate with Direnv

Tired of forgetting flox activate? Install direnv to automatically activate environments when you cd into project folders.

Create a default environment and install direnv on the home folder level:

flox init -d $HOME
flox install -d $HOME direnv
Enter fullscreen mode Exit fullscreen mode

Add to your ~/.bashrc (or ~/.zshrc for zsh, etc...):

eval "$(flox activate -d $HOME)"
eval "$(direnv hook bash)" # bash or zsh/fish/tcsh/starship
Enter fullscreen mode Exit fullscreen mode

Reload your shell:

source ~/.bashrc
Enter fullscreen mode Exit fullscreen mode

In any Flox project, create an .envrc file:

echo 'use flox' > .envrc && direnv allow
Enter fullscreen mode Exit fullscreen mode

Now the environment activates automatically when you enter the directory and deactivates when you leave.

Resources:

Top comments (0)