DEV Community

Noor Latif
Noor Latif

Posted on • Edited on

Running Local AI with Flox and Ollama

#ai #devops #python #ollama

Flox creates reproducible development environments that work the same on any machine. This guide shows you how to use it to run a local AI model.

Install Flox

Review the install script first (good security practice):

curl -LsSf https://gist.githubusercontent.com/noor-latif/3f64834ae832d8fbceac6214a2cc5e70/raw/aec3d6076e0e601a08ccf75de6c911ba61222750/flox-install.sh | less
Enter fullscreen mode Exit fullscreen mode

Then install:

curl -LsSf https://gist.githubusercontent.com/noor-latif/3f64834ae832d8fbceac6214a2cc5e70/raw/aec3d6076e0e601a08ccf75de6c911ba61222750/flox-install.sh | sh
Enter fullscreen mode Exit fullscreen mode

Set Up Your Environment

Initialize Flox in your project folder:

flox init
flox activate
Enter fullscreen mode Exit fullscreen mode

Install Python and Ollama:

flox install python3
flox install ollama
Enter fullscreen mode Exit fullscreen mode

Configure Ollama as a Service

Edit your Flox environment:

flox edit
Enter fullscreen mode Exit fullscreen mode

Add these configurations:

Under [services]:

ollama.command = "ollama serve"
Enter fullscreen mode Exit fullscreen mode

Under [vars] (to make the API accessible on your network):

OLLAMA_HOST = "0.0.0.0:11434"
Enter fullscreen mode Exit fullscreen mode

Start Everything

Activate Flox with services:

flox activate -s
Enter fullscreen mode Exit fullscreen mode

The -s flag automatically starts services (like Ollama) when you enter the environment.

Run an AI Model

Deploy a lightweight model:

ollama run tinyllama
Enter fullscreen mode Exit fullscreen mode

This starts an interactive chat. Press Ctrl+D to exit. The model API remains available at http://localhost:11434 while the service is running.

Verify Service Status

Check if Ollama is running:

flox services status
Enter fullscreen mode Exit fullscreen mode

If you need to manually stop/start:

flox services stop
flox services start
Enter fullscreen mode Exit fullscreen mode

Why This Works Better

Flox handles the complexity automatically. When you exit the environment with exit, Flox cleanly stops all services. No orphaned background processes to track down.

Bonus: Auto-Activate with Direnv

Tired of forgetting flox activate? Install direnv to automatically activate environments when you cd into project folders.

Create a default environment and install direnv on the home folder level:

flox init -d $HOME
flox install -d $HOME direnv
Enter fullscreen mode Exit fullscreen mode

Add to your ~/.bashrc (or ~/.zshrc for zsh, etc...):

eval "$(flox activate -d $HOME)"
eval "$(direnv hook bash)" # bash or zsh/fish/tcsh/starship
Enter fullscreen mode Exit fullscreen mode

Reload your shell:

source ~/.bashrc
Enter fullscreen mode Exit fullscreen mode

In any Flox project, create an .envrc file:

echo 'use flox' > .envrc && direnv allow
Enter fullscreen mode Exit fullscreen mode

Now the environment activates automatically when you enter the directory and deactivates when you leave.

Resources:

Top comments (0)