DEV Community

Pratyush Mohanty
Pratyush Mohanty

Posted on

Running llama3 in WSL2 using Docker in your PC 🐧🦙🐋

#linux #ai #llm #tutorial

In order to run Ollama in your local system, the best way is to use docker in the Windows Subsystem for Linux. This is a tutorial going over what needs to be done.

🐧 Installing WSL and Docker on Windows

This guide will walk you through setting up Windows Subsystem for Linux (WSL) and Docker inside WSL on a Windows machine.

1. Enable WSL

Open PowerShell as Administrator and run:

wsl --install
Enter fullscreen mode Exit fullscreen mode

This installs Ubuntu as the default distribution. If WSL 2 isn't already enabled, this command handles it too.

Powershell WSL install

Reboot Your PC

This will install Ubuntu. Once done, restart your PC to complete the setup.

Set Up Your Linux Distribution

After reboot, WSL will launch and prompt you to create a username and password.

If it doesn't launch automatically, run:

wsl
Enter fullscreen mode Exit fullscreen mode

Or launch Ubuntu from the Start Menu.

Update Ubuntu

Inside WSL terminal:

sudo apt update && sudo apt upgrade -y
Enter fullscreen mode Exit fullscreen mode

2. 🐳Install Docker in WSL (Ubuntu)

Instructions taken from Docker Ubuntu Installation.

Add the repository to Apt sources: 

# Add Docker's official GPG key:
sudo apt-get update
sudo apt-get install ca-certificates curl
sudo install -m 0755 -d /etc/apt/keyrings
sudo curl -fsSL https://download.docker.com/linux/ubuntu/gpg -o /etc/apt/keyrings/docker.asc
sudo chmod a+r /etc/apt/keyrings/docker.asc

# Add the repository to Apt sources:
echo \
  "deb [arch=$(dpkg --print-architecture) signed-by=/etc/apt/keyrings/docker.asc] https://download.docker.com/linux/ubuntu \
  $(. /etc/os-release && echo "${UBUNTU_CODENAME:-$VERSION_CODENAME}") stable" | \
  sudo tee /etc/apt/sources.list.d/docker.list > /dev/null
sudo apt-get update
Enter fullscreen mode Exit fullscreen mode

Install the Docker packages.

To install the latest version, run:

sudo apt-get install docker-ce docker-ce-cli containerd.io docker-buildx-plugin docker-compose-plugin
Enter fullscreen mode Exit fullscreen mode

Run Docker without sudo

(Optional but recommended):

sudo usermod -aG docker $USER
Enter fullscreen mode Exit fullscreen mode

Then restart WSL or your terminal:

exit
Enter fullscreen mode Exit fullscreen mode

And open it again. Test with:

docker run hello-world
Enter fullscreen mode Exit fullscreen mode

The following screen should be visible

Docker Hello-world Image

3. 🦙 Installing Ollama in the WSL Ubuntu Environment

We will install and run Ollama using Docker, following the instructions on the official to access the docker Image Ollama Docker page.

Since I’m running this on a laptop without GPU support, I opted for the CPU version. If you have a GPU, using the GPU version is highly recommended for better performance.

1. Run the Docker container: 

docker run -d -v ollama:/root/.ollama -p 11434:11434 --name ollama ollama/ollama
Enter fullscreen mode Exit fullscreen mode

2. Run a model (e.g., llama3): 

docker exec -it ollama ollama run llama3
Enter fullscreen mode Exit fullscreen mode

This will download and run the llama3 model (or whichever model you specify). You can now interact with it.

Use the command below to see help and available options:

/?
Enter fullscreen mode Exit fullscreen mode

Ollama running in WSL

3. Starting Ollama on future sessions:

When you reopen WSL, you can start the container using:

docker start ollama
Enter fullscreen mode Exit fullscreen mode

Then, re-run your desired model:

docker exec -it ollama ollama run llama3
Enter fullscreen mode Exit fullscreen mode

Running llama3

4. Accessing via browser or client:

Once running, visit:

http://localhost:11434
Enter fullscreen mode Exit fullscreen mode

You should see a response like:

{"message":"Ollama is running"}
Enter fullscreen mode Exit fullscreen mode

Localhost Ollama Message

5. To stop Ollama: 

docker stop ollama
Enter fullscreen mode Exit fullscreen mode

Top comments (0)