DEV Community

Cover image for How to Setup an LLM Model Locally on Your Machine (Linux)
Ganesh Kumar
Ganesh Kumar

Posted on Edited on

How to Setup an LLM Model Locally on Your Machine (Linux)

#git

Hello, I'm Ganesh Kumar, and I'm building LiveReview — a blast-radius aware AI code review built for your business-critical systems. Star us to help devs discover the project, give it a try, and share your feedback to help improve the product.

In the age of ChatGPT and Claude, we often rely on cloud services to run AI. But did you know you can run powerful Large Language Models (LLMs) right on your own laptop?

Running an LLM locally offers total privacy, zero subscription fees, and offline access. In this guide, we’ll walk through exactly how to set up Google’s lightweight Gemma 2 (2B) model on a Linux machine using a tool called Ollama.

1. What is an LLM?

A Large Language Model (LLM) is a type of Artificial Intelligence trained on massive amounts of text data. Think of it as a super-advanced "autocomplete" that understands context, logic, and coding.

While famous models like GPT-4 have trillions of parameters (the "brain cells" of the AI), modern "Small Language Models" (SLMs) like Gemma 2 (2B) are optimized to run on consumer hardware without needing a massive server farm.

2. Prerequisites

  • A computer running Linux (Ubuntu, Mint, Fedora, etc.).
  • Basic familiarity with the Terminal.
  • At least 4GB of RAM (The Gemma 2B model is very efficient).

3. The Tool: Why Ollama?

We will use Ollama, the most popular open-source tool for running LLMs locally. It handles all the complex configuration (drivers, weights, interfaces) automatically.

Step 1: Install Ollama

Open your terminal and run the official installation script. This single command downloads and installs the Ollama service.

curl -fsSL https://ollama.com/install.sh | sh
Enter fullscreen mode Exit fullscreen mode

Note: You may be asked for your sudo password to complete the installation.

Step 2: Verify the Installation

Once the script finishes, ensure Ollama is running by checking its version:

ollama --version
Enter fullscreen mode Exit fullscreen mode

If you see a version number (e.g., ollama version is 0.5.4), you are ready to go.

4. Get the Model: Gemma 2 (2B)

We will use Gemma 2:2b. This is Google's open model. It is heavily optimized, making it fast and surprisingly smart for its size.

To download and run the model in one go, use the following command:

ollama run gemma2:2b
Enter fullscreen mode Exit fullscreen mode

What happens next?

  1. Ollama will pull the model manifest.
  2. It will download the model layers (approx. 1.6 GB).
  3. Once finished, it will drop you directly into a chat prompt.

5. Test: Your First Prompt

Once the download is complete, you will see a prompt that looks like >>>. You are now chatting directly with the AI on your machine.

Try this simple test prompt:

>>> Write a Python function to check if a number is prime.
Enter fullscreen mode Exit fullscreen mode

Expected Output:
The model should instantly generate code similar to this:

def is_prime(n):
    if n <= 1:
        return False
    for i in range(2, int(n**0.5) + 1):
        if n % i == 0:
            return False
    return True
Enter fullscreen mode Exit fullscreen mode

Another fun test:

>>> Why is the sky blue? Explain it like I'm 5.
Enter fullscreen mode Exit fullscreen mode

6. Managing Your Models

To exit the chat, press Ctrl + d.
To see which models you have installed later, simply run:

ollama list
Enter fullscreen mode Exit fullscreen mode

Conclusion

You now have a private AI assistant running entirely on your Linux machine! You can use it to summarize logs, write scripts, or brainstorm ideas without sending a single byte of data to the cloud.

I'm building LiveReview, a blast-radius aware AI code review built for your business-critical systems.

Instead of presenting every diff with equal emphasis, LiveReview scores each change by blast radius — how far its impact reaches through your call graph — so you can focus attention where it actually matters.

Spend code review effort where business risk is highest — not spread evenly across every diff.

⭐ Star it on GitHub:

GitHub logo HexmosTech / LiveReview

Blast-Radius Aware AI Code Review for Business-Critical Systems

LiveReview

gitleaks.yml osv-scanner.yml govulncheck.yml semgrep.yml dependabot-enabled mcp-testcases.yml

LiveReview: Blast-Radius Aware AI Code Review for Business-Critical Systems

LiveReview is an AI code reviewer that scores every hunk of a diff by blast radius: how far a change reaches through your call graph, how much persistent state it touches, and how well-tested it is. A 3-line change to a shared auth check can outrank a 300-line UI tweak. Your team's attention goes to the highest-risk code first, not spread evenly across every diff.

blast-radius-demo.mp4

LiveReview's Blast Radius & Review Priority scoring, live in the diff viewer.
















The exact math, not a black box Visualize blast radius at a glance Every factor that feeds the score

How does Blast Radius scoring work? (a more technical explanation)

Here's the goal:

  • A 3-line fix in a function used by 40 other files, that also writes to a database, should score high.
  • A 300-line UI change in one file, fully covered by…


View on GitHub


Click below to try LiveReview with your codebase:

LiveReview Banner

Top comments (0)