AI Code Completion with Local LLM: Install Ollama in 5 Steps

One of the most common things I encounter while developing software these days is that the code completion suggestions in my IDE are usually provided by a cloud-based artificial intelligence. This situation makes me think about both code privacy and sometimes latency. Code completion with a local LLM (Large Language Model) offers an elegant solution to these problems, and Ollama is a very practical tool to bring this solution to life.

In this article, I will explain step-by-step how to set up an AI-powered code completion environment on your own machine using Ollama. My goal is to help you reduce reliance on cloud services and achieve a faster, more secure development experience. In five simple steps, we will start running your own local code assistant.

Why Have Local LLMs Become Important?

Recently, many developers write code with the ease offered by cloud-based AI tools. However, in some projects I develop, especially when working on production ERP systems that contain customer data or sensitive business logic, sending the code I write to an external service always makes me uneasy. Concerns about privacy and data security have increased my interest in local LLM solutions.

Local LLMs prevent your code from leaving your company network or personal computer. This is a great advantage, especially in projects subject to NDAs or when developing sensitive algorithms. Also, being able to use code completion and other AI features without an internet connection does not interrupt my workflow when I am doing mobile development or working while traveling.

What is Ollama and Why Should It Be Preferred?

Ollama is an open-source, simple, and powerful framework designed to run large language models locally. It is easy to install, practical to use, and supports a wide range of models. The main reasons I prefer Ollama are its ability to quickly set up LLMs in bare-metal or container environments and its excellent compatibility with Linux systems.

Thanks to this platform, you can easily download and run different LLMs (such as Code Llama, Deepseek Coder), and even customize your own models via Modelfiles. In terms of performance, with the right model selection and hardware optimization, it is possible to achieve experiences close to, or even better than, cloud services in some cases. Especially in situations requiring high bandwidth or low latency, the difference of a local solution is noticeable.

Step 1: Ollama Installation – An Operating System-Specific Approach

Installing Ollama is quite simple. Since I usually work on Linux systems, I will show the installation on a Debian/Ubuntu-based system. Detailed instructions for other operating systems can also be found on Ollama's official website. First, open your terminal and run the following command to download and install Ollama on your system:

curl -fsSL https://ollama.com/install.sh | sh

This command downloads Ollama, installs the necessary dependencies, and configures it as a systemd service. Once the installation is complete, Ollama will start running in the background. You can use the following command to check if the installation was successful:

ollama --version

If you see the version number, your installation is complete.

ℹ️ Installation Note

The curl command installs Ollama by default in the /usr/local/bin directory and creates an ollama user to run the service under this user. If you want to install it in a different directory or user, you can download the install.sh script and edit it manually. However, the default installation is sufficient for most scenarios.

Step 2: Pulling the First Code Completion Model (Choosing Code Llama)

After Ollama is installed, it's time to download a language model for code completion. Ollama supports many models, but Code Llama is one of the models specifically optimized for code completion. I usually prefer the 7B parameter version because it is powerful enough and offers acceptable performance even on average hardware.

To download the model, run the following command in the terminal:

ollama pull codellama

This command will pull the default (usually 7B) version of Code Llama. If you want a different version or quantization level, you can specify the model as codellama:13b or codellama:7b-code-q4_0. Since the model size can be several gigabytes, the download time may vary depending on your internet connection speed.

⚠️ Disk Space and Internet Speed

Models can be quite large, so make sure you have enough disk space and a stable internet connection. For example, even quantized versions of Code Llama 7B can take up several GBs. If you plan to download multiple models, consider your disk space.

After the model is downloaded, you can start a simple chat to verify that Ollama is running and the model is ready for use:

ollama run codellama "Hello, can you write me a simple Fibonacci series function in Python?"

This command will run the Code Llama model and allow it to respond to you. I use this step to test that the model is working correctly and its basic capabilities.

Step 3: Integration: Using Ollama in Your IDE

Bringing the power of local LLM to our IDE is the step that provides the real productivity boost. Since I usually use VS Code, I will explain the integration through VS Code. There are many IDE extensions compatible with Ollama on the market, but the Continue extension is quite successful and offers a flexible structure in this regard.

First, open VS Code and go to the Extensions tab. Type "Continue" in the search bar and install the extension. Once installed, you will need to open your settings.json file and configure Continue to communicate with Ollama. You can find the settings.json file by opening Ctrl+, (Windows/Linux) or Cmd+, (macOS) and typing "Preferences: Open User Settings (JSON)" in the search box.

Add the following configuration to your settings.json file:

{
  "continue.models": [
    {
      "name": "codellama",
      "provider": "ollama",
      "model": "codellama",
      "apiBase": "http://localhost:11434"
    }
  ],
  "continue.defaultModel": "codellama"
}

This configuration defines a model named codellama for the Continue extension and tells it to connect to the Ollama server at http://localhost:11434 via the Ollama provider. The apiBase address is Ollama's default API address. If you are running Ollama on a different port or IP address, you will need to change this value.

You may need to restart VS Code after saving the configuration. Now, the Continue extension will use the Code Llama model on your local Ollama server for code completion and other AI-based development tasks. When you start writing code, you will see the extension providing context-aware suggestions.

Step 4: Performance Tips and Optimizations

Local LLMs can be resource-intensive, so optimizing performance is critical. On my own system, especially when memory-intensive services like PostgreSQL and Redis are running, I make some adjustments to prevent the LLM from overwhelming system resources. Here are a few tips:

1. Model Selection and Quantization: You don't always have to use the largest model. For example, quantized versions of Code Llama with 7B parameters (like Q4_0 or Q5_K_M) often offer good performance with smaller file sizes and less memory consumption. Experiment by pulling specific quantization levels like ollama run codellama:7b-code-q4_0.
2. GPU Usage: If your system has a compatible GPU (NVIDIA or some AMD models), Ollama will automatically try to use GPU acceleration. This provides a significant performance increase compared to the CPU. You can check if running models are using the GPU with the ollama ps command.

3. Memory Management: LLMs typically consume a lot of memory. In Linux, you can limit the amount of memory allocated to the Ollama service with cgroup limits or systemd units. This prevents your other critical services from being terminated by the OOM (Out Of Memory) killer. For example, by creating a systemd override file:
   

   # /etc/systemd/system/ollama.service.d/limits.conf
   [Service]
   MemoryLimit=8G
   

   Then restart the service with sudo systemctl daemon-reload && sudo systemctl restart ollama. This way, you prevent Ollama from using more than 8GB of RAM.

4. Unloading Idle Models: Ollama keeps used models in memory. You can free up resources by unloading models you are not using from memory with the ollama unload <model_name> command.

💡 Monitoring System Resources

Monitor your system resources with tools like htop or nvidia-smi (if you are using an NVIDIA GPU) while Ollama is running. This helps you understand how much memory or GPU each model is using. When developing AI-powered features in the backend of one of my side products, I detected memory leaks and unnecessary resource consumption early thanks to these monitorings.

Step 5: Advanced Usage Scenarios and Next Steps

With Ollama, you can not only do code completion but also use it for more advanced scenarios. I integrate Ollama in different ways in my own projects.

1. Custom Models (Modelfile): If you want to change the behavior of an existing model or fine-tune it on your own small dataset, you can create Modelfiles. A Modelfile works similarly to a Dockerfile and allows you to add additional instructions (system messages, temperature settings, custom instructions) on top of a base model. For example, with a Modelfile, you can make the model generate only Python code or adhere to a specific coding style.
   

   FROM codellama:7b-code-q4_0
   PARAMETER temperature 0.2
   SYSTEM """You are a highly skilled Python developer. Only generate Python code, no explanations."""
   

   You can create this Modelfile with the command ollama create my-python-coder -f Modelfile and use it with ollama run my-python-coder.

2. RAG (Retrieval-Augmented Generation) Integration: You can use RAG architectures to search for specific code snippets in large codebases and get AI assistance in that context. In an integration I made into my own system, I divide my codebase into chunks and store them in a vector database. When a user asks a question, relevant code snippets are retrieved and given as context to the LLM in Ollama. This way, the model can generate answers not only with general knowledge but also related to the project's own code structure.

3. Multiple Provider Fallback: In the AI applications I develop, there are sometimes situations where the local model is insufficient or a more powerful model is needed. In these cases, I set up an architecture that uses Ollama as the primary provider and falls back to cloud-based services like Groq or OpenRouter if the response quality is low or a certain token limit is exceeded. This both optimizes costs and increases flexibility.

Such advanced usage scenarios show how powerful local LLMs can be beyond just code completion. Shaping Ollama according to your own needs can make your development process much more efficient.

Conclusion

AI-powered code completion with local LLMs transforms our development experience by offering privacy, speed, and cost advantages. Ollama is a simple yet powerful tool that brings this power to our own machines. By following the 5 steps I explained in this article, you can quickly set up and run your own local AI code assistant.

In my own projects, especially when working with sensitive data or when I don't have an internet connection, the local capability provided by Ollama has been invaluable. This setup is just the beginning; thanks to the flexibility offered by Ollama, you can do much more by creating your own Modelfiles or integrating with more advanced systems like RAG. The next step could be building your own custom code assistant.