Local LLMs (Ollama): Why They Don't Always Deliver as Expected

A few months ago, when I decided to develop a custom summarization module for my note-taking application, I turned to Ollama, a local LLM solution, due to cost and data privacy concerns. The initial test results, especially in terms of processing speed and consistency, were significantly below my expectations. Accustomed to the convenience offered by cloud-based solutions, I realized during this process that the promises of local LLMs don't always translate into real-world performance.

Performance issues with local LLMs usually cannot be attributed to a single cause; they often arise from a combination of multiple factors such as hardware limitations, model selection, context window management, and even integration architecture. In this post, I will delve into why we don't always get the expected efficiency from local LLM solutions like Ollama and the underlying technical reasons for this.

What are Local LLMs (Ollama) and Why Are They Appealing?

Ollama is a user-friendly tool that enables easy execution of large language models (LLMs) on local machines. Its ability to download and run a model with a single command offers a very attractive starting point for developers and enthusiasts. This platform stands out for its capability to run on a CPU for most models, eliminating the need for specialized hardware.

The main reasons that make Ollama appealing are data privacy and cost. When working with sensitive data, running models on our own servers or computers instead of sending data to cloud-based APIs provides a significant security advantage. Furthermore, especially with heavy usage, local execution costs (after the initial hardware investment) are considerably lower compared to the token-based pricing of cloud APIs. These factors lead many developers, including myself, towards local solutions.

ℹ️ Key Advantages of Ollama

Ollama provides a simple interface for running LLMs locally. This offers:

• Data Privacy: Sensitive data never leaves your device.
• Cost Control: Avoids API fees, making it more economical in the long run.
• Full Control: Provides more control over how the model operates and with what data it was trained.

However, these advantages come with certain challenges. The performance potential of local LLMs is directly related to the power of the hardware used and the complexity of the model. This can lead to disappointment, especially for users with high initial expectations.

Why Do Performance Expectations Often Fall Short?

A common situation with local LLMs is that the impressive speed and performance promises in promotional materials don't always hold true in real-world scenarios. The number of tokens a model generates per second (tokens/s) depends on many variables, from the hardware used to the model's architecture. This complexity leads to a gap between expectations and reality.

This was my experience with my own application. Even summarizing a small piece of text took longer than I expected, negatively impacting the user experience. This inconsistency often stems from overlooking the fundamental factors that affect LLM performance. For example, a model being described as "fast" usually reflects optimal results achieved on a specific hardware configuration, and this configuration may not be accessible to every user.

⚠️ Misleading Performance Metrics

Most LLM performance comparisons are often conducted on high-performance and expensive GPUs. Expecting the same results on your own machine is unrealistic. Models running on a CPU will be significantly slower than on a GPU.

Factors such as model size, the level of quantization used, and the context window directly impact performance. Users often focus on the "power" of the model itself, overlooking the critical role these technical details play in performance. Therefore, when choosing a local LLM solution, it's necessary to deeply examine its technical specifications and compatibility with your own hardware, rather than just the model's name or general reviews.

Hardware Barriers: The Importance of CPU, RAM, and Especially GPU

One of the most critical factors determining the performance of local LLMs is, undoubtedly, the hardware used. The GPU (Graphics Processing Unit), in particular, is indispensable for the high-speed and efficient operation of such models. LLMs rely on intensive parallel computations like matrix multiplications, and GPUs, with their thousands of cores, can perform these operations many times faster than CPUs. On my own system, when I ran Ollama with an older CPU and limited RAM, I noticed that the text generation speed was noticeably slower compared to cloud-based APIs.

While CPUs are optimized for general-purpose computing, GPUs are designed to process specific types of operations (especially graphics and AI) in parallel. Running an LLM solely on a CPU leads to significant performance degradation, especially for large models or long context windows. Since model parameters and context data are held in memory, if there isn't enough RAM or VRAM (GPU memory), the system has to read data from disk, which increases latency. This was a situation I observed particularly when trying to run even a 7B (7 billion parameter) model on a laptop with 8GB RAM; the system frequently froze, and responses were very slow.

💡 Hardware Tips

To improve local LLM performance:

• High VRAM GPU: If possible, opt for a GPU with high VRAM, such as the NVIDIA RTX series. GPUs with less than 8GB VRAM may be insufficient for large models.
• Sufficient System RAM: At least 16GB or 32GB of system RAM is recommended for the entire model and context to remain in memory.
• Fast Storage: SSD or NVMe drives reduce model loading times.

In some cases, it may not be possible to fit an entire model onto the GPU. In this scenario, part of the model runs on the GPU, and part runs in system RAM (CPU offloading). This complex distribution can further impact performance because data transfer between GPU and CPU memory creates an additional bottleneck. Therefore, a powerful GPU and sufficient system memory are critical to achieving the expected performance from local LLMs.

What Do Model Size and Quantization Mean?

Two important concepts that directly affect the performance and usability of local LLMs are model size and quantization techniques. Model size refers to the number of learnable parameters in an LLM (e.g., 7 billion parameters - 7B). Larger models generally tend to be more capable and produce more accurate responses, but this implies greater computational power and memory requirements. For users with limited hardware like myself, running models like 70B or 13B locally is often either impossible or unacceptably slow.

Quantization, on the other hand, is the process of converting a model's parameters to a lower bit precision. For example, reducing default 32-bit (FP32) float values to lower bit precisions like 16-bit (FP16), 8-bit (INT8), or even 4-bit (INT4) shrinks the model's file size and reduces memory usage. This allows the model to run with less RAM or VRAM and can increase processing speed. However, quantization, especially at very low bit levels (e.g., Q4_K_M), can lead to a noticeable decrease in the model's accuracy and output quality.

Ollama offers many model variants at different quantization levels (e.g., llama2:7b-chat-q4_K_M or llama2:7b-chat-q8_0). More aggressive quantizations like Q4_K_M allow the model to fit into smaller memory footprints, while less aggressive ones like Q8_0 offer better accuracy but require more memory. Understanding this trade-off is vital for setting realistic expectations for local LLMs. In my side project, I initially tried to use a quantized version of a high-quality model, but its summarization capabilities were not as good as I expected; so I had to switch to a larger, but still quantized, model. This switch, while straining my hardware, significantly improved the output quality.

Context Window and Context Management Issues

An LLM's "context window" refers to the number of tokens the model can process at once. This is a critical factor that determines how well the model can "remember" a chat history, an entire article, or a piece of code. When working with local LLMs, the size of the context window has a significant impact on performance and memory usage. For example, if a model has a 4096-token context window and you provide it with an 8000-token long text, the model will either truncate the text or experience a severe performance drop.

Large context windows allow the model to process more information at once, enabling it to produce more consistent and comprehensive responses. However, each additional token increases the amount of data the model needs to hold in memory and thus raises the computational load. This becomes a significant bottleneck, especially on GPUs with limited VRAM or systems running solely on a CPU. When I tried to use a local LLM to summarize a complex planning report in my production ERP, I observed that due to the length of the report, the model either lost its context or took an excessively long time to generate a response.

🔥 Risks of Long Context

Inputs exceeding the context window can lead to:

• Performance Degradation: The model's token generation speed dramatically decreases.
• Memory Insufficiency: Can lead to Out Of Memory (OOM) errors, especially in VRAM.
• Quality Loss: The model may produce superficial or irrelevant responses because it cannot process the entire input.

Techniques like Retrieval-Augmented Generation (RAG) come into play for context management. RAG aims to use the context window more efficiently by retrieving relevant chunks from large documents and presenting only these chunks to the LLM. This ensures that the model focuses only on the most relevant information, improving both performance and response quality. In my Android spam application, when analyzing incoming SMS messages, instead of feeding long messages directly to the LLM, I narrowed the context through keyword extraction and summarization steps, achieving more efficient results. This approach enables local LLMs to work more effectively with limited resources.

Application Architecture and Integration Challenges

Integrating local LLMs into an application requires much more than just downloading and running a model. The application architecture is a crucial factor that largely determines whether you will get the expected efficiency from local LLMs. Solutions like Ollama typically offer integration via a REST API or Python libraries. However, the latency of these API calls directly affects the overall response speed of the application. In a financial calculator I developed, when instant feedback to user input was required, the response from the local LLM sometimes took unacceptably long, negatively impacting the user experience.

Cloud-based LLM APIs generally offer low latency and high concurrent request processing capacity. In a local setup, this capacity is directly limited by the hardware power of a single machine. If multiple users or applications send requests to the local LLM simultaneously, queues can form, and response times can lengthen. This can create a significant bottleneck, especially in enterprise software or systems accessed by multiple users.

import ollama

def generate_response_local(prompt: str, model_name: str = 'llama2'):
    """
    Gets a response from the local LLM using the Ollama API.
    """
    try:
        response = ollama.generate(model=model_name, prompt=prompt)
        return response['response']
    except Exception as e:
        print(f"Error connecting to Ollama or getting response: {e}")
        return None

# Usage example
if __name__ == "__main__":
    my_prompt = "What is the capital of Turkey?"
    result = generate_response_local(my_prompt)
    if result:
        print(f"LLM Response: {result}")
    else:
        print("Could not get a response from the local LLM.")

A simple Python code like the one above demonstrates the basis of integration with Ollama. However, in a real application, issues such as the concurrency of these calls, error handling, caching responses, and, if necessary, routing to different models complicate the architecture. In a client project, realizing that the local LLM alone could not handle the entire load during peak reporting periods, we had to set up a cloud-based fallback mechanism for high-priority or more complex requests, while using the local model for non-critical tasks. This hybrid approach both kept costs under control and met performance expectations.

Strategies for Maximizing Efficiency from Local LLMs

Getting the best performance from local LLMs requires conscious choices and optimization strategies. Instead of just downloading and running a model, you should carefully plan your system and approach. In my own experience, each of these strategies played a significant role in realizing the potential of local LLMs.

First, optimizing your hardware is a fundamental step. If you don't have a GPU or it's insufficient, you should consider investing in one if possible. Specifically, a GPU with 12GB or 16GB VRAM makes a noticeable difference for running models like 7B or 13B at satisfactory speeds. System RAM is also important for the model and context to remain in memory; 32GB RAM is generally a good starting point.

Second, choosing the right model and quantization level is critically important. The largest model is not always the best. Start with smaller (e.g., 3B or 7B) and well-quantized models according to your application's needs. Quantizations like q4_K_M or q5_K_M can offer a good balance between performance and quality. Regularly evaluate the quality of the model's outputs and try different quantization levels if necessary.

Third, focus on prompt engineering and context management. Try to keep the prompt given to the LLM as short and clear as possible. Eliminate unnecessary information. If you're working with long texts, use techniques like RAG (Retrieval-Augmented Generation) to present only the most relevant information to the model. This both reduces token usage and enables the model to respond faster and more accurately.

Fourth, consider batching requests and caching strategies. If your application generates multiple LLM requests simultaneously, batching these requests and sending them in a single call can improve performance. Additionally, caching LLM responses for frequently asked questions or repetitive inputs eliminates unnecessary computational load. In my anonymous Turkish data platform, I improved overall performance by caching responses to frequently asked queries for some data analysis summaries.

Finally, establish monitoring and error management mechanisms. Monitoring metrics like memory and CPU/GPU usage of local LLMs allows you to detect bottlenecks and performance issues early. Following Ollama's logs (e.g., journalctl -u ollama) helps you understand potential errors and why the model isn't working as expected. In a production ERP, when I saw unexpected increases in the response time of a particular API, I quickly identified it was due to a memory leak in the LLM service thanks to my monitoring tools.

By implementing these strategies, you can overcome the limitations of local LLMs and achieve more efficient results in your projects.

Conclusion

Local LLMs, especially with tools like Ollama, undoubtedly offer great potential in terms of data privacy and cost control. However, I've learned from my own experiences that this potential doesn't always align with expectations, and often factors such as hardware limitations, model size, quantization levels, context window management, and application integration are decisive. From the delays I encountered in "a corporate project's reporting module" to the summarization issues in "my own mobile application," these problems always pushed me to think more deeply and develop more pragmatic solutions.

In summary, to get the maximum benefit from local LLMs, simply running the model is not enough. You must realistically assess your hardware's capacity, choose the most suitable model and quantization level for your application's needs, effectively manage the context window, and establish a robust integration architecture. By following these steps, you can turn the promises of local LLMs into reality in your own projects. In my next post, I will address the security risks and mitigation strategies that may arise when moving such local models to production environments.