Introduction
llama.cpp is a wonderful project for running llms locally on your system. It is lightweight and provide state-of-the-art performance. It comes with GPU offloading support allowing you to use your GPU capabilities to run llms.
I am personally using it for running llms on my arch system and I found it better than ollama in terms of performance, while trying to set this it up, I found their documentation confusing and there were no guides specifically for arch linux, that's why I decided to write an article after figuring things out.
So, lets gets started with it.
Guide
- lets start by cloning the repo and cd into that
git clone https://github.com/ggerganov/llama.cpp.git
cd llama.cpp
- After that we need to build this, it can be done by using
make, but like me if you have nvidia GPU then for using it to for offloading you will need to build it with cuBLAS and for that we need cuda toolkit, You can download it from your Linux distro's package manager. (e.g.apt install nvidia-cuda-toolkit) here I am using aur helper, you can also install it manually.
paru -S cuda
now to build project with cuda run
make GGML_CUDA=1
Using CMake:
cmake -B build -DGGML_CUDA=ON
cmake --build build --config Release
It might take a while, after build is finished we can now finally run llms.
The environment variable GGML_CUDA_ENABLE_UNIFIED_MEMORY=1 can be used to enable unified memory in Linux. This allows swapping to system RAM instead of crashing when the GPU VRAM is exhausted
How to use
In llama.cpp folder you will find a executable named llama-server and llama-cli
which can be used to run the model
for cli mode just run
./llama-cli -m path-to-model -n no-of-tokens -ngl no-of-layers-to-offload-to-gpu
And for server run this
./llama-server -m path-to-model -n no-of-tokens -ngl no-of-layers-to-offload-to-gpu
You can now open your browser and in the URL section type http://localhost:8080/ and a web UI will appear.
Now you can have fun with your local llm.
Hope you will find this article helpful.
Top comments (0)