DEV Community

Cover image for Setting Up a Local AI Coding Agent with Ollama and Aider (part 4)
eleonorarocchi
eleonorarocchi

Posted on

Setting Up a Local AI Coding Agent with Ollama and Aider (part 4)

#ai #llm

Limitations

Once the setup with Aider, Ollama, Grafana, and Streamlit was working, I decided to push it a little further.

The idea was to test a newer model based on a different approach from traditional autoregressive models: DiffusionGemma.

The model I wanted to try was:

google/diffusiongemma-26B-A4B-it
Enter fullscreen mode Exit fullscreen mode

The naming is interesting:

26B = total number of parameters
A4B = approximately 4B active parameters
Enter fullscreen mode Exit fullscreen mode

This is a Mixture-of-Experts model, but that does not mean it runs like a 4B model.

When running it locally, you still have to handle a very large model.

Why I Did Not Use It with Aider

My main workflow was:

Aider → Ollama → local model
Enter fullscreen mode Exit fullscreen mode

However, DiffusionGemma was not supported by Ollama in my setup. I therefore changed the goal: instead of using it with Aider, I tested it directly with the dedicated llama.cpp runner.

I did not expect it to become part of my daily workflow, but I still wanted to run a technical experiment.

Installing the Hugging Face CLI

Inside WSL, I installed the Hugging Face CLI using its dedicated installer:

cd ~
curl -LsSf https://hf.co/cli/install.sh | bash
exec $SHELL -l
Enter fullscreen mode Exit fullscreen mode

Then I verified the installation:

hf --help
Enter fullscreen mode Exit fullscreen mode

Downloading the GGUF Model

I downloaded the Q4_K_M quantization:

mkdir -p ~/models/diffusiongemma

hf download unsloth/diffusiongemma-26B-A4B-it-GGUF \
  --local-dir ~/models/diffusiongemma \
  --include "*Q4_K_M*"
Enter fullscreen mode Exit fullscreen mode

Then I checked that the file was present:

find ~/models/diffusiongemma -name "*.gguf" -type f
Enter fullscreen mode Exit fullscreen mode

The download completed successfully, but the file is large: approximately 16 GB.

Building the Dedicated Runner

To test the model without Ollama, I used the DiffusionGemma branch of llama.cpp.

First, I installed the required dependencies:

sudo apt update
sudo apt install -y git cmake build-essential
Enter fullscreen mode Exit fullscreen mode

Then I cloned the repository and checked out the relevant branch:

mkdir -p ~/src
cd ~/src

git clone https://github.com/ggml-org/llama.cpp llama.cpp-diffusiongemma
cd llama.cpp-diffusiongemma

git fetch origin pull/24423/head:diffusiongemma
git checkout diffusiongemma
Enter fullscreen mode Exit fullscreen mode

For a CPU-only build:

cmake -B build -DGGML_NATIVE=ON
cmake --build build -j $(nproc) --config Release --target llama-diffusion-cli
Enter fullscreen mode Exit fullscreen mode

To verify the build:

ls -l build/bin/llama-diffusion-cli
Enter fullscreen mode Exit fullscreen mode

Running the Model

I stored the model path in a variable:

MODEL=$(find ~/models/diffusiongemma -name "*Q4_K_M*.gguf" -type f | head -n 1)
echo "$MODEL"
Enter fullscreen mode Exit fullscreen mode

Then I launched the runner:

cd ~/src/llama.cpp-diffusiongemma

./build/bin/llama-diffusion-cli \
  -m "$MODEL" \
  -ngl 0 \
  -cnv \
  -n 256
Enter fullscreen mode Exit fullscreen mode

The command ran successfully, but on the machine I was using, performance was far too slow for practical use.

I also tried a lower output limit:

./build/bin/llama-diffusion-cli \
  -m "$MODEL" \
  -ngl 0 \
  -cnv \
  -n 32
Enter fullscreen mode Exit fullscreen mode

And another variant with an explicit thread count:

./build/bin/llama-diffusion-cli \
  -m "$MODEL" \
  -ngl 0 \
  -cnv \
  -n 64 \
  -t 10
Enter fullscreen mode Exit fullscreen mode

The Bottleneck

My machine has:

Intel Core i7-1355U
48 GB of RAM
Intel Iris Xe
no dedicated GPU
Enter fullscreen mode Exit fullscreen mode

The available RAM is enough to load many quantized models, but that does not make them fast enough for interactive use.

The real bottlenecks are:

mobile CPU
memory bandwidth
lack of dedicated VRAM
the higher computational cost of the diffusion architecture
Enter fullscreen mode Exit fullscreen mode

The fact that a model is described as 26B-A4B does not mean it is equivalent to a dense 4B model.

It only means that a subset of the parameters is activated for each token. The full model is still large, and when running it locally, you still pay the cost in memory usage, I/O, and runtime complexity.

Top comments (0)