Why your long llama.cpp chats get slow after a restart, and how to fix it

If you run models locally with llama.cpp, you have probably seen this. A long chat feels fine while the server is up. Then you restart the server, the machine reboots, the process crashes, or you swap models and come back. You send the next message and wait. And wait. The fans spin up, and only after a long pause does the first token show up.

Nothing is broken. The model is re-reading your whole conversation from the start. Here is why that happens and what you can do about it.

What is actually going on: prompt processing and the KV cache

Before a model can write a reply, it has to process every token of the prompt: the system prompt, every earlier message, every file your coding agent pasted in, and your new message. For each token, the model computes attention keys and values at every layer. Those results are stored in the KV cache.

While llama-server is running, it keeps that KV cache in memory for each slot. When you send the next message in the same chat, the server sees that the start of the prompt matches what it already processed, reuses the cached part, and only processes the new tokens. That is why a long chat can still feel quick while the server stays up.

The catch is that the KV cache lives in RAM or VRAM. When the process exits, it is gone. After a restart, crash or model reload, the server has nothing cached, so it has to process the entire conversation again before it can produce the first token. The longer the chat, the longer that wait. Local coding agents get hit hardest, because their sessions carry large contexts full of files, tool output and history.

The do-it-yourself fix: slot save and restore

llama-server has a built-in way to write a slot's KV cache to disk and load it back later.

Start the server with a folder for saved slots:

llama-server -m ./models/your-model.gguf --slot-save-path ./slot-cache/

Before you shut down, save the slot that holds your chat:

curl -X POST "http://127.0.0.1:8080/slots/0?action=save" \
  -H "Content-Type: application/json" \
  -d '{"filename": "my-chat.bin"}'

After the server comes back up with the same model, restore it:

curl -X POST "http://127.0.0.1:8080/slots/0?action=restore" \
  -H "Content-Type: application/json" \
  -d '{"filename": "my-chat.bin"}'

There is also ?action=erase to clear a slot. Once the slot is restored, your next request with the same conversation prefix can reuse the cache instead of processing everything again.

This works, and if you only have one chat you care about, it may be all you need. But it has some rough edges in daily use:

• You have to remember to save. A crash does not wait for you to run curl, so whatever was not saved is lost.
• You have to map chats to slots and files yourself. With several chats or agent sessions, you need your own bookkeeping to know which file belongs to which conversation and which slot to restore it into.
• Your client does not know about any of this. Open WebUI, Cline, Continue, aider and similar tools just send requests to an OpenAI-compatible endpoint. They will not call the slot endpoints for you, so you end up writing glue scripts.
• Saved state is tied to the model. Restore into the same model you saved from.

The automatic option: RAMen

Disclosure: I make RAMen. It is a paid add-on for llama.cpp that does the steps above for you.

RAMen saves every chat and agent session to disk automatically. After a restart or crash, a long chat comes back in seconds instead of the model re-reading the whole prompt, and it keeps long chats fast while you work. There is nothing to remember to save and no slot bookkeeping.

It serves an OpenAI-compatible API at http://127.0.0.1:8090/v1, so you point Open WebUI, Cline, Continue, aider or OpenCode at that address and keep using them as before.

It runs on your own machine: Macs with Apple silicon, and Windows or Linux x64 PCs on CPU or NVIDIA GPUs.

Some honest limits:

• RAMen uses its own llama.cpp server. It does not save sessions inside Ollama, LM Studio or Strata. If you use those, you would switch the server you connect to, not your chat app.
• AMD and Intel GPUs are not accelerated. On those machines RAMen runs on CPU.

There is a 30-second demo here: https://getramenai.com/watch

Every install starts with a 14-day trial of everything, no card needed. As a launch deal, the first 1,000 buyers can get Pro for life for $19.99, paid once. More at https://getramenai.com

Summary

Long chats get slow after a restart because the KV cache is in memory and disappears with the process, so the model has to process the whole conversation again. You can fix that yourself with --slot-save-path and the slot save and restore endpoints, as long as you save on time and track which file goes with which chat. If you would rather have it happen on its own, that is what RAMen does.

Drafted with AI assistance.