DEV Community

Deduped
Deduped

Posted on Originally published at dedupedai.com

How to serve an OpenAI-compatible local API with llama.cpp

#ai #llm #opensource #tutorial

Your app already talks to the OpenAI SDK. You want to test it against a free local model without rewriting anything. llama.cpp is a dependency-free C/C++ engine for running LLMs and vision language models, and its server exposes OpenAI-compatible chat completions, responses and embeddings routes. Point your client's base URL at it and most code just works.

This guide installs llama.cpp, starts a server with a model pulled straight from Hugging Face, and calls it with curl and the OpenAI Python library.

Install 

# macOS / Linux
curl -LsSf https://llama.app/install.sh | sh

# Windows (PowerShell)
irm https://llama.app/install.ps1 | iex
Enter fullscreen mode Exit fullscreen mode

The quick install pipes a script into your shell. If you would rather not, download pre-built binaries from the releases page, run it with Docker, or build from source. llama.cpp runs on macOS, Linux and Windows, with backends for CUDA, Metal, Vulkan, HIP, SYCL and more.

Try the model in a terminal first 

llama cli -hf ggml-org/Qwen3.5-0.8B-GGUF
Enter fullscreen mode Exit fullscreen mode

-hf downloads a GGUF model directly from Hugging Face. The quant is optional and defaults to Q4_K_M (for example ggml-org/GLM-4.7-Flash-GGUF:Q4_K_M).

Start the OpenAI-compatible server 

llama serve -hf ggml-org/Qwen3.5-0.8B-GGUF
Enter fullscreen mode Exit fullscreen mode

According to the server README, it listens on 127.0.0.1:9931 by default. Use --host and --port to change that. The built-in web UI is served at the same address, so open http://localhost:9931 in a browser to test the model before writing any code.

Check readiness with the health endpoint. It returns 503 while the model is loading and {"status": "ok"} once it is ready:

curl http://localhost:9931/health
Enter fullscreen mode Exit fullscreen mode

Call it with curl 

curl http://localhost:9931/v1/chat/completions \
  -H "Content-Type: application/json" \
  -H "Authorization: Bearer no-key" \
  -d '{
    "model": "gpt-3.5-turbo",
    "messages": [
      {"role": "user", "content": "Write a limerick about python exceptions"}
    ]
  }'
Enter fullscreen mode Exit fullscreen mode

Call it with the OpenAI Python library 

import openai

client = openai.OpenAI(
    base_url="http://localhost:9931/v1",
    api_key="sk-no-key-required"
)

completion = client.chat.completions.create(
    model="gpt-3.5-turbo",
    messages=[{"role": "user", "content": "Write a limerick about python exceptions"}]
)

print(completion.choices[0].message)
Enter fullscreen mode Exit fullscreen mode

That is the only change in your app: the base URL. GET /v1/models returns the loaded model. By default the model id is the path to the model file; set a friendlier one with --alias, for example --alias gpt-4o-mini. The README is careful to say it makes no strong claims of full OpenAI spec compatibility, but in their experience it is enough for many apps.

Useful server options

  • -c, --ctx-size N: context size (0 loads it from the model)
  • -ngl, --n-gpu-layers N: how many layers to keep in VRAM (auto by default)
  • --api-key KEY: require a key, so the server is not wide open if you bind it beyond localhost
  • --alias NAME: the model name your API clients will see

JSON output that matches a schema

/v1/chat/completions accepts response_format, both plain {"type": "json_object"} and schema-constrained JSON with {"type": "json_schema", "schema": {...}}. That is handy when your app expects structured output.

Run it in Docker instead 

docker run -p 8080:8080 -v /path/to/models:/models ghcr.io/ggml-org/llama.cpp:server -m models/7B/ggml-model.gguf -c 512 --host 0.0.0.0 --port 8080
Enter fullscreen mode Exit fullscreen mode

There is also a server-cuda image; add --gpus all and --n-gpu-layers 99.

Fit a bigger model on limited hardware

llama.cpp supports 1.5-bit to 8-bit integer quantization and CPU plus GPU hybrid inference, so models larger than your VRAM can still be partially accelerated. Pick a 4-bit or lower GGUF, let layers spill to CPU, and see the performance troubleshooting doc if it is slow. Lower-bit quantization saves memory but can reduce output quality.

One honest caveat

The 0.8B example model is fast and great for wiring things up, but it is much weaker than hosted frontier models. Swap in a larger GGUF once your plumbing works.

Use cases with copy-paste prompts

Each page has who it is for, the steps, and a prompt you can paste into your coding agent.

Use case: Serve a local model behind an OpenAI-compatible API

Use case: Run an open model locally in one command

Use case: Fit a bigger model on limited hardware

Get one email every Monday with the week's hottest AI repos and exactly how to use them: subscribe free at dedupedai.com

Top comments (0)