How to Serve LLM Completions in Production

Preparations

To start, you need to compile llama.cpp. You can follow their README for instructions.

The server is compiled alongside other targets by default.

Once you have the server running, we can continue. We will use PHP Resonance framework.

Troubleshooting

Obtaining Open-Source LLM

I recommend starting either with llama2 or Mistral. You need to download the pretrained weights and convert them into GGUF format before they can be used with llama.cpp.

Starting Server Without a GPU

llama.cpp supports CPU-only setups, so you don't have to do any additional configuration. It will be slow, but you will still have tokens generated.

Running With a Low VRAM Memory

You can try quantization if you don't have enough VRAM on your GPU to run a specific model. That lowers the response quality and the memory the model needs to use. Llama.cpp has a utility to quantize models:

$ ./quantize ./models/7B/ggml-model-f16.gguf ./models/7B/ggml-model-q4_0.gguf q4_0

10GB of VRAM is enough to run most quantized models.

Starting llama.cpp Server

While writing this tutorial, I had a server started with a command:

$ ./server 
    --model ~/llama-2-7b-chat/ggml-model-q4_0.gguf 
    --n-gpu-layers 200000 
    --ctx-size 2048 
    --parallel 8 
    --cont-batching
    --mlock 
    --port 8081

cont-batching parameter is essential, because it enables continuous batching, which is an optimization technique that allows parallel request.

Without it, even with multiple parallel slots, the server could answer to only one request at a time. cont-batching allows the server to respond to multiple completion requests in parallel.

Configuring Resonance

All you need to do is add a configuration section that specifies the llama.cpp server location:

[llamacpp]
host = 127.0.0.1
port = 8081

Testing

Resonance has built-in commands that connect to llama.cpp and issue requests.

You can send a sample prompt through llamacpp:completion:

$ php ./bin/resonance.php llamacpp:completion "How to write a 'Hello, world' in PHP?"
To write a "Hello, world" in PHP, you can use the following code:

<?php
  echo "Hello, world!";
?>

This will produce a simple "Hello, world!" message when executed.

Programmatic Use

In your class, you need to use Dependency Injection to inject LlamaCppClient:

<?php

namespace App;

use Distantmagic\Resonance\LlamaCppClient;
use Distantmagic\Resonance\LlamaCppCompletionRequest;

#[Singleton]
class LlamaCppGenerate 
{
    public function __construct(protected LlamaCppClient $llamaCppClient) 
    {
    }

    public function doSomething(): void
    {
        $request = new LlamaCppCompletionRequest('How to make a cat happy?');

        $completion = $this->llamaCppClient->generateCompletion($request);

        // each token is a chunk of text, usually few-several letters returned
        // from the model you are using
        foreach ($completion as $token) {
            swoole_error_log(SWOOLE_LOG_DEBUG, (string) $token);

            if ($token->isLast) {
                // ...do something else
            }
        }
    }
}

Summary

In this tutorial, we went through how to start llama.cpp server and connect to it with Resonance.

---

If you like Resonance, check us out on GitHub and give us a star! :)

https://github.com/distantmagic/resonance

Comments

[Mateusz Charytoniuk]

Thanks for sharing your concerns and tips. I am building a framework around it to mitigate those resources issues, so hopefully I will be able to mitigate most of them.

My understanding is you can also tweak it by adjusting slot sizes to not ever exceed the maximum context, so in general I’m more optimistic. :D I will remember your advice, however.

[Małgorzata Zagajewska]

Looks very promising, congrats on the integration!