DEV Community: Mateusz Charytoniuk

Paddler - open-source llama.cpp load balancer (self-host LLMs in production)

Mateusz Charytoniuk — Fri, 28 Jun 2024 12:16:20 +0000

Paddler is an open-source load balancer and reverse proxy designed to optimize servers running llama.cpp.

Typical strategies like round robin or least connections are not effective for llama.cpp servers, which need slots for continuous batching and concurrent requests.

Paddler overcomes this by maintaining a stateful load balancer that is aware of each server's available slots, ensuring efficient request distribution. Additionally, Paddler uses agents to monitor the health of individual llama.cpp instances, providing feedback to the load balancer for optimal performance. Paddler also supports the dynamic addition or removal of llama.cpp servers, enabling integration with autoscaling tools.

Feature Highlight

Aggregated Health Status

Paddler overrides /health endpoint of llama.cpp and reports the total number of available and processing slots.

Buffered Requests (Scaling from Zero Hosts)

Load balancer's buffered requests allow your infrastructure to scale from zero hosts by providing an additional metric (requests waiting to be handled).

It also gives your infrastructure some additional time to add additional hosts. For example, if your autoscaler is setting up an additional server, putting an incoming request on hold for 60 seconds might give it a chance to be handled even though there might be no available llama.cpp instances at the moment of issuing it.

Scaling from zero hosts is especially suitable for low-traffic projects because it allows you to cut costs on your infrastructure—you won't be paying your cloud provider anything if you are not using your service at the moment.

State Dashboard

Although Paddler integrates with the StatsD protocol, you can preview the cluster's state using a built-in dashboard.

Final Thoughts

The project is gaining some traction. Let me know if you also use it in prod, and I will highlight your project in the repo. Thank you all for giving me feedback on it so far. I always appreciate it. :)

Give us a star on GitHub

Migrating Cloudflare Page Rules to Redirect Rules (www to non-www)

Mateusz Charytoniuk — Wed, 29 May 2024 14:20:54 +0000

At the time of this writing, Cloudflare is deprecating Page Rules, and the Terraform migration guide has not yet been published.

This article will show how to convert a simple Page Rule (redirecting www to non-www hosts) from Page Rules to Rules.

UI

In the configuration rule, incoming requests should match a hostname starting with www.. The rewrite rule should be set to dynamic and contain an expression like concat("https://example.com", http.request.url.path) to combine the target hostname with a remaining path.

Remember to check the Preserve query string if needed. Status Code should be either 302 or 301. 301 is permanent and hard to remove from the cache, so if you are experimenting, try 302 first.

Terraform

Your API token needs to have Zone | Dynamic Redirect | Edit permission.

Before (Page Rules)



resource "cloudflare_page_rule" "www-to-non-www-redirect" {
  zone_id  = var.cloudflare_zone_id
  target   = "www.example.com/*"
  priority = 2

  actions {
    forwarding_url {
      status_code = 302
      url         = "https://example.com/$1"
    }
  }
}

After (Rules)



resource "cloudflare_ruleset" "redirect-www-to-non-www" {
  zone_id     = var.cloudflare_zone_id
  name        = "redirects"
  description = "Redirects ruleset"
  kind        = "zone"
  phase       = "http_request_dynamic_redirect"

  rules {
    action = "redirect"
    action_parameters {
      from_value {
        status_code = 302
        target_url {
          expression = "concat(\"https://example.com\", http.request.uri.path)"
        }
        preserve_query_string = true
      }
    }
    expression  = "(starts_with(http.host, \"www.\"))"
    description = "Redirect www to non-www"
    enabled     = true
  }
}

I made a framework, looking for early adopters! :)

Mateusz Charytoniuk — Mon, 27 May 2024 16:50:38 +0000

The framework: https://resonance.distantmagic.com

I started to work on it >6 months ago and I am nearing to the 1.0.0 release. I didn’t feel the need to add any more features so I think it’s getting pretty much complete.

It solves some specific issues with concurrency that other frameworks did not solve for me (at the same time it’s not made to compete with any other specific framework). I am issuing a lot of long running requests to LLMs that are resolved concurrently (you can issue tens of thousands of them from a cheap VPN), and it has a built in WebSocket server.

In short it is made for IO-bound applications, although it’s also really fast in itself (about 25x faster than Laravel Octane when serving a “hello world” view).

I think we are moving into the world where websites can/should integrate with ML models, AI and other microservices more and more, thus focus on the IO.

It also reimagines dependency injection - it does not allow cycles which makes it very easy on the GC - no more unexplained performance spikes.

I am not trying to sell anything and I have absolutely nothing from open sourcing it and writing all the articles around it. I am working on a different commercial product, I just wanted to open source something that can be useful to the community.

I’ve been working alone on the thing, it solves the issues for me. I would love to hear from you, to have someone try it out and share their opinion. That is my dream to find others who it might be useful for and to work on it together at some point. :D Hope that person is you. Feel free to reach out to me if you have any questions .

Structured: Extract Data from Unstructured Input with LLM

Mateusz Charytoniuk — Fri, 10 May 2024 15:27:07 +0000

The Structured project started as a Go conversion of Instructor,
but it is a more general-purpose library. It is designed to be extremely easy to use and set up.

It also features a language-agnostic HTTP server that you can set up in front of llama.cpp.

Same features, Go-like API. Model agnostic - maps data from arbitrary JSON schema to arbitrary Go struct (or just plain JSON).

It is focused on llama.cpp. Support for other vendor APIs (like OpenAI or Anthropic) might be added in the future.

HTTP API

Start a server and point it to your local llama.cpp instance:

./structured \
    --llamacpp-host 127.0.0.1 \
    --llamacpp-port 8081 \
    --port 8080

Structured server connects to llama.cpp to extract the data.

Now, you can issue requests. Include schema and data in your POST body.
The server will respond with JSON matching your schema:

POST http://127.0.0.1:8080/extract/entity
{
  "schema": {
    "type": "object",
    "properties": {
      "hello": {
        "type": "string"
      }
    },
    "required": ["hello"]
  },
  "data": "Say 'world'"
}

Response:
{
  "hello": "world"
}

Programmatic Usage (Optional)

API can change with time until all features are implemented.

Initializing the Mapper

Point it to your local llama.cpp instance:

import (
    "fmt"
    "net/http"
    "testing"

    "github.com/distantmagic/structured/structured"
    "github.com/distantmagic/paddler/llamacpp"
    "github.com/distantmagic/paddler/netcfg"
)

var entityExtractor *EntityExtractor = &structured.EntityExtractor{
    LlamaCppClient: &llamacpp.LlamaCppClient{
        HttpClient: http.DefaultClient,
        LlamaCppConfiguration: &llamacpp.LlamaCppConfiguration{
            HttpAddress: &netcfg.HttpAddressConfiguration{
                Host:   "127.0.0.1",
                Port:   8081,
                Scheme: "http",
            },
        },
    },
    MaxRetries: 3,
}

Extracting Structured Data from String

import "github.com/distantmagic/structured/structured"

responseChannel := make(chan structured.EntityExtractorResult)

go entityExtractor.ExtractFromString(
    responseChannel,
    map[string]any{
        "type": "object",
        "properties": map[string]any{
            "name": map[string]string{
                "type": "string",
            },
            "surname": map[string]string{
                "type": "string",
            },
            "age": map[string]string{
                "description": "Age in years.",
                "type":        "integer",
            },
        },
    },
    "I am John Doe - living for 40 years and I still like to play chess.",
)

for result := range responseChannel {
    if result.Error != nil {
        panic(result.Error)
    }

    // map[name:John, surname:Doe, age:40]
    fmt.Print(result.Result)
}

Mapping Extracted Result onto an Arbitrary Struct

Once you obtain the result:

import "github.com/distantmagic/structured/structured"

type myTestPerson struct {
    Name    string `json:"name"`
    Surname string `json:"surname"`
    Age     int    `json:"age"`
}

func DoUnmarshalsToStruct(result structured.EntityExtractorResult) {
    var person myTestPerson

    err := structured.UnmarshalToStruct(result, &person)

    if nil != err {
        panic(err)
    }

    person.Name // John
    person.Surname // Doe
}

Summary

That's it! :) You can use it as a language-agnostic server. Visit the repository and leave a star to show your support and get notified about new developments.

Semi-Scripted Conversational Applications

Mateusz Charytoniuk — Wed, 01 May 2024 20:29:18 +0000

What Are Conversational Applications?

Conversational applications are more than just chatbots. Large Language Models, when used as a user interface, enables users to create advanced applications that can replace most of the traditional interfaces.

They can use LLM (Large Language Models) or other means of processing human input as a front-end. The simplest example is a custom GPT that can use your API to interact with your application.

Is it a silver bullet to solve all the problems? Of course not. Currently most of AI integrations act as copilots or assistants. What if we try to inverse that dynamic where CUI (Conversational User Interface) is an add-on to the Graphical User Interface and make GUI components a complimentary feature instead?

How It Started

Ok, maybe Zork is not really a conversational application, but it is really symbolic for me.

All the interactions with the game are done through a series of specific commands. That kind of interface is rigorous in its usage because although it theoretically allows us to input any text, we still have to guess our options.

It is strict both on input and output data with no room for interpretation - like using some scripting programming language.

image source: lemon64

How Is It Going

Several observations:

We have an AI that can recreate the Zork experience in seconds without requiring almost any human labor.
AI understands any input that we can throw at it.
AI responds in the style and tone we asked, making the experience uniquely personalized.

Now, we can create a conversational application that can understand any input and provide a unique experience for each user. It is, however, very general in its use cases - which is both a blessing and a curse in disguise (see: infamous Chevrolet chatbot incident).

Where Is It Going?

What if we could mold a general-use AI to fit our specific applications and use cases while benefiting from its reasoning and language capabilities? To be flexible when accepting inputs and simultaneously avoiding hallucinations.

How about building the applications themselves without writing code?

Semi-Scripted Dialogues

Let us meet the AI halfway. We will create a semi-scripted dialogue to list predefined dialogue nodes and potential responses. The system will accept any input and try to match it with one of the predefined responses. This allows us flexibility when accepting inputs and be strict and predictable about our outputs.

Ingredients

Swoole
Extractors (to interpret the user's input and allow the AI to make further decisions based on it)
Dialogue Nodes (to create a conversational flow)
WebSockets (to actually serve the application)

What Are We Going to Build?

We will create a simple CRUD task management system. User will be able to name a new task, add a description, and list available tasks through a conversational interface.

We are going to use an LLM (through llama.cpp) to match user inputs with specific conversational nodes.

The application itself will alternate between handling potential responses from the user and executing side effects.

We will model the following conversation in a semi-scripted way so it will be capable of handling pretty much any user input:

System: 
    Message: Hello! How can I help you with your tasks?
    Potential Responses:
        - I want to create a task
        - I want to list all my tasks.

You can serve it in any way, either as a standalone application, a
Telegram bot or a web application. We will focus on the core of the conversational application and skip the delivery method for now.

Dialogue Controllers

We are all familiar with HTTP Controllers who mediate
between the user and the application. Why don't we reuse a similar concept and
create a dialogue controller?

It should keep the general state of the conversation and be able to react to user requests.

First, let us define the initial dialogue node and all the potential responses:

#[Singleton]
readonly class TaskConversationController extends DialogueController
{
    private DialogueNodeInterface $rootNode;

    public function __construct(
        private DoctrineEntityManagerRepository $doctrineEntityManagerRepository,
        private HelpfulAssistant $helpfulAssistantPersona,
        private LlamaCppClientInterface $llamaCppClient,
        LlamaCppExtractWhenInterface $llamaCppExtractWhen,
        private LlamaCppExtractSubject $llamaCppExtractSubject,
    ) {
        $this->rootNode = DialogueNode::withMessage('Hello! How can I help you with your tasks?');
        $this->rootNode->addPotentialResponse(new CatchAllResponse(
            followUp: $this->welcomeNode,
        ));

        $this->welcomeNode->addPotentialResponse(new LlamaCppExtractWhenResponse(
            llamaCppExtractWhen: $llamaCppExtractWhen,
            condition: 'User wants to create a new task',
            persona: $helpfulAssistantPersona,
            whenProvided: $this->whenUserWantsToCreateTask(...)
        ));

        $this->welcomeNode->addPotentialResponse(new LlamaCppExtractWhenResponse(
            llamaCppExtractWhen: $llamaCppExtractWhen,
            condition: 'User wants to list all their tasks',
            persona: $helpfulAssistantPersona,
            whenProvided: $this->whenUserWantsToListAllTasks(...)
        ));
    }

    public function getRootDialogueNode(): DialogueNodeInterface
    {
        return $this->rootNode;
    }

// (...)

Now, we need to handle all the predefined responses. Let's start with the one to create a new task. Let us try to extract the task name. If that is successful, let us move forward and add the task; in any other case, ask the user to specify further:

// (...)

    private function whenUserWantsToCreateTask(LlamaCppExtractWhenResult $response): Generator
    {
        if ($response->result->isNo()) {
            return DialogueResponseResolution::cannotRespond();
        }

        $result = $this->llamaCppExtractSubject->extract(
            input: $response->input,
            topic: 'task name',
        );

        //user provided a name of the task
        if ($result->content) {
            $followUp->addSideEffect(new CreateTask(
                taskName: $result->content,
            ));

            $followUp = new DialogueNode(new LlamaCppPromptMessageProducer(
                llamaCppClient: $this->llamaCppClient,
                request: new LlamaCppCompletionRequest(
                    llmChatHistory: new LlmChatHistory([
                        new LlmChatMessage(
                            actor: 'system',
                            message: 'Let user know that their task is added',
                        ),
                    ]),
                ),
            ));
        } else {
            $followUp = new DialogueNode(new LlamaCppPromptMessageProducer(
                llamaCppClient: $this->llamaCppClient,
                request: new LlamaCppCompletionRequest(
                    llmChatHistory: new LlmChatHistory([
                        new LlmChatMessage(
                            actor: 'system',
                            message: 'Ask user to specify the name of the task',
                        ),
                    ]),
                ),
            ));
        }

        // Go back to the primary dialogue node in any case
        $followUp->copyResponsesFrom($this->rootNode);

        return DialogueResponseResolution::canRespond($followUp);
    }

// (...)

When a user wants to list all tasks, just produce a response in a similar way you would produce an HTML view:

// (...)

    private function whenUserWantsToCreateTask(LlamaCppExtractWhenResult $response): Generator
    {
        if ($response->result->isNo()) {
            return DialogueResponseResolution::cannotRespond();
        }

        $responseMessage = "Your tasks:\n%s";

        $tasks = $this
            ->doctrineEntityManagerRepository
            ->withEntityManager(function (EntityManagerInterface $entityManager) {
                // ...obtain user tasks somehow
            })
        ;

        foreach ($tasks as $task) {
            $responseMessage .= "- {$task->name}\n";
        }

        $followUp = DialogueNode::withMessage($responseMessage);

        // Go back to the primary dialogue node in any case
        $followUp->copyResponsesFrom($this->rootNode);

        return DialogueResponseResolution::canRespond($followUp);
    }
}

Summary

It was a simple example of a conversational application; it's like writing a basic "Hello, world!". I hope it inspires you to create something incredible!

How to Write a PHP Extension with Zig?

Mateusz Charytoniuk — Wed, 10 Apr 2024 16:11:50 +0000

When writing code in a scripting language, sometimes you need that extra bit of performance (or maybe an async feature from Zig).

In that case, consider using it in your project.

Why Zig?

Zig is a statically typed, memory-safe, and thread-safe programming language.

It has a great interop with C, which makes it a perfect candidate for writing extensions for PHP. You don't have to worry about writing your bindings; you can call the functions from PHP internals straight up as they can also use C.

The Plan

We will still use a simple C facade to register Zig extension because Zig cannot translate all the C macros from the PHP source code into Zig code. You usually have to unwind them manually, so to avoid that, we will move as much of the extension initialization into C as possible.

Create a basic C extension wrapper
Fill in the gaps with Zig (statically linking functions)
Phpize the extension
Profit :)

We will create a basic "Hello from ZIG!" extension.

Basic C Extension Structure

All of those functions and entry points are mandatory.

In the header file, we will register myextension_functions as extern because they will be statically linked later.

#ifndef MYEXTENSION_H
#define MYEXTENSION_H
#define PHP_MY_PHP_EXTENSION_VERSION "1.0.0"

#include "php.h"

extern zend_function_entry myextension_functions[];

#endif

As you can see, many C macros are Zig's weak (arguably) point.

I am internally conflicted about whether I should call that a "weak" point because Zig, by design, decided not to support any macros and that type of metaprogramming - I think rightfully so. On the other hand, that approach also makes it harder for Zig to interop with macro-heavy C libraries.

#include "hello.h"

PHP_MINIT_FUNCTION(my_php_extension) {
    return SUCCESS;
}

PHP_MSHUTDOWN_FUNCTION(my_php_extension) {
    return SUCCESS;
}

PHP_RINIT_FUNCTION(my_php_extension) {
    return SUCCESS;
}

PHP_RSHUTDOWN_FUNCTION(my_php_extension) {
    return SUCCESS;
}

PHP_MINFO_FUNCTION(my_php_extension) {
    php_info_print_table_start();
    php_info_print_table_header(2, "My PHP Extension", "enabled");
    php_info_print_table_end();
}

zend_module_entry my_php_extension_module_entry = {
    STANDARD_MODULE_HEADER,
    "my_php_extension",
    myextension_functions,
    PHP_MINIT(my_php_extension),
    PHP_MSHUTDOWN(my_php_extension),
    PHP_RINIT(my_php_extension),
    PHP_RSHUTDOWN(my_php_extension),
    PHP_MINFO(my_php_extension),
    PHP_MY_PHP_EXTENSION_VERSION,
    STANDARD_MODULE_PROPERTIES
};

ZEND_GET_MODULE(my_php_extension)

You need a similar C structure, no matter what extension you intend to write.

Zig Part

That is the core of our extension. Here, we define our hello_world function that will be exported into PHP alongside its arguments and return type.

As you can see, we can use C functions directly with Zig code, which is great.

We can follow PHP internals documentation
here.

myextension_functions has to end with an empty terminator, hence the extra entry.

That is the basic extension that prints "Hello from ZIG!" but you can expand it with more complex features.

const std = @import("std");
const php = @cImport({
    @cInclude("php.h");
});

export fn hello_world(execute_data: ?*php.zend_execute_data, return_value: ?*php.zval) void {
    _ = execute_data;
    _ = return_value;
    _ = php.php_printf("Hello from ZIG!\n");
}

const arg_info = [_]php.zend_internal_arg_info{
    .{
        .name = null,
        .type = .{
            .type_mask = php.MAY_BE_NULL,
            .ptr = null,
        },
    },
};

export const myextension_functions = [_]php.zend_function_entry{
    php.zend_function_entry{
        .fname = "hello_world",
        .handler = hello_world,
        .arg_info = &arg_info,
        .num_args = 0,
        .flags = 0,
    },
    php.zend_function_entry{
        .fname = null,
        .handler = null,
        .arg_info = null,
        .num_args = 0,
        .flags = 0,
    },
};

Here, we configure the Zig build system to build our extension as a static library.

I took the included paths from the php-config --includes command. If you feel like it, you can invoke it in the build system, so you don't have to spell out include paths explicitly.

/usr/include/x86_64-linux-gnu is a common include path for Linux distributions. It uses my system's include paths, so you might have to adjust them.

const std = @import("std");

pub fn build(b: *std.Build) void {
    const optimize = b.standardOptimizeOption(.{});
    const target = b.standardTargetOptions(.{});

    const library = b.addStaticLibrary(.{
        .name = "my_php_extension",
        .root_source_file = .{
            .path = "hello.zig",
        },
        .target = target,
        .optimize = optimize,
    });

    library.addIncludePath(.{
        .path = "/usr/include/php/20220829"
    });
    library.addIncludePath(.{
        .path = "/usr/include/php/20220829/main"
    });
    library.addIncludePath(.{
        .path = "/usr/include/php/20220829/TSRM"
    });
    library.addIncludePath(.{
        .path = "/usr/include/php/20220829/Zend"
    });
    library.addIncludePath(.{
        .path = "/usr/include"
    });
    library.addIncludePath(.{
        .path = "/usr/include/x86_64-linux-gnu"
    });

    b.installArtifact(library);
}

With those two files in place, after running zig build, you should have zig-out/lib/libmy_php_extension.a file compiled, a static library that we will link later with the C part of the PHP extension.

PHP Part

To start with, we need Autoconf
configuration file for our extension.

PHP_ARG_ENABLE(my_php_extension, whether to enable My PHP Extension,
[ --enable-my-php_extension   Enable My PHP Extension support])

if test "$PHP_MY_PHP_EXTENSION" != "no"; then
    PHP_REQUIRE_CXX()
    PHP_ADD_LIBRARY(stdc++, 1, MY_PHP_EXTENSION_SHARED_LIBADD)
    PHP_SUBST(MY_PHP_EXTENSION_SHARED_LIBADD)
    PHP_ADD_LIBRARY_WITH_PATH(my_php_extension, ./zig-out/lib, MY_PHP_EXTENSION_SHARED_LIBADD)
    PHP_NEW_EXTENSION(my_php_extension, hello.c, $ext_shared)
fi

The important line is

PHP_ADD_LIBRARY_WITH_PATH(my_php_extension, ./zig-out/lib, MY_PHP_EXTENSION_SHARED_LIBADD)

Which tells the PHP build system to link the libmy_php_extension.a file from the Zig build.

Putting it All Together

Now that we have all the pieces, we can compile our extension.

Run the following commands:

zig build - to build the Zig static library
phpize - to prepare the PHP extension build
./configure - to configure the extension
make - to compile the extension

After that, you should have a modules/my_php_extension.so file that you can load into your PHP.

To test if it works, you can run:

php -d extension=./modules/my_php_extension.so -r 'echo hello_world();'

You should see Hello from ZIG!.

Congratulations! You have just written a PHP extension with Zig!

Conclusion

Zig is a great choice for writing extensions that require high performance or async features and can enrich your PHP experience.

How to Find a Software Career Path?

Mateusz Charytoniuk — Tue, 13 Feb 2024 23:04:47 +0000

Okay, you've finished your bachelor's degree, just left boot camp, or gained skills on a self-taught path. You have some knowledge of a variety of topics. What next? What should you focus on?

By definition, the college or any coding school doesn't teach the most modern or cutting-edge skills. They must wait until the dust settles, prepare their curriculums, prepare the cadres, and start teaching the topic. Sometimes, it takes years. So, while everyone is jumping on the AI wagon, you are learning how webdev was done several years ago.

Do not panic! It has always been like that. Fifteen years ago, when webdev was a hot topic, they still taught us Assembly language and C and almost nothing about webdev. So that's constant and normal.

What to do then? Where to find a path?

I think it's worth considering an alternative approach:

Do not hoard the knowledge for the sake of hoarding it. Invert that process instead. Think about how you can help others with software or any other CS-related product, then learn the skills necessary to make that a reality.
Try not to overthink "what niche to take" or "what career to take" - everything changes every several years anyway. Try to figure out what technology is helpful to others - it works out much better in the long run and helps prevent spending time on overhyped solutions. You can also stay true to yourself, which is good for your mental health.

In my experience, colleges teach the basics of a wide variety of topics (at least a bachelor's) so you can orient yourself, but what to focus is on you. If you concentrate on what is truly useful, or if you try to come up with something new that may help others, you will certainly have both a marketable skill and something you want to do.

Strive for Something Better than a Career

Mateusz Charytoniuk — Thu, 01 Feb 2024 20:37:54 +0000

First of all, yes, the job market is tough. Yes, it's understandable many people are looking for ways to improve their situation.

However, I am baffled by how many people focus on building up their portfolios, resumes, and profiles instead of setting a goal of doing meaningful work. Recently, lots of people, even when contributing to open-source projects, do not do that because they want to help someone but to make their profile look better (which already causes problems: https://dev.to/jitendrachoudhary/stop-contributing-to-open-source-13nb).

Yes, it's tough, but we must ask ourselves why we want to be software developers. This line of work aims to optimize processes and information flow- in short, to make other people's lives more convenient.

We have the almost unlimited liberty to create whatever we want, whether an app, website, or any platform, yet we choose to focus on finding ways to optimize how we want to drain money from an employer. Is that what this is about?

It's hard to find a community of people who want to create, build, experiment, and challenge the status quo. Almost everywhere, it's just the same story all over again: " I finished boot camp but can't get a job. What should I do?" "I can't get a frontend job. How can I become a fullstack?" " Should I get a degree?" " Should I accept the job?" " Where should I apply?" etc.

Does anyone want to be creative and think about others? What happened? Yes, I understand finding a job and making a living is hard. But even at the point in life where it's necessary to do any job to make a living, I think it's still essential to preserve the internal drive and ambition at all costs - to create something good and beautiful instead of constantly chasing ways to impress an employer without any meaning behind it.

It feels like we are just wasting potential and making it harder for ourselves. There are only so many resources to take from the market. We need to create to make it grow again. Stop listening to influencers blindly - start thinking for yourself.

How to Setup Postfix for Outgoing Emails

Mateusz Charytoniuk — Thu, 01 Feb 2024 06:06:45 +0000

Assumptions

You are using 3rd party email service provider (like Gmail, Outlook, ProtonMail, iCloud, etc), you have a server capable of running Postfix and you want to have the capability to send transactional emails while preserving your 3rd party service for incoming emails.

No Guarantees

It takes a lot of work to set up a deliverable email server. For me, the setup I described here works perfectly - my emails weren't blacklisted or automatically classified as spam. I was able to deliver them to all major email service providers successfully.

Remember that your results may vary; many factors contribute to deliverability, and I'm not responsible if something goes wrong. You have been warned.

The Goal

Our final goal is to be able to send transactional emails from our server and still be able to both receive and send them through 3rd party service provider. Let me explain.

Postfix is an open-source email transfer agent. It can act both as an SMTP server and client, but we will only use its server capabilities.
When it comes to the client part, you can use any email service provider.

Setting up an email server is unreasonably burdensome because you must handle the incoming spam and storage. It's much easier to rely on an external email provider. On the other hand, free email providers are unsuitable for sending transactional emails.

What if we could combine the best parts of both worlds and be able to send transactional emails through our server and receive them through a 3rd party client?

Postfix

Installation

There are multiple ways to install Postfix, depending on your operating system so that I won't be covering that here. I'll assume you have it running.

It's pretty simple; for example, on a Debian-based system, it's usually just a matter of:

$ sudo apt install Postfix

Configuration

To deliver your emails to most inboxes, you need to enable TLS email encryption in your Postfix server. To do so, you need to add the lines:

smtpd_tls_security_level=encrypt
smtpd_tls_loglevel = 1

smtp_tls_security_level=encrypt
smtp_tls_loglevel = 1

*_loglevel setting is optional to add; it just helps with debugging potential issues.

Also, you need to make sure that myhostname variable is set accurately to either the current server IP or its reverse DNS address. That is crucial for setting up SPF.

DNS Setup

DKIM Record

To make your emails deliverable, you need to create a DKIM record for your server (see Mail to learn how to generate the encryption keys):

Type:  TXT
Name:  YOURDKIMSELECTOR._domainkey
Value: v=DKIM1; k=rsa; p=YOURPUBLICKEY

YOURDKIMSELECTOR will be used when sending emails to tell the recipient to validate them against the server we are currently setting up.

YOURPUBLICKEY is the public key with newlines and the envelope removed (without the -----BEGIN PUBLIC KEY----- and -----END PUBLIC KEY----- parts - or similar). Your Resonance application will have to use the same key pair you are using here to encrypt emails.

DMARC Record

You most likely already have DMARC record configured, as most 3rd party email providers require it. It should point to an email that handles potential deliverability issues. For example:

Type:  TXT
Name:  _dmarc
Value: v=DMARC1; p=none; rua=mailto:postmaster@YOURDOMAIN

Replace YOURDOMAIN with your actual domain.

SPF Record

SPF records tell what servers can send emails from your domain.

You probably already have a configured SPF record from your email provider. For example, if you are using Protonmail you might have a record like this:

Type:  TXT
Name:  @
Value: v=spf1 include:_spf.protonmail.ch mx ~all

You need to change it to incorporate your server's IP address (you can also delegate that to another record by using include:). For example:

Type:  TXT
Name:  @
Value: v=spf1 include:_spf.protonmail.ch ip4:YOURSERVERIP mx ~all

Replace YOURSERVERIP with your actual server IP address.

By setting up the SPF record like above, you will be able to use both your email provider and your server to send emails.

Resonance Setup

Once you have both Postfix and DNS records set up, you can configure Resonance to handle outgoing emails through Postfix.

Configuration

Add Postfix section to your mailer (you can name it any way you want; it doesn't have to be postfix):

; ...

[mailer]
postfix[transport_dsn] = native://default
postfix[dkim_domain_name] = YOURDOMAIN
postfix[dkim_selector] = YOURDKIMSELECTOR
postfix[dkim_signing_key_passphrase] = YOURPASSPHRASE
postfix[dkim_signing_key_private] = dkim/private.key
postfix[dkim_signing_key_public] = dkim/public.key

; ...

Replace YOURDOMAIN, YOURDKIMSELECTOR, YOURPASSPHRASE with appropriate values.

Sending Emails

You can refer to Mail documentation to learn how to send emails with Resonance.

You should be able to immediately send a deliverable email by using the command line:

$ php bin/resonance.php mail:send \
    --from "me@YOURDOMAIN" \
    --to "you@YOURDOMAIN" \
    --subject "Hello!" \
    --transport "postfix" \
    "How is it going?"

Or programmatically:

/**
 * @var Distantmagic\Resonance\MailerRepository $mailerRepository
 * @var Symfony\Component\Mime\Email $email
 */
$mailerRepository->mailer->get('postfix')->enqueue($email);

Remember to warm up your IP address and domain before lots of transactional emails.

Summary

By now, you should have a transactional email server with its own dedicated IP address and still retain the possibility of receiving incoming emails in the same inbox you have used and loved. :) Congratulations!

I made a static site generator in PHP (and it's fast)

Mateusz Charytoniuk — Sat, 27 Jan 2024 01:15:19 +0000

I published an async Resonance PHP framework. I decided also to implement a static site generator to create elaborate documentation or static pages:

Why?

I aimed to solve a few issues:

I wanted to use something other than Docusaurus because its build times are generally slow: https://github.com/facebook/docusaurus/discussions/3132. I made performance my priority
I wanted to use the same tech stack as the core framework, and no major Static Site Generators in PHP were ready to use.
I wanted to use something other than React. Instead, I opted for Stimulus + Turbo, which is a much lighter approach.

Design Choices

Front-Matter

I borrowed the idea of Front-Matter from Jekyll, but I made some adjustments,

Jekyll's front-matter is pretty loose - you can input any properties you want in Resonance. I used a strict validation for the sake of consistency:



---
collections: 
    - documents
draft: true
layout: dm:document
parent: docs/features/index
title: "Machine Learning"
description: ">"
    Learn how to incorporate Machine Learning features into your application.
---

Front matter is used to render metadata like <meta name="description"> and various hints in the documentation.

Consistency Above All

It's possible to link documents in a wiki-like fashion by using {{ and }} and pointing to the linked file. For example, if you have a file in docs/features/test.md,, you can create an <a> link to that file with {{docs/features/test}}. All the links are checked while rebuilding the documentation to ensure each is valid.

Ordering of Documents

Instead of some global index of documents, I ordered them by references. You can use fields like next or parent in the front matter to order them relatively to each other:



---
collections: 
    - documents
layout: dm:document
next: docs/features/asset-bundling-esbuild/preloading-assets
parent: docs/features/asset-bundling-esbuild/index
title: Cache Busting
description: >
    Prevent unwanted caching with esbuild features.
---

Thanks to that, maintaining the documentation is more straightforward than maintaining another global list of pages with their ordering rules.

Custom Layouts

You can create your custom layouts and reference them in the front matter, making it easy to have shared layouts across multiple pages.

Collections

You can add pages to collections to put them into the site's primary navigation or side menus:



---

collections: 

    - documents

    - name: primary_navigation

      next: tutorials/index

description: docs

layout: dm:document

title: Documentation

---

Summary

I hope this generator will be helpful to you. Feel free to reach out if you have any questions, and leave me a star if you would like it. Good luck!

You can find documentation and installation instructions here: https://resonance.distantmagic.com/docs/features/generating-static-sites/

Semantic Search with KNN and LLM - Explained with a Poem

Mateusz Charytoniuk — Thu, 25 Jan 2024 22:53:50 +0000

Creating a simple semantic search engine is possible by creating embeddings with an LLM model and then searching through them with the KNN algorithm.

The entire process is surprisingly simple.

Running LLM Locally

You can use tools like Ollama or llama.cpp.

Let's say we have a Mistral LLM running on Ollama.

Preparing Your Data

Input Data

You need to have some data to search through it. Let's assume you have a PDF document or something similar. We will use a funny poem as an example:

Daddy Fell into the Pond
By Alfred Noyes

Everyone grumbled. The sky was grey.
We had nothing to do and nothing to say.
We were nearing the end of a dismal day,
And then there seemed to be nothing beyond,
Then Daddy fell into the pond!

And everyone's face grew merry and bright,
And Timothy danced for sheer delight.
"Give me the camera, quick, oh quick!
He's crawling out of the duckweed!" Click!

Then the gardener suddenly slapped his knee,
And doubled up, shaking silently,
And the ducks all quacked as if they were daft,
And it sounded as if the old drake laughed.
Oh, there wasn't a thing that didn't respond
When Daddy Fell into the pond!

First, you need to split it into chunks (that's what that is called) - a piece of text of arbitrary size (you need to figure out the best length for a chunk - it's a part of your research). It can be an entire page or just a paragraph.

Generating Embeddings

We will use Ollama's embeddings endpoint:

POST /api/embeddings

Let's use one stanza per embedding to have three of them. That is very simplified just for the sake of the example; usually, you will have thousands or millions of embeddings to deal with (or more).

After we send our text's chunk to Ollama, it will forward that to an underlying LLM, and that LLM will generate an embedding for us. You should have three vectors:

Everyone grumbled. The sky (...) -> [number, number, ...]
And everyone's face grew merry (...) -> [number, number, ...]
Then the gardener suddenly (...) -> [number, number, ...]

What is an Embedding?

You can think of embedding as a set of coordinates. If we want to place something in a 3D world, we use the [x,y,z] vector to point to that item.

The spatial system for words is much more complicated; embeddings can have 4096 or more such coordinates. The closer the embeddings are to each other, the more semantically similar they are.

Semantic Search

So, how do we search for similar phrases? Simply put, the closer the embeddings are to each other (in a spatial sense, by distance), the more similar they are.

After the user enters an input prompt, convert it to another embedding; for example, if the user asks, Who fell into the pond? - that will also be converted into a vector. Then, you can use KNN (K Nearest Neighbor algorithm) to find K closest embeddings to that vector. Those are the most likely related embeddings to your query.

Notice that you don't need extensive use of an LLM to perform such a search.

Summary

Where to go from there? Try setting up such semantic search with your favorite LLM and framework, and experiment with different chunk sizes to see which produces the best results. Good luck!

How to Create LLM Chat from Open-Source Components?

Mateusz Charytoniuk — Sat, 20 Jan 2024 04:04:06 +0000

Preparations

Before starting this tutorial, it might be useful for you to familiarize yourself with the other tutorials and documentation pages first:

https://dev.to/mcharytoniuk/how-to-serve-llm-completions-in-production-33c2

In this tutorial, we will also use Stimulus for the front-end code.

Once you have both Resonance and llama.cpp runing, we can continue.

Backend

Security

First, we must create a security gate and decide which users can use our chat. Let's say all authenticated users can do that:

<?php

namespace App\SiteActionGate;

use App\Role;
use Distantmagic\Resonance\Attribute\DecidesSiteAction;
use Distantmagic\Resonance\Attribute\Singleton;
use Distantmagic\Resonance\AuthenticatedUser;
use Distantmagic\Resonance\SingletonCollection;
use Distantmagic\Resonance\SiteAction;
use Distantmagic\Resonance\SiteActionGate;

#[DecidesSiteAction(SiteAction::StartWebSocketRPCConnection)]
#[Singleton(collection: SingletonCollection::SiteActionGate)]
final readonly class StartWebSocketRPCConnectionGate extends SiteActionGate
{
    public function can(?AuthenticatedUser $authenticatedUser): bool
    {
        return true === $authenticatedUser?->user->getRole()->isAtLeast(Role::User);
    }
}

HTTP Responder

First, we need to create a route in our application that will render the HTML chat.

We are using a few attributes:

Can - this is an Authorization attribute it checks if user can connect with WebSocket. If they can't - it will return 403 error page instead. That is optional, but without Can attribute, all users will be able to visit the page, but they won't be able to establish a WebSocket connection anyway.
RespondsToHttp - registers route
Singleton - adds your HttpResponder to Dependency Injection container. We added wantsFeature: Feature::WebSocket parameter to enable WebSocket server.

<?php

namespace App\HttpResponder;

use App\HttpRouteSymbol;
use Distantmagic\Resonance\Attribute\Can;
use Distantmagic\Resonance\Attribute\RespondsToHttp;
use Distantmagic\Resonance\Attribute\Singleton;
use Distantmagic\Resonance\Feature;
use Distantmagic\Resonance\HttpInterceptableInterface;
use Distantmagic\Resonance\HttpResponder;
use Distantmagic\Resonance\RequestMethod;
use Distantmagic\Resonance\SingletonCollection;
use Distantmagic\Resonance\SiteAction;
use Distantmagic\Resonance\TwigTemplate;
use Swoole\Http\Request;
use Swoole\Http\Response;

#[Can(SiteAction::StartWebSocketRPCConnection)]
#[RespondsToHttp(
    method: RequestMethod::GET,
    pattern: '/chat',
    routeSymbol: HttpRouteSymbol::LlmChat,
)]
#[Singleton(
    collection: SingletonCollection::HttpResponder,
    wantsFeature: Feature::WebSocket,
)]
final readonly class LlmChat extends HttpResponder
{
    public function respond(Request $request, Response $response): HttpInterceptableInterface
    {
        return new TwigTemplate('turbo/llmchat/index.twig');
    }
}

WebSocket Message Validation

First, we need to register our WebSocket RPC methods. Default WebSocket implementation follows a simple RPC protocol. All messages need to be a tuple:

[method, payload, null|request id]

Method, the first field, is validated against RPCMethodInterface, through RPCMethodValidator (you need to provide both):

<?php

namespace App;

use Distantmagic\Resonance\EnumValuesTrait;
use Distantmagic\Resonance\RPCMethodInterface;

enum RPCMethod: string implements RPCMethodInterface
{
    use EnumValuesTrait;

    case LlmChatPrompt = 'llm_chat_prompt';
    case LlmToken = 'llm_token';

    public function getValue(): string
    {
        return $this->value;
    }
}

<?php

namespace App;

use Distantmagic\Resonance\Attribute\Singleton;
use Distantmagic\Resonance\Feature;
use Distantmagic\Resonance\RPCMethodInterface;
use Distantmagic\Resonance\RPCMethodValidatorInterface;

#[Singleton(
    provides: RPCMethodValidatorInterface::class,
    wantsFeature: Feature::WebSocket,
)]
readonly class RPCMethodValidator implements RPCMethodValidatorInterface
{
    public function cases(): array
    {
        return RPCMethod::cases();
    }

    public function castToRPCMethod(string $methodName): RPCMethodInterface
    {
        return RPCMethod::from($methodName);
    }

    public function values(): array
    {
        return RPCMethod::values();
    }
}

The above means your application will support llm_chat_prompt and llm_chat_token messages.

WebSocket Message Responder

Now, we need to register a WebSocket responder that will run every time someone sends 'llama_chat_prompt' message to the server.

It will then use LlamaCppClient to fetch tokens from llama.cpp server and forward themto the WebSocket connection:

<?php

namespace App\WebSocketRPCResponder;

use App\RPCMethod;
use Distantmagic\Resonance\Attribute\RespondsToWebSocketRPC;
use Distantmagic\Resonance\Attribute\Singleton;
use Distantmagic\Resonance\Feature;
use Distantmagic\Resonance\LlamaCppClient;
use Distantmagic\Resonance\LlamaCppCompletionCommand;
use Distantmagic\Resonance\LlamaCppCompletionRequest;
use Distantmagic\Resonance\RPCNotification;
use Distantmagic\Resonance\RPCRequest;
use Distantmagic\Resonance\RPCResponse;
use Distantmagic\Resonance\SingletonCollection;
use Distantmagic\Resonance\WebSocketAuthResolution;
use Distantmagic\Resonance\WebSocketConnection;
use Distantmagic\Resonance\WebSocketRPCResponder;

#[RespondsToWebSocketRPC(RPCMethod::LlmChatPrompt)]
#[Singleton(
    collection: SingletonCollection::WebSocketRPCResponder,
    wantsFeature: Feature::WebSocket,
)]
final readonly class LlmChatPromptResponder extends WebSocketRPCResponder
{
    public function __construct(
        private LlamaCppClient $llamaCppClient,
    ) {}

    protected function onNotification(
        WebSocketAuthResolution $webSocketAuthResolution,
        WebSocketConnection $webSocketConnection,
        RPCNotification $rpcNotification,
    ): void {
        $request = new LlamaCppCompletionRequest($rpcNotification->payload->prompt);

        $completion = $this->llamaCppClient->generateCompletion($request);

        foreach ($completion as $token) {
            if ($webSocketConnection->status->isOpen()) {
                $webSocketConnection->push(new RPCNotification(
                    RPCMethod::LlmToken,
                    $token->content,
                ));
            } else {
                $completion->send(LlamaCppCompletionCommand::Stop);
            }
        }
    }
}

Frontend

Stimulus controller starts WebSocket connection and sends user prompts to the server. Not that WebSocket connection uses both CSRF Protection and dm-rpc protocol:

import { Controller } from "@hotwired/stimulus";

import { stimulus } from "../stimulus";

@stimulus("llmchat")
export class controller_llmchat extends Controller<HTMLElement> {
  public static targets = ["userInputField", "userInputForm", "chatLog"];
  public static values = {
    csrfToken: String,
  };

  private declare readonly chatLogTarget: HTMLElement;
  private declare readonly csrfTokenValue: string;
  private declare readonly userInputFieldTarget: HTMLTextAreaElement;

  private webSocket: null|WebSocket = null;

  public connect(): void {
    const servertUrl = new URL(__WEBSOCKET_URL);

    servertUrl.searchParams.append("csrf", this.csrfTokenValue);

    const webSocket = new WebSocket(servertUrl, ["dm-rpc"]);

    webSocket.addEventListener("close", () => {
      this.webSocket = null;
    });

    webSocket.addEventListener("open", () => {
      this.webSocket = webSocket;
    });

    webSocket.addEventListener("message", (evt: MessageEvent) => {
      if ("string" !== typeof evt.data) {
        return;
      }

      const parsed: unknown = JSON.parse(evt.data);

      if (Array.isArray(parsed) && "string" === typeof parsed[1]) {
        this.chatLogTarget.append(parsed[1]);
      }
    });
  }

  public disconnect(): void {
    this.webSocket?.close();
  }

  public onFormSubmit(evt: Event): void {
    evt.preventDefault();

    this.chatLogTarget.innerHTML = '';

    this.webSocket?.send(JSON.stringify([
      "llm_chat_prompt",
      {
        prompt: this.userInputFieldTarget.value,
      },
      null
    ]));

    this.userInputFieldTarget.value = '';
  }
}

We will use esbuild to bundle front-end TypeScript code.

$ ./node_modules/.bin/esbuild \
    --bundle \
    --define:global=globalThis \
    --entry-names="./[name]_$(BUILD_ID)" \
    --format=esm \
    --log-limit=0 \
    --metafile=esbuild-meta-app.json \
    --minify \
    --outdir=./$(BUILD_TARGET_DIRECTORY) \
    --platform=browser \
    --sourcemap \
    --target=es2022,safari16 \
    --tree-shaking=true \
    --tsconfig=tsconfig.json \
    resources/ts/controller_llmchat.ts \
;

Finally, the HTML:

<script 
    defer 
    type="module" 
    src="{{ esbuild(request, 'controller_llmchat.ts') }}"
></script>

<div
    data-controller="llmchat"
    data-llmchat-csrf-token-value="{{ csrf_token(request, response) }}"
>
    <div data-llmchat-target="chatLog"></div>
    <form data-action="submit->llmchat#onFormSubmit">
        <input
            autofocus
            data-llmchat-target="userInputField"
            type="text"
        ></input>
        <button type="submit">Send</button>
    </form>
</div>

Summary

Now, you can serve LLM chat in your application. Enjoy!

If you like Resonance, check us out on GitHub and give us a star! :)

https://github.com/distantmagic/resonance