DEV Community: Albert Colom

PHP Symfony Microservice with gRPC: A Practical Guide

Albert Colom — Wed, 24 Dec 2025 16:22:26 +0000

A Step-by-Step Guide to Building High-Performance Hexagonal Microservices in PHP with Symfony and gRPC.

Why use gRPC?

gRPC is a modern, high-performance communication framework designed to enable microservices to communicate with each other more efficiently than traditional methods. Unlike standard REST APIs, which send data in the form of bulky, human-readable text (JSON), gRPC converts data into a compact binary format using Protocol Buffers. This results in much smaller messages and significantly faster processing, which is vital for high-traffic systems.

gRPC runs on HTTP/2 , which enables multiplexing — sending multiple requests over a single connection simultaneously — and supports real-time, bidirectional streaming. This means that your services can maintain a constant, two-way flow of information, rather than just “requesting and waiting”.

You can find more information on the official website: gRPC

Previous requirements

Docker with Docker Compose or any other container-compatible alternative
Composer

Install Symfony + Docker + Roadrunner

Let’s see how to install a basic version of Symfony running PHP CLI and Roadrunner via Docker.

Create a Symfony project

In this tutorial create a Symfony project from scratch, feel free to use an existing Symfony project.

composer create-project symfony/skeleton symfony-grpc

Install roadrunner-bundle

To facilitate the integration of Roadrunner with Symfony, we will use the baldinof/roadrunner-bundle vendor. You can find more information on the project’s GitHub page.

composer require baldinof/roadrunner-bundle

NOTE: Usually if you used the Recipe, the roadrunner-bundle create an example rr.yaml and rr.dev.yaml on your root project. If not create you can copy manually from the vendor:

cp vendor/baldinof/roadrunner-bundle/.rr.dev.yaml .
cp vendor/baldinof/roadrunner-bundle/.rr.yaml .

Create a simple Docker Image with PHP and Roadrunner

We created a basic Docker image containing PHP CLI and the RoadRunner binary. Including the RoadRunner binary in our PHP image means that we are no longer dependent on external services such as Apache or Nginx.

#resources/docker/php-roadrunner/Dockerfile

FROM ghcr.io/roadrunner-server/roadrunner:2025.1.6 AS roadrunner

FROM php:8.4-cli-alpine
COPY --from=roadrunner /usr/bin/rr /usr/local/bin/rr
EXPOSE 8080 9000

WORKDIR /var/www/app

Create Docker compose

To make the project easier to execute, let’s create a simple Docker Compose file.

#docker-compose.yml

services:
  php-rr:
    build:
      context: resources/docker/php-roadrunner/
    container_name: php-symfony-grpc
    ports:
      - "8080:8080"
      - "6001:6001"
      - "9001:9001"
    volumes:
      - .:/var/www/app
    command: rr serve -c .rr.yaml

Let’s go! you can try to run your first Symfony App via Roadrunner.

docker compose up -d

Now you can open url in your browser: http://127.0.0.1:8080/ and show a default Symfony welcome page:

Congrats! you run your first Symfony Application via Roadrunner.

Implement gRPC Server con RoadRunner

We have RoadRunner running with an HTTP server, so now is the time to learn how to implement our gRPC server with Symfony.

Install gRPC Radrunner dependences

First, we installed the Roadrunner gRPC server and the Protocol Buffers library.

composer require google/protobuf spiral/roadrunner-grpc

Create gRPC Worker

We create the gRPC worker using the Symfony kernel and container.

<?php

# public/grpc-worker.php

use App\Kernel;
use Spiral\RoadRunner\GRPC\Server;
use Spiral\RoadRunner\Worker;

require_once dirname( __DIR__ ) . '/vendor/autoload.php';

$kernel = new Kernel($_SERVER['APP_ENV'] ?? 'prod', (bool) ($_SERVER['APP_DEBUG'] ?? false));
$kernel->boot();
$container = $kernel->getContainer();

$worker = Worker::create();
$server = new Server();

try {
    $server->serve($worker);
} catch (Throwable $e) {
    error_log($e->getMessage());
    exit(1);
}

exit(0);

Update RoadRunner server config

The next step is to update the RoadRunner configuration filerr.yaml with the gRPC configuration.

# rr.yaml

version: "3"

server:
    command: "php public/grpc-worker.php"
    relay: pipes

grpc:
    listen: "tcp://0.0.0.0:9001"

logs:
  mode: production
  channels:
    http:
      level: debug
    server:
      level: info
      mode: raw
    metrics:
      level: error

Well done! We now have our first gRPC server running with the Symfony container.

docker compose up -d

Generating gRPC codes

gRPC requires code generation because a .proto file only defines a contract, it is not executable code. The generated code creates typed classes and methods, enabling your language to safely send and receive Protobuf messages.

Create a Docker Image to generate gRPC files

The next step is to create a Dockerfile with an entrypoint that will generate the necessary files using protoc-gen-grpc .

#resources/docker/php-roadrunner/Dockerfile

FROM alpine:3.23

RUN apk add --no-cache protobuf wget && \
    ARCH=$(uname -m) && \
    if ["$ARCH" = "aarch64"]; then ARCH="arm64"; else ARCH="amd64"; fi && \
    cd /tmp && \
    wget https://github.com/roadrunner-server/roadrunner/releases/download/v2025.1.6/protoc-gen-php-grpc-2025.1.6-linux-${ARCH}.tar.gz && \
    tar -xzf protoc-gen-php-grpc-2025.1.6-linux-${ARCH}.tar.gz && \
    cd protoc-gen-php-grpc-2025.1.6-linux-${ARCH} && \
    mv protoc-gen-php-grpc /usr/local/bin/ && \
    chmod +x /usr/local/bin/protoc-gen-php-grpc && \
    rm -rf /tmp/*

COPY entrypoint.sh /usr/local/bin/
RUN chmod +x /usr/local/bin/entrypoint.sh

WORKDIR /app
ENTRYPOINT ["/usr/local/bin/entrypoint.sh"]

NOTE : You can specify different paths for the prototype files and the location where the generated files should be stored.

default : PROTO_PATH=/app/proto and OUTPUT_PATH=/app/generated

#!/bin/sh

#resources/docker/php-roadrunner/entrypoint.sh

set -e

PROTO_PATH="${PROTO_PATH:-/app/proto}"
OUTPUT_PATH="${OUTPUT_PATH:-/app/generated}"

echo "[INFO] Generating gRPC files | Proto path: $PROTO_PATH | Output: $OUTPUT_PATH"

if [! -d "$PROTO_PATH"]; then
    echo "[ERROR] Proto path directory not found: $PROTO_PATH"
    exit 1
fi

PROTO_FILES=$(find "$PROTO_PATH" -name "*.proto")

if [-z "$PROTO_FILES"]; then
    echo "[ERROR] No .proto files found in $PROTO_PATH"
    exit 1
fi

echo -n "[INFO] Proto files: " && echo "$PROTO_FILES" | xargs -n 1 basename | paste -sd ',' - | sed 's/,/, /g'

mkdir -p "$OUTPUT_PATH"

protoc --proto_path="$PROTO_PATH" \
    --php_out="$OUTPUT_PATH" \
    --php-grpc_out="$OUTPUT_PATH" \
    $PROTO_FILES

echo "[INFO] gRPC files generated successfully"

Update Docker Compose

Add the new Docker to your docker file proto-gen .

services:
    proto-gen:
        build:
            context: resources/docker/protobuf/
        volumes:
            - ./proto:/app/proto
            - ./generated:/app/generated

    php-rr:
        build:
            context: resources/docker/php-roadrunner/
        container_name: php-symfony-grpc
        ports:
            - "8080:8080"
            - "6001:6001"
            - "9001:9001"
        volumes:
            - .:/var/www/app
        command: rr serve -c .rr.yaml

Create the proto file

A .proto file is a schema that defines gRPC messages and services.

It is language-neutral and is used to generate client and server code in different programming languages Protocol Buffers (Protobuf)

// proto/user.proto

syntax = "proto3";

package user.v1;

option php_namespace = "GRPC\\User\\V1";
option php_metadata_namespace = "GRPC\\GPBMetadata";

service UserService {
    rpc GetUser (GetUserRequest) returns (GetUserResponse);
}

message GetUserRequest {
    string id = 1;
}

message GetUserResponse {
    string id = 1;
    string email = 2;
}

Generate gRPC files

Now we have everything we need to generate our files with this simple command.

 docker compose run --rm proto-gen

This will generate the files required by our application.

Add the generated files to the autoload.

Add the path of the generated files to the autoload section of the composer.json

...

"autoload": {
        "psr-4": {
            "App\\": "src/",
            "GRPC\\": "generated/GRPC"
        }
    },

...

Sample User code

This section provides a concise code example of a PHP Symfony microservice that has been structured with hexagonal architecture and domain-driven design (DDD). The implementation of gRPC is as an infrastructure service that is exclusively responsible for external communication. Symfony is responsible for application wiring, and the domain remains fully isolated from technical concerns.

Domain

The User class represents the domain entity and encapsulates only the business logic for validating IDs and emails, independently of Symfony or gRPC.

<?php

# src/User/Domain/User.php

declare(strict_types=1);

namespace App\User\Domain;

use InvalidArgumentException;

final readonly class User
{
    public function __construct(
        public string $id,
        public string $email
    )
    {
        if (!filter_var($email, FILTER_VALIDATE_EMAIL)) {
            throw new InvalidArgumentException('Invalid email');
        }
    }

    public function equals(User $other): bool
    {
        return $this->id === $other->id && $this->email === $other->email;
    }
}

This UserRepositoryInterface defines the contract for the domain’s user repository.

<?php

# src/User/Domain/UserRepositoryInterface.php

namespace App\User\Domain;

interface UserRepositoryInterface
{
    public function getById(string $id): ?User;
}

Application

The GetUserQuery class is an application-layer query object that carries the user ID and encapsulates the request to retrieve a user. It does not contain any business logic.

<?php

# src/User/Application/Query/GetUserQuery.php

declare(strict_types=1);

namespace App\User\Application\Query;

final readonly class GetUserQuery
{
    public function __construct(public string $id)
    {
    }
}

The GetUserQueryHandler class represents the application-layer service that handles GetUserQuery.

<?php

# src/User/Application/Query/GetUserQueryHandler.php

declare(strict_types=1);

namespace App\User\Application\Query;

use App\User\Domain\User;
use App\User\Domain\UserRepositoryInterface;

final readonly class GetUserQueryHandler
{
    public function __construct(private UserRepositoryInterface $repository)
    {
    }

    public function __invoke(GetUserQuery $query): ?User
    {
        return $this->repository->getById($query->id);
    }
}

Infrastructure

The InMemoryUserRepository class is an implementation of the UserRepositoryInterface in the infrastructure layer.

<?php

# src/User/Infrastructure/Persistence/InMemoryUserRepository.php

declare(strict_types=1);

namespace App\User\Infrastructure\Persistence;

use App\User\Domain\User;
use App\User\Domain\UserRepositoryInterface;

final class InMemoryUserRepository implements UserRepositoryInterface
{
    /** @var array<string, User> */
    private array $users;

    public function __construct()
    {
        $this->users = [
            'f5fb962d-60f6-4096-8bd6-f717249d46fa' => new User('f5fb962d-60f6-4096-8bd6-f717249d46fa', 'foo1@example.com'),
            'f7eb16c3-5b92-480b-8d58-e6a3d3b293a7' => new User('f7eb16c3-5b92-480b-8d58-e6a3d3b293a7', 'foo2@example.com'),
            '923ecb26-a08b-41e2-861e-8f9449addc93' => new User('923ecb26-a08b-41e2-861e-8f9449addc93', 'foo3@example.com'),
            '20aa442b-5031-4db6-be04-df15ca9881d3' => new User('20aa442b-5031-4db6-be04-df15ca9881d3', 'foo4@example.com'),
            '464d60fb-3823-4e9d-8413-4e0b1c68c77d' => new User('464d60fb-3823-4e9d-8413-4e0b1c68c77d', 'foo5@example.com'),
        ];
    }

    public function getById(string $id): ?User
    {
        return $this->users[$id] ?? null;
    }
}

The UserService class is the gRPC adapter in the infrastructure layer. It receives gRPC requests, calls the application layer handler to fetch the User entity, and maps the result to a gRPC response, keeping the domain logic completely decoupled from transport concerns.

<?php

declare(strict_types=1);

# src/User/Infrastructure/GRPC/UserService.php

namespace App\User\Infrastructure\GRPC;

use App\User\Application\Query\GetUserQuery;
use App\User\Application\Query\GetUserQueryHandler;
use GRPC\User\V1\GetUserRequest;
use GRPC\User\V1\GetUserResponse;
use GRPC\User\V1\UserServiceInterface;
use Spiral\RoadRunner\GRPC\ContextInterface;
use Spiral\RoadRunner\GRPC\Exception\GRPCException;
use Spiral\RoadRunner\GRPC\StatusCode;

final readonly class UserService implements UserServiceInterface
{
    public function __construct(private GetUserQueryHandler $handler)
    {
    }

    public function GetUser(ContextInterface $ctx, GetUserRequest $in): GetUserResponse
    {
        $user = $this->handler->__invoke(new GetUserQuery($in->getId()));

        if ($user === null) {
            throw new GRPCException(
                sprintf('User with id "%s" not found', $in->getId()),
                StatusCode::NOT_FOUND
            );
        }

        $response = new GetUserResponse();
        $response->setId($user->id);
        $response->setEmail($user->email);

        return $response;
    }
}

Updating gRPC Configuration

We add the path of theproto/user.proto file to the RoadRunner configuration file rr.yaml.

# rr.yaml
...

grpc:
    listen: "tcp://0.0.0.0:9001"
    proto:
        - "proto/user.proto"
...

You need to define and register the gRPC service manually public/grppc-worker.php because gRPC in PHP (via RoadRunner) doesn’t automatically discover your service implementations.

# public/grpc-worker.php
...

$worker = Worker::create();
$server = new Server();
$server->registerService(UserServiceInterface::class, $container->get(UserService::class));

...

Update services

Setting public: true for UserService allows you to manually retrieve it from the container in your gRPC worker script:

# config/services.yaml
...
    App\:
        resource: '../src/'

    App\User\Infrastructure\GRPC\UserService:
        public: true
...

Run the project

Finally, it’s time to test all our code!

Start the server

We use Docker Compose to generate the files and start our gRPC server.

docker compose up -d

Testing the gRPC User Service

We use our preferred client to test that our service is functioning correctly. In my case, I will use GRPCurl.

 grpcurl -plaintext \
  -proto proto/user.proto \
  -d '{"id":"f5fb962d-60f6-4096-8bd6-f717249d46fa"}' \
  localhost:9001 \
  user.v1.UserService/GetUse

Congratulations! You now have your first gRPC microservice up and running with Symfony.

Repository

The code that the tutorial is based on can be found in the public repository.

GitHub - albertcolom/symfony-grpc

NOTE: The tutorial uses the minimum required version of the code. The complete project can be found in the repository.

Original published at: albertcolom.com

How to process gigantic files in Golang — Fast and Memory-Efficient

Albert Colom — Fri, 20 Jun 2025 02:08:22 +0000

How to process gigantic files in Golang — Fast and Memory-Efficient

A practical guide with code for handling millions of lines using Go’s concurrency and object pooling.

Why Golang is a good choice for very huge files?

Efficient Memory Usage : Go allows reading files in chunks (e.g., using bufio or io.Reader) so you don’t have to load everything into RAM.
Concurrency Support : Goroutines and channels let you process parts of the file in parallel, speeding up tasks like filtering, transformation, or computation.
Strong Standard Library : The os, io, bufio, and encoding packages provide tools to process files efficiently.
Static Typing & Performance : Faster and safer than scripting languages for processing large volumes of data.

Sample project

As developers, we often face the challenge of processing very large files without consuming too much memory or CPU — because fewer resources means lower costs 🤑. In this post, I’ll show you how to handle huge files efficiently in Go, line by line, using concurrency and memory reuse.

Project Structure

├── dummy_100000000_rows.csv
├── generate.sh
├── main.go
├── model.go
├── processor.go

Model

The model.go defines reusable data structures (Payload and Chunk) and uses sync.Pool to efficiently manage their memory, minimizing allocations and garbage collection when processing large amounts of data.

package main

import "sync"

type Payload struct {
 Data []byte
}

type Chunk struct {
 payloads []*Payload
}

var payloadPool = sync.Pool{
 New: func() interface{} {
  return &Payload{}
 },
}

var chunkPool = sync.Pool{
 New: func() interface{} {
  return &Chunk{
   payloads: make([]*Payload, 0),
  }
 },
}

func getPayload() *Payload {
 return payloadPool.Get().(*Payload)
}

func releasePayload(payload *Payload) {
 payload.Data = payload.Data[:0]
 payloadPool.Put(payload)
}

func getChunk() *Chunk {
 return chunkPool.Get().(*Chunk)
}

func releaseChunk(chunk *Chunk) {
 chunk.payloads = chunk.payloads[:0]
 chunkPool.Put(chunk)
}

Processor

The processor.go reads a large file using buffered scanning, groups lines into chunks, and processes them concurrently using a worker pool. It leverages buffered channels and memory pools to efficiently manage data flow and minimize allocations, making it ideal for fast, low-overhead processing of large files.

package main

import (
 "bufio"
 "io"
 "log"
 "sync"
)

func processFile(file io.Reader, numWorkers int, chunkSize int) {
 scanner := bufio.NewScanner(file)
 payloadChan := make(chan *Chunk, numWorkers)
 var wg sync.WaitGroup

 for i := 0; i < numWorkers; i++ {
  wg.Add(1)
  go func(workerId int) {
   defer wg.Done()
   for chunk := range payloadChan {
    for _, payload := range chunk.payloads {
     // Actual processing logic goes here
     // fmt.Println(string(payload.Data))

     releasePayload(payload)
    }

    releaseChunk(chunk)
   }
  }(i)
 }

 go func() {
  defer close(payloadChan)
  for {
   chunk := getChunk()
   for len(chunk.payloads) < chunkSize && scanner.Scan() {
    payload := getPayload()

    payload.Data = payload.Data[:0]
    payload.Data = append(payload.Data[:0], scanner.Bytes()...)

    chunk.payloads = append(chunk.payloads, payload)
   }

   if len(chunk.payloads) > 0 {
    payloadChan <- chunk
   } else {
    releaseChunk(chunk)
    break
   }
  }

  if err := scanner.Err(); err != nil {
   log.Printf("Scanner error: %v", err)
  }
 }()

 wg.Wait()
}

Main

The main.go is a simple command-line interface that accepts user parameters, opens the specified file, and runs the processing logic with those settings. After processing, it outputs useful information about the process, such as execution time and memory usage, to help monitor performance during large file handling.

package main

import (
 "flag"
 "log"
 "os"
 "runtime"
 "time"
)

func main() {
 filePath := flag.String("file", "", "Path to CSV file to process (required)")
 chunkSize := flag.Int("chunk", 1000, "Number of lines per chunk (default: 1000)")
 workers := flag.Int("workers", runtime.NumCPU(), "Number of concurrent workers (default: number of CPUs)")
 flag.Parse()

 if *filePath == "" {
  log.Fatal("Missing required --file parameter")
 }

 startTime := time.Now()

 file, err := os.Open(*filePath)
 if err != nil {
  log.Fatalf("Failed to open file: %v", err)
 }
 defer file.Close()

 stat, err := file.Stat()
 if err != nil {
  log.Fatalf("Failed to get file stats: %v", err)
 }

 fileSize := stat.Size()
 log.Printf("Processing file: %s (%.2f MB)", *filePath, float64(fileSize)/1024/1024)
 log.Printf("Number of workers: %d", *workers)
 log.Printf("Chunks size: %d", *chunkSize)

 processFile(file, *workers, *chunkSize)

 printMemStats()
 log.Printf("Processing completed in %v", time.Since(startTime))
}

func printMemStats() {
 var m runtime.MemStats
 runtime.ReadMemStats(&m)

 log.Printf("Memory stats: Alloc=%.2fMB, TotalAlloc=%.2fMB, Sys=%.2fMB, NumGC=%d",
  float64(m.Alloc)/1024/1024,
  float64(m.TotalAlloc)/1024/1024,
  float64(m.Sys)/1024/1024,
  m.NumGC,
 )
}

CLI-supported parameters:

+------------+---------------------------------------------------+---------+-----------------------+----------+
| Flag | Description | Type | Default Value | Required |
+------------+---------------------------------------------------+---------+-----------------------+----------+
| --file | Path to the CSV file to process | string | None | Yes |
| --chunk | Number of lines per processing chunk | int | 1000 | No |
| --workers | Number of concurrent worker goroutines to use | int | Number of CPU cores | No |
+------------+---------------------------------------------------+---------+-----------------------+----------+

Example Usage:

go run main.go --file=dummy_10000000_rows.csv --chunk=1000 --workers=8

You will get a similar output depending on your file and computer.

Bonus point: Generate sample data

It includes a handy simple Bash script (generate.sh) that quickly creates realistic sample data in CSV format, perfect for testing and demonstration.

#!/bin/bash

DEFAULT_ROWS=1000000
NUM_ROWS=${1:-$DEFAULT_ROWS}
FILENAME="dummy_${NUM_ROWS}_rows.csv"

echo "Generating $NUM_ROWS rows..."
echo "ID,Name,Surname,Gender,Age,City,Score,Code,Status,Date,Email,Country,Verified,Phone,IP,Browser,OS,Device,Source,Plan,Referral" > "$FILENAME"

awk -v rows="$NUM_ROWS" '
BEGIN {
  srand();

  split("Alice,Bob,Charlie,David,Eve,Frank,Grace,Hannah,Ian,Julia,Sam,Manuel,Francisco,Fred,Carlos", names, ",");
  split("Smith,Johnson,Williams,Brown,Jones,Garcia,Miller,Davis,Martinez,Lopez,Marshall,Andres", surnames, ",");
  split("Male,Female,Other", genders, ",");
  split("New York,London,Berlin,Barcelona,Tokyo,Sydney,Paris,Dubai,Rome,Madrid,Beijing", cities, ",");
  split("USA,UK,Germany,Japan,Australia,France,UAE,Italy,Spain,China", countries, ",");
  split("example.com,test.org,demo.net,sample.io", domains, ",");
  split("Chrome,Firefox,Safari,Edge,Opera", browsers, ",");
  split("Windows,macOS,Linux,Android,iOS", oss, ",");
  split("Desktop,Laptop,Tablet,Mobile", devices, ",");
  split("Google,Facebook,Twitter,LinkedIn,Direct,Email", sources, ",");
  split("Free,Basic,Pro,Enterprise", plans, ",");

  for(i=1;i<=rows;i++) {
    name = names[int(rand()*length(names))+1];
    surname = surnames[int(rand()*length(surnames))+1];
    gender = genders[int(rand()*length(genders))+1];
    age = 18 + int(rand()*63);
    city = cities[int(rand()*length(cities))+1];
    score = sprintf("%.2f", rand()*100);
    code = int(rand()*99999) + 10000;
    status = (rand() > 0.7) ? "Suspended" : ((rand() > 0.5) ? "Active" : "Inactive");
    day = sprintf("%02d", 1 + int(rand()*28));
    date = "2025-" sprintf("%02d", 1 + int(rand()*12)) "-" day;
    email = tolower(name) i "@" domains[int(rand()*length(domains))+1];
    country = countries[int(rand()*length(countries))+1];
    verified = (rand() > 0.5) ? "true" : "false";
    phone = sprintf("+1-%03d-%03d-%04d", int(rand()*900+100), int(rand()*900+100), int(rand()*10000));
    ip = int(rand()*255) "." int(rand()*255) "." int(rand()*255) "." int(rand()*255);
    browser = browsers[int(rand()*length(browsers))+1];
    os = oss[int(rand()*length(oss))+1];
    device = devices[int(rand()*length(devices))+1];
    source = sources[int(rand()*length(sources))+1];
    plan = plans[int(rand()*length(plans))+1];
    referral = sprintf("REF_%04d", int(rand()*10000));

    printf "%d,%s,%s,%s,%d,%s,%s,%d,%s,%s,%s,%s,%s,%s,%s,%s,%s,%s,%s,%s,%s\n",
           i, name, surname, gender, age, city, score, code, status, date, email, country, verified,
           phone, ip, browser, os, device, source, plan, referral;
  }
}' >> "$FILENAME"

echo "CSV file with $NUM_ROWS rows generated: $FILENAME"

FILE_SIZE=$(du -h "$FILENAME" | cut -f1)
echo "File size: $FILE_SIZE"

#Assign execution permissions
chmod +x generate.sh

#Example to execute with number of lines. Default value 1000000
./generate.sh 10000000

And when we run it we have a similar output:

NOTE : Depending on the number of rows and the power of the computer, this script can take a long time.

Conclusion

Processing huge files in Go doesn’t have to be hard. Using Go’s concurrency and memory reuse features, you can handle big CSVs or text files efficiently without using too much memory. This makes your programs faster and more reliable, especially when dealing with large datasets

Repository

The code for this tutorial can be found in the public: GitHub - albertcolom/go-file-processor

Original published at: albertcolom.com

How to use migrations with Golang

Albert Colom — Wed, 06 Nov 2024 17:32:32 +0000

Simple sample application showing how to use golang-migrate

Why you should use migrations?

Many people ask this question and I have tried to make this list to highlight the main advantages about the use of migrations:

Version Control: One of the main and most important is to be able to have a versioning of the different modifications of the database schema. Without the migrations, these schema changes would be incoherent and impossible to track, which would cause versioning problems and possible errors.

Rollback: It's always necessary to have a rollback system in case of any failure. A migrations system always has two methods up to apply the changes in the database and down in charge of reverting the changes quickly and consistently :-)

Automation and CI/CD Integration: Migrations can be automated, allowing them to be part of the CI/CD pipeline. This helps in deploying changes smoothly and consistently without manual intervention.

We can find many more advantages but I think these points represent a good summary of the main advantages.

How to implement migrations in Golang?

Go doesn't support migrations natively for that propuso we can use the popular golang-migrate package also if you use an ORM like GORM you can use it for that.

Both packages are very popular but in this example I'll use golang-migrate because I am not interested in implementing an ORM.

Show me the code!

Let's see step by step how to implement a simple application to see how it is used.

To follow this article you'll need: Go and Docker with Docker Compose

Infrastructure

Create the file docker-compose.yml in your root directory where we'll define your favourite DB, in my case use a MariaDB but feel free to use another one.

services:
  mariadb:
    image: mariadb:11.5.2
    container_name: mariadb_example_go_migration
    ports:
      - "3306:3306"
    environment:
      - MYSQL_DATABASE=app
      - MYSQL_ROOT_PASSWORD=root
      - TZ=Europe/Berlin
    volumes:
      - mariadbdata:/var/lib/mysql

volumes:
  mariadbdata:
    driver: local

docker compose up -d

If you prefer you can use Docker directly instead of docker-compose:

docker volume create -d local mariadbdata
docker run --name mariadb_example_go_migration -p 3306:3306 -e MYSQL_DATABASE=app -e MYSQL_ROOT_PASSWORD=root -e TZ=Europe/Berlin -v mariadbdata:/var/lib/mysql mariadb:11.5.2

Environment values

Create or update the file .env in your root directory where you need to define the variables to connect our data base.

DATABASE_DSN=root:root@tcp(localhost:3306)/app

Create a simple golang app

Create a simple golang application to ensure successful DB connection and list all tables and structure in the database with their structure. cmd/main.go

package main

import (
 "database/sql"
 "fmt"
 "log"
 "os"
 "text/tabwriter"

 _ "github.com/go-sql-driver/mysql"
 "github.com/joho/godotenv"
)

func main() {
 // Load .env variables
 err := godotenv.Load()
 if err != nil {
  log.Fatal("Error loading .env file")
 }

 // Open connection with MySQL DB
 db, err := sql.Open("mysql", os.Getenv("DATABASE_DSN"))
 if err != nil {
  log.Fatalf("Error opening database: %v\n", err)
 }
 defer db.Close()

 // Ensure that the connection works
 err = db.Ping()
 if err != nil {
  log.Fatalf("Error connecting database: %v\n", err)
 }

 fmt.Println("Connected to database")

 // Execute the SHOW TABLES query to list all tables in the database
 tables, err := db.Query("SHOW TABLES")
 if err != nil {
  log.Fatalf("Failed to execute SHOW TABLES query: %v\n", err)
 }
 defer tables.Close()

 fmt.Println("Database structure:")

 for tables.Next() {
  var tableName string
  if err := tables.Scan(&tableName); err != nil {
   log.Fatalf("Failed to scan table name: %v\n", err)
  }

  w := tabwriter.NewWriter(os.Stdout, 0, 0, 2, ' ', tabwriter.Debug)

  fmt.Printf("\n[Table: %s]\n\n", tableName)
  fmt.Fprintf(w, "%s\t%s\t%s\t%s\t%s\t%s\t\n", "Field", "Type", "Null", "Key", "Default", "Extra")

  // Get the structure of the current table
  structureQuery := fmt.Sprintf("DESCRIBE %s", tableName)
  columns, err := db.Query(structureQuery)
  if err != nil {
   log.Fatalf("Failed to describe table %s: %v\n", tableName, err)
  }
  defer columns.Close()

  for columns.Next() {
   var field, colType, null, key, defaultVal, extra sql.NullString
   err := columns.Scan(&field, &colType, &null, &key, &defaultVal, &extra)
   if err != nil {
    log.Fatalf("Failed to scan column: %v\n", err)
   }

   fmt.Fprintf(w, "%s\t%s\t%s\t%s\t%s\t%s\t\n",
    field.String, colType.String, null.String, key.String, defaultVal.String, extra.String)
  }

  w.Flush()
 }
}

And when we run it we have a similar output:

Migrate CLI

To run golang-migrate CLI basically you have two methods install CLI locally or run through oficial Docker image: migrate/migrate.

Personally I prefer de docker variant but in this tutorial illustrate both variants.

How to generate migration

The first step is create an empty migration with the next command.

#CLI variant
migrate create -ext sql -dir ./database/migrations -seq create_users_table

#Docker CLI variant
docker run --rm -v $(pwd)/database/migrations:/migrations migrate/migrate \
    create -ext sql -dir /migrations -seq create_users_table

ext: Extension of the file to be generated.
dir: Directory where our migration will be created.
seq: Migration sequence name.

This command will be generated two empty files on database/migrations/ folder: 000001_create_users_table.up.sql and 000001_create_users_table.down.sql

On 000001_create_users_table.up.sql file define SQL for create a table users:

CREATE TABLE `users` (
    `id` VARCHAR(36) NOT NULL PRIMARY KEY,
    `name` VARCHAR(255) NOT NULL,
    `email` VARCHAR(255) NOT NULL UNIQUE,
    `password` VARCHAR(255) NOT NULL
);

On 000001_create_users_table.down.sql file define SQL to revert all changes made by up, in this case we have to delete users table:

DROP TABLE IF EXISTS `users`;

How to apply migration

The following command applies all pending migrations. You can also define the number of migrations to apply by adding the number after the up.

#CLI variant
migrate -path=./database/migrations -database "mysql://root:root@tcp(localhost:3306)/app" up

#Docker CLI variant
docker run --rm -v $(pwd)/database/migrations:/migrations --network host migrate/migrate \
    -path=/migrations -database "mysql://root:root@tcp(localhost:3306)/app" up

path: Path to migrations directory.
database: Define you database DSN connection.

NOTE : When running the migration for the first time a table "schema_migrations" will be created in which the migration knows the version number applied.

And run our Golang application to display the results:

Adding new migration

Add a new column phone on users table

#CLI variant
migrate create -ext sql -dir ./database/migrations -seq add_column_phone

#Docker CLI variant
docker run --rm -v $(pwd)/database/migrations:/migrations migrate/migrate \
    create -ext sql -dir /migrations -seq add_column_phone

-- 000002_add_column_phone.up.sql
ALTER TABLE `users` ADD `phone` VARCHAR(255) NULL;

-- 000002_add_column_phone.down.sql
ALTER TABLE `users` DROP `phone`;

#CLI variant
migrate -path=./database/migrations -database "mysql://root:root@tcp(localhost:3306)/app" up

#Docker CLI variant
docker run --rm -v $(pwd)/database/migrations:/migrations --network host migrate/migrate \
    -path=/migrations -database "mysql://root:root@tcp(localhost:3306)/app" up

And when you run it from our Golang application you can see the new field:

How to revert migration

With the following command we can easily rollback the applied. migrations. In the following example we can see how we reverse the last migration applied:

#CLI variant
migrate -path=./database/migrations -database "mysql://root:root@tcp(localhost:3306)/app" down 1

#Docker CLI variant
docker run --rm -it -v $(pwd)/database/migrations:/migrations --network host migrate/migrate \
    -path=/migrations -database "mysql://root:root@tcp(localhost:3306)/app" down 1

WARNING : If you don't define the number of migrations, ROLLBACK will be applied to ALL MIGRATIONS!

And then we can show that the last migration has been reverted and the phone field has been removed :-)

How to Resolve Migration Errors

If a migration contains errors and is executed, that migration cannot be applied and the migration system will prevent any further migrations on the database until this migration is fixed.

And when trying to apply we will get a message like this:

error: Dirty database version 2. Fix and force version.

Don't panic, it is not difficult to get back to a consistent system.

First we have to resolve the corrupted migration, in this case version 2.

Once the migration is solved we have to force the system to the last valid version, in this case version 1.

#CLI variant
migrate -path=./database/migrations -database "mysql://root:root@tcp(localhost:3306)/app" force 1

#Docker CLI variant
docker run --rm -it -v $(pwd)/database/migrations:/migrations --network host migrate/migrate \
    -path=/migrations -database "mysql://root:root@tcp(localhost:3306)/app" force 1

And now you can reapply the migrations without any problem ;-)

Makefile

To improve our productivity and facilitate the use of these commands we can use Makefile. Below you can see the two variants: native client and docker.

CLI variant

include .env

.PHONY: help create-migration migrate-up migrate-down migrate-force

help: ## Show help
 @echo "\n\033[1mAvailable commands:\033[0m\n"
 @@awk 'BEGIN {FS = ":.*##";} /^[a-zA-Z_-]+:.*?##/ { printf " \033[36m%-20s\033[0m %s\n", $$1, $$2 } /^##@/ { printf "\n\033[1m%s\033[0m\n", substr($$0, 5) } ' $(MAKEFILE_LIST)

create-migration: ## Create an empty migration
 @read -p "Enter the sequence name: " SEQ; \
   migrate create -ext sql -dir ./database/migrations -seq $${SEQ}

migrate-up: ## Migration up
 @migrate -path=./database/migrations -database "mysql://${DATABASE_DSN}" up

migrate-down: ## Migration down
 @read -p "Number of migrations you want to rollback (default: 1): " NUM; NUM=$${NUM:-1}; \
 migrate -path=./database/migrations -database "mysql://${DATABASE_DSN}" down $${NUM}

migrate-force: ## Migration force version
 @read -p "Enter the version to force: " VERSION; \
 migrate -path=./database/migrations -database "mysql://${DATABASE_DSN}" force $${VERSION}

Docker CLI variant

include .env

.PHONY: help create-migration migrate-up migrate-down migrate-force

help: ## Show help
 @echo "\n\033[1mAvailable commands:\033[0m\n"
 @@awk 'BEGIN {FS = ":.*##";} /^[a-zA-Z_-]+:.*?##/ { printf " \033[36m%-20s\033[0m %s\n", $$1, $$2 } /^##@/ { printf "\n\033[1m%s\033[0m\n", substr($$0, 5) } ' $(MAKEFILE_LIST)

create-migration: ## Create an empty migration
 @read -p "Enter the sequence name: " SEQ; \
   docker run --rm -v ./database/migrations:/migrations migrate/migrate \
       create -ext sql -dir /migrations -seq $${SEQ}

migrate-up: ## Migration up
 @docker run --rm -v ./database/migrations:/migrations --network host migrate/migrate \
       -path=/migrations -database "mysql://${DATABASE_DSN}" up

migrate-down: ## Migration down
 @read -p "Number of migrations you want to rollback (default: 1): " NUM; NUM=$${NUM:-1}; \
 docker run --rm -it -v ./database/migrations:/migrations --network host migrate/migrate \
       -path=/migrations -database "mysql://${DATABASE_DSN}" down $${NUM}

migrate-force: ## Migration force version
 @read -p "Enter the version to force: " VERSION; \
 docker run --rm -it -v ./database/migrations:/migrations --network host migrate/migrate \
       -path=/migrations -database "mysql://${DATABASE_DSN}" force $${VERSION}

Repository

The code for this tutorial can be found in the public: GitHub - albertcolom/example-go-migration

Original published at: albertcolom.com

Top 10 Most Used Regex Patterns

Albert Colom — Wed, 15 Nov 2023 14:47:16 +0000

List of the most commonly used regular expressions that you should know

Regular expressions or regex are used to search for and match patterns in strings or text data. This is useful for tasks such as validating input, searching for specific words or phrases, and extracting information from text.

In this post I have tried to gather some of the most used examples, I hope you find them useful ;-)

1- Regex Email Validation

Regex pattern that matches a valid email address.

Example: name@domain.com➞ https://regex101.com/r/1sIXyA/1

/^([a-zA-Z0-9._%-]+@[a-zA-Z0-9.-]+\.[a-zA-Z]{2,6})*$/

2- Regex URL Validation

Regex pattern that matches a valid URL starting with http or https.

Example: https://www.domain.com➞ https://regex101.com/r/gTO9nq/1

/^(https?:\/\/)?([a-z0-9]+([\-\.]{1}[a-z0-9]+)*\.[a-z]{2,5})(:[0-9]{1,5})?(\/.*)?$/

3- Regex Dates Validation

Regex pattern that matches a valid date in different formats.

3.1- Date Format YYYY-mm-dd

Example: 2001–10-25➞ https://regex101.com/r/xZOOo9/1

/^([12]\d{3}-(0[1-9]|1[0-2])-(0[1-9]|[12]\d|3[01]))$/

3.2- Date Format dd-mm-YYYY

Example: 25–10-2001➞ https://regex101.com/r/p4pFsm/1

/^((0[1-9]|[12]\d|3[01])-(0[1-9]|1[0-2])-[12]\d{3})$/

4- Regex Time Validation

Regex pattern that matches a valid time in different formats.

4.1- Time Format HH:mm AM/PM

Example: 11:45 PM➞ https://regex101.com/r/vq9s8Z/1

/^(1[0-2]|0?[1-9]):[0-5][0-9] (AM|PM)$/

4.1- Time Format hh:mm:ss

Example: 19:45:54➞ https://regex101.com/r/cMqbxL/1

/^(0[0-9]|1[0-9]|2[1-4]):(0[0-9]|[1-5][0-9]):(0[0-9]|[1-5][0-9])$/

5- Regex Datetime Validation

Regex pattern that matches a valid datetime.

5.1- Datetime Format YYYY-mm-dd hh:mm:ss

Example: 2001–10-25 10:59:59➞ https://regex101.com/r/GcnzIJ/1

/^([12]\d{3}-(0[1-9]|1[0-2])-(0[1-9]|[12]\d|3[01]) (0[0-9]|1[0-9]|2[1-4]):(0[0-9]|[1-5][0-9]):(0[0-9]|[1-5][0-9]))$/

5.2- Datetime Format dd-mm-YYYY hh:mm:ss

Example: 10–11-2001 10:59:59➞ https://regex101.com/r/qfc0oc/1

/^((0[1-9]|[12]\d|3[01])-(0[1-9]|1[0-2])-[12]\d{3} (0[0-9]|1[0-9]|2[1-4]):(0[0-9]|[1-5][0-9]):(0[0-9]|[1-5][0-9]))$/

6- Regex UUID Validation

Regex pattern that matches a valid UUID (Universal Unique Identifier).

Example: 20354d7a-e4fe-47af-8ff6-187bca92f3f9 ➞ https://regex101.com/r/ESgo7B/1

/^[0-9a-fA-F]{8}\-[0-9a-fA-F]{4}\-[0-9a-fA-F]{4}\-[0-9a-fA-F]{4}\-[0-9a-fA-F]{12}$/

7- Regex IP Address Validation

Regex pattern that matches a valid IP Address version 4 and version 6.

7.1- IPv4 address

Example: 127.3.1.1 ➞ https://regex101.com/r/6kWr5c/1

/^(([0-9]|[1-9][0-9]|1[0-9]{2}|2[0-4][0-9]|25[0-5])\.){3}([0-9]|[1-9][0-9]|1[0-9]{2}|2[0-4][0-9]|25[0-5])$/

7.2- IPv6 address

Example: ee1a:5b37:e33f:811d:1cc8:3607:af73:1e23 ➞ https://regex101.com/r/4yy1CE/1

/^(([0-9a-fA-F]{1,4}:){7,7}[0-9a-fA-F]{1,4}|([0-9a-fA-F]{1,4}:){1,7}:|([0-9a-fA-F]{1,4}:){1,6}:[0-9a-fA-F]{1,4}|([0-9a-fA-F]{1,4}:){1,5}(:[0-9a-fA-F]{1,4}){1,2}|([0-9a-fA-F]{1,4}:){1,4}(:[0-9a-fA-F]{1,4}){1,3}|([0-9a-fA-F]{1,4}:){1,3}(:[0-9a-fA-F]{1,4}){1,4}|([0-9a-fA-F]{1,4}:){1,2}(:[0-9a-fA-F]{1,4}){1,5}|[0-9a-fA-F]{1,4}:((:[0-9a-fA-F]{1,4}){1,6})|:((:[0-9a-fA-F]{1,4}){1,7}|:)|fe80:(:[0-9a-fA-F]{0,4}){0,4}%[0-9a-zA-Z]{1,}|::(ffff(:0{1,4}){0,1}:){0,1}((25[0-5]|(2[0-4]|1{0,1}[0-9]){0,1}[0-9])\.){3,3}(25[0-5]|(2[0-4]|1{0,1}[0-9]){0,1}[0-9])|([0-9a-fA-F]{1,4}:){1,4}:((25[0-5]|(2[0-4]|1{0,1}[0-9]){0,1}[0-9])\.){3,3}(25[0-5]|(2[0-4]|1{0,1}[0-9]){0,1}[0-9]))$/

8- Regex File Validation

Regex pattern that matches a valid file path.

8.1- Absolute file path with extension

Example: /some/path-to-fie/resource.zip ➞ https://regex101.com/r/3jdxKY/1

/^((\/|\\|\/\/|https?:\\\\|https?:\/\/)[a-z0-9_@\-^!#$%&+={}.\/\\\[\]]+)+\.[a-z]+$/

8.2- File with extension having 3 chars

Example: resource.zip ➞ https://regex101.com/r/mRqdIg/1

/^[\w,\s-]+\.[A-Za-z]{3}$/

8.3- File with extension validation

Example: resource.png ➞ https://regex101.com/r/Rj980C/1

/^[\w,\s-]+\.(jpg|jpeg|png|gif|pdf)$/

9- Regex Password Strength Validation

Regex pattern matching a strength rules of the password.

9.1- Complex

Should have 1 lowercase letter, 1 uppercase letter, 1 number, 1 special character and be at least 8 characters long.

Example: mYpa$$word123 ➞ https://regex101.com/r/mAC0uS/1

/^(?=(.*[0-9]))(?=.*[\!@#$%^&*()\\[\]{}\-_+=~`|:;"'<>,.\/?])(?=.*[a-z])(?=(.*[A-Z]))(?=(.*)).{8,}$/

9.2- Moderate

Should have 1 lowercase letter, 1 uppercase letter, 1 number, and be at least 8 characters long.

Example: passWord123 ➞ https://regex101.com/r/7JBDjg/1

/^(?=(.*[0-9]))((?=.*[A-Za-z0-9])(?=.*[A-Z])(?=.*[a-z]))^.{8,}$/

10- Regex Slug Validation

Regex pattern that matches a valid Slug.

Example: some-valid-slug ➞ https://regex101.com/r/0Bo0eH/1

/^[a-z0-9]+(?:-[a-z0-9]+)*$/

Original published at: albertcolom.com

Easy steps to install K3s with SSL certificate by traefik, cert manager and Let’s Encrypt

Albert Colom — Sun, 24 Sep 2023 12:08:52 +0000

How to install k3s + Traefik + CertManager + LetsEncrypt

Why use k3s?

k3s is a lightweight Kubernetes distribution designed to be minimal and efficient, making it well-suited for resource-constrained environments and use cases where simplicity and ease of deployment are important. It was created by Rancher Labs and is intended to simplify the installation and management of Kubernetes clusters.

In this example use k3s with Traefik ingress controller so it’s a default by K3s and it’s a lightweight, easy, and fast solution, but if you prefer another one feel free to use it.

Why use cert manager?

Using cert-manager on Kubernetes simplifies SSL/TLS certificate management, automates the renewal process, integrates seamlessly with Kubernetes resources, and provides the flexibility to work with various certificate issuers. This results in enhanced security and reduced operational overhead for securing your Kubernetes applications.

1. Install k3s

curl -sfL https://get.k3s.io | sh -

If you want to have access to the k3s cluster outside the node, you can use the following parameter when creating the cluster --tls-san.

curl -sfL https://get.k3s.io | INSTALL_K3S_EXEC="--tls-san <public ip address or hostname>" sh -

By default, you do not have to execute permissions on k3.conf to resolve you need to move the file and give it the necessary permissions.

NOTE : It’s not recommended to give permissions to the original file.

mkdir $HOME/.kube
sudo cp /etc/rancher/k3s/k3s.yaml $HOME/.kube/config
sudo chmod 644 $HOME/.kube/config

export KUBECONFIG=~/.kube/config

You can add KUBECONFIG=~/.kube/config to your ~/.profile or ~/.bashrc to make it persist on reboot.

2. Install helm (optional)

This step is completely optional in order to follow the tutorial but highly recommended.

Helm is a package manager for Kubernetes that simplifies the deployment and management of applications in Kubernetes clusters.

Install Helm on K3s is really easy, just execute the script and you don’t need to modify any config !

curl -fsSL -o get_helm.sh https://raw.githubusercontent.com/helm/helm/main/scripts/get-helm-3
chmod 700 get_helm.sh
./get_helm.sh

3. Install cert manager

Mainly, we have two ways to install with helm or with kubectl. Personally I prefer to use the helm package manager with all the advantages that comes with it.

Option 1: Install by Helm (recommended)

Add the oficial repository on Helm

helm repo add jetstack https://charts.jetstack.io

Update your local Helm chart repository

helm repo update

And install de cert-manager with namespace cert-manager

helm install \
 cert-manager jetstack/cert-manager \
  --namespace cert-manager \
  --create-namespace \
  --set installCRDs=true

NOTE : You can find the all config parameters on the oficial chart page: https://artifacthub.io/packages/helm/cert-manager/cert-manager

Option 2: Install by kubectl

kubectl apply -f https://github.com/cert-manager/cert-manager/releases/download/v1.12.4/cert-manager.crds.yaml

3.1 Verify the cert manager installation

kubectl -n cert-manager get pod

4. Create the ClusterIssuer resource

Create ClusterIssuer for staging environment

# cluster-issuer-staging.yaml

apiVersion: cert-manager.io/v1
kind: ClusterIssuer
metadata:
  name: letsencrypt-staging
  namespace: default
spec:
  acme:
    server: https://acme-staging-v02.api.letsencrypt.org/directory
    email: <YOUR_EMAIL> # replace for your valid email
    privateKeySecretRef:
      name: letsencrypt-staging
    solvers:
    - selector: {}
      http01:
        ingress:
          class: traefik

kubectl apply -f cluster-issuer-staging.yaml

Create ClusterIssuer for production environment

# cluster-issuer-production.yaml

apiVersion: cert-manager.io/v1
kind: ClusterIssuer
metadata:
  name: letsencrypt-production
  namespace: default
spec:
  acme:
    server: https://acme-v02.api.letsencrypt.org/directory
    email: <YOUR_EMAIL> # replace for your valid email
    privateKeySecretRef:
      name: letsencrypt-production
    solvers:
    - selector: {}
      http01:
        ingress:
          class: traefik

kubectl apply -f cluster-issuer-production.yaml

Verify that it has been properly applied

kubectl get ClusterIssuer -A

And check the status of ClusterIssuer

kubectl describe clusterissuer letsencrypt-staging
kubectl describe clusterissuer letsencrypt-production

5. Let’s play!

Finally we are going to create our certificate

5.1 Create a dummy application

In this step just create a very basic dummy nginx application, if you already have an application you can go to the next step.

Create a deployment using a default image from nginx:alpine

kubectl create deployment nginx --image nginx:alpine

Show the deployments status

kubectl get deployments

kubectl describe deployment nginx

Expose the server at port 80

kubectl expose deployment nginx --port 80 --target-port 80

Check that the service is correct and running

kubectl get svc

5.2 Create a ingress traefik controller

Define the trafik ingress with the cert-manager annotations and the tsl section to be able to manage our certificate.

# ingress ingress-nginx.yaml

apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  annotations:
    cert-manager.io/cluster-issuer: letsencrypt-production
    kubernetes.io/ingress.class: traefik
  labels:
    app: nginx
  name: nginx
  namespace: default
spec:
  rules:
  - host: example.com # Change by your domain
    http:
      paths:
      - backend:
          service:
            name: nginx
            port: 
              number: 80
        path: /
        pathType: Prefix  
  tls:
  - hosts:
    - example.com # Change by your domain
    secretName: example-com-tls

kubectl apply -f ingress-nginx.yaml

Verify that the certificate has actually been created

kubectl get cert -n default

NOTE : I change the host example.com to letsencrypt-k3s.albertcolom.com and change example-com-tls to letsencryptk3s-albertcolom-com-tls .

You can show the valid certificated by Let’s Encrypt!

Conclusion

Once you have installed cert manager it is really easy to manage your certificates together with traefik. You just have to set a couple of parameters in the ingress and the system takes care of everything.

No more excuses for not using a valid certificate!

Original published at: albertcolom.com

How to create your first Golang Lambda With Serverless Framework

Albert Colom — Tue, 29 Aug 2023 21:28:42 +0000

This article explains step-by-step how to implement a simple AWS Lambda with Serverless Framework.

Prerequisites to follow this article

AWS account
aws-cli with the property permissions
Node.js to use Serverless Framework
Go

Why use a AWS lambda?

AWS Lambda is a serverless computing service provided by Amazon Web Services (AWS). It allows you to run your code in response to events without needing to provision or manage servers. With AWS Lambda, you can focus solely on writing your application code while AWS takes care of the infrastructure and scaling for you.

Pros

Cost: You only pay for the compute time your functions consume. There’s no need to pay for idle server resources, making it cost-effective for applications with varying workloads.
Auto-Scaling : Lambda automatically scales your code to handle incoming requests or events. It can run thousands of instances of your function in parallel.
Stateless : Lambda functions are designed to be stateless, meaning they don’t retain memory between invocations. You can use external storage services for persistent data.

Const

Cold Start Latency : Cold starts occur when a function is invoked after being idle for a period of time. Cold starts can introduce latency in your application, which might not be suitable for all use cases.
Limitations: Lambda functions have a maximum execution time limit and have predefined resource limits (CPU, memory) that might not be sufficient for certain applications.

Why use a AWS Serverless Framework?

The Serverless Framework is a tool written in Node that simplifies the deployment and management of serverless applications and functions. It abstracts away many of the complexities involved in setting up and configuring serverless services like AWS Lambda, API Gateway, and others.

Get Started

Step 1: Install Serverless Framework

Open your terminal and run the following command to install the Serverless Framework globally:

npm install -g serverless

NOTE : If you type serverless in the terminal you can choose a template to generate a skeleton of the project.

Serverless crete skeleton project

Step 2: Create simple Health Checks in Go

Create a directory from the project and initialize a Golang project.

NOTE : Feel free to change the name of the project.

mkdir GoServerless

cd GoServerless

go mod init go-lambda-serverless

Finally, create your main.go with a simple health check.

package main

import (
 "context"
 "github.com/aws/aws-lambda-go/events"
 "github.com/aws/aws-lambda-go/lambda"
)

type Response events.APIGatewayProxyResponse

func Handler(ctx context.Context, request events.LambdaFunctionURLRequest) (Response, error) {
 return Response{Body: "It works!", StatusCode: 200}, nil
}

func main() {
 lambda.Start(Handler)
}

Then install all dependencies

go mod tidy

Step 3: Compile de project

env GOARCH=amd64 GOOS=linux go build -ldflags="-s -w" -o bin/health main.go

Step 4: Create a Serverless config file

Create serverless.yml on the root project with the content:

service: go-serverless # Define your service name

provider:
  name: aws
  runtime: go1.x
  region: eu-central-1 # Put your AWS region

package:
  patterns:
    - '!./**'
    - ./bin/**

functions:
  health:
    handler: bin/health # route to binary
    events:
      - http:
          path: /health
          method: get
          cors: true
          private: false

NOTE : You can find the all information on the oficial documentation: https://www.serverless.com/framework/docs/providers/aws/guide/serverless.yml

Step 5: Deploy!

Thanks to the Serverless framework it is very easy to deploy any application, just serverless deploy on the console and the magic happens :-)

serverless deploy

NOTE : If you get more information about the deploy you can add --verbose parameter.

Serverless deploy output

And that's all! the framework takes the responsibility to create the necessary infrastructure on AWS and return the URL to the defined endpoint.

You can try the endpoint by CURL

curl -X GET https://jqlar1r8pd.execute-api.eu-central-1.amazonaws.com/dev/health

Or if you prefer you can try via Postman

Try the endpoint by postman

You can also use the same Serverless Framework

serverless invoke -f health

Try the service by Serverless Framework

Bonus

You can remove all the application including the infrastructure by de Serverless Framework.

serverless remove

Conclusion

In this article we have been able to see step by step how to create a very simple application with GO and deploy it using Serverless Framework, a great tool that makes our life much easier.

I did not pretend to make a very detailed article, I just wanted to show how to make a simple application from zero.

Original published at: albertcolom.com

How to use Arrow Function in PHP with examples

Albert Colom — Fri, 18 Aug 2023 08:54:53 +0000

It is available since PHP version 7.4 but many do not understand how to use it.

Arrow functions, also known as “short closures”, were introduced in PHP 7.4 as a more concise way to define anonymous functions. They are particularly useful for small, simple functions.

Probably if you come from other languages you are already used to using them but if this is not your case I encourage you to continue reading the article where I show you some examples of use.

How to use Arrow Function

Arrow functions are defined using the fn keyword, followed by the function parameters, the arrow (=>) symbol, and the function body.

Here we can see a basic example of how to implement it.

// Anonymous function
$add = function($a, $b) {
    return $a + $b;
};

// Arrow function
$addArrow = fn($a, $b) => $a + $b;

echo $add(2, 3);        // Output: 5
echo $addArrow(2, 3);   // Output: 5

Arrow functions can also capture variables from the surrounding scope, unlike traditional closures, arrow functions do not need to be explicitly defined variables from scope.

$factor = 10;

// Closure
$multiplier = function($n) use ($factor) {
    return $n * $factor;
};

// Arrow function
$multiplierArrow = fn($n) => $n * $factor;

echo $multiplier(5);        // Output: 50
echo $multiplierArrow(5);   // Output: 50

Examples

Now that we have seen how to define an arrow function in PHP, let’s see some examples of how to use them.

In the examples we can see that both the traditional anonymous function and the arrow function achieve the same result.

Using Arrow Functions in Array Map

They square each number in the input array using the array_map function.

“Applies the callback to the elements of the given arrays”

array_map(?callable $callback, array $array, array ...$arrays): array

$numbers = [1, 2, 3, 4, 5];

// Anonymous function with array_map
$squared = array_map(function($n) {
    return $n * $n;
}, $numbers);

// Arrow function with array_map
$squaredArrow = array_map(fn($n) => $n * $n, $numbers);

print_r($squared);
print_r($squaredArrow);

// Output:
//Array
//(
//  [0] => 1
//  [1] => 4
//  [2] => 9
//  [3] => 16
//  [4] => 25
//)
//Array
//(
//  [0] => 1
//  [1] => 4
//  [2] => 9
//  [3] => 16
//  [4] => 25
//)

Using Arrow Functions with Array Filter

We filter all the even numbers of an array using the array_filter function.

Filters elements of an array using a callback function

“Filters elements of an array using a callback function”

array_filter(array $array, ?callable $callback = null, int $mode = 0): array

$numbers = [1, 2, 3, 4, 5, 6, 7, 8, 9, 10];

// Anonymous function with array_filter
$evenNumbers = array_filter($numbers, function($n) {
    return $n % 2 === 0;
});

// Arrow function with array_filter
$evenNumbersArrow = array_filter($numbers, fn($n) => $n % 2 === 0);

print_r($evenNumbers);
print_r($evenNumbersArrow);

// Output:
//Array
//(
//  [1] => 2
//  [3] => 4
//  [5] => 6
//  [7] => 8
//  [9] => 10
//)
//Array
//(
//  [1] => 2
//  [3] => 4
//  [5] => 6
//  [7] => 8
//  [9] => 10
//)

Using Arrow Functions with Array Reduce

We sum all values in array with callback function using the array_reduce function.

“Iteratively reduce the array to a single value using a callback function”

array_reduce(array $array, callable $callback, mixed $initial = null): mixed

$numbers = [1, 2, 3, 4, 5];

// Anonymous function with array_reduce
$sumNumbers = array_reduce($numbers, function($carry, $number) {
    return $carry + $number;
}, 0);

// Arrow function with array_reduce
$sumNumbersArrow = array_reduce($numbers, fn($carry, $number) => $carry + $number, 0);

echo $sumNumbers;       // Output: 15
echo $sumNumbersArrow;  // Output: 15

Using Arrow Functions with usort

Arrow functions can be used in various contexts, including callbacks for array by reference functions like usort. Sort the array by age.

“Sort an array by values using a user-defined comparison function”

usort(array &$array, callable $callback): true

$people = [
    ['name' => 'Alice', 'age' => 28],
    ['name' => 'Bob', 'age' => 22],
    ['name' => 'Charlie', 'age' => 25],
];

// Sorting using usort with anonymous function
usort($people, function($a, $b) {
    return $a['age'] <=> $b['age'];
});

// Sorting using arrow function
usort($people, fn($a, $b) => $a['age'] <=> $b['age']);

print_r($people);

// Output:
//Array
//(
//  [0] => Array
//      (
//          [name] => Bob
//          [age] => 22
//      )
//
//  [1] => Array
//      (
//          [name] => Charlie
//          [age] => 25
//      )
//
//  [2] => Array
//    (
//      [name] => Alice
//      [age] => 28
//    )
//
//)

I think that with these examples we can see how to implement the arrow functions in the most common cases.

If you think that I have left some or you have some doubt about some case, leave me a comment :-)

Conclusion

Arrow functions were introduced in PHP 7.4 as a more concise way to define anonymous functions. They offer several advantages, but also have some limitations. Here are the pros and cons of using arrow functions in PHP:

Pros:

Concise Syntax : Arrow functions have a more compact syntax compared to traditional closures, which can make your code cleaner and more readable, especially for simple operations.
Less Boilerplate : Arrow functions eliminate the need for writing the function keyword and curly braces, reducing the amount of boilerplate code.
Implicit Return : Arrow functions have an implicit return, meaning that the result of the expression is automatically returned without needing a return statement.
Short-lived Functions : Arrow functions are particularly useful for short-lived functions used in higher-order functions like array functions (e.g., array_map, array_filter, array_reduce) and callbacks.

Cons:

Limited Scope : Arrow functions have a limited scope and cannot be used to modify variables outside their scope. They don’t support the use keyword to capture variables from the surrounding scope.
Single Expression : Arrow functions are limited to a single expression. They cannot contain multiple statements or complex logic. For more complex functions, you’ll still need to use traditional closures.
Readability for Complex Logic : While arrow functions can improve code readability for simple operations, they might make more complex logic harder to understand. Traditional closures might be more appropriate in such cases.
Compatibility : Arrow functions were introduced in PHP 7.4. If you need to maintain compatibility with older PHP versions, you cannot use arrow functions.

Choosing Between Arrow Functions and Traditional Closures:

Use arrow functions for simple operations that fit within a single expression and don’t require modifying variables from the surrounding scope.
Use traditional closures when you need more complex logic, multiple statements, or need to capture variables from the surrounding scope.
Consider the overall readability and maintainability of your code when deciding whether to use arrow functions or traditional closures.

In summary, arrow functions can be a powerful tool for writing concise and readable code for simple operations. However, it’s important to be aware of their limitations and choose the appropriate tool based on the specific context and complexity of your code.

Original published at: albertcolom.com

How to create CD with GitHub Actions and Kubernetes in your Golang application

Albert Colom — Wed, 16 Aug 2023 08:24:18 +0000

An example to create pipeline to deploy a simple GO application into k3s cluster with Docker image from scratch.

Previous requirements:

GitHub Account as PHP framework
Kubernetes cluster in this example I use a lightweight version of Kubernetes with K3s

Simple http application with GO

A simple illustrative example with http server on port 80 with basic handle respond on a main url.

If you already have any application you can go to the next step.

package main

import (
    "fmt"
    "net/http"
    "os"
)

func handler(w http.ResponseWriter, r *http.Request) {
    var name, _ = os.Hostname()
    fmt.Fprintf(w, "<h1>This request was processed by host: %s</h1>\n", name)
}

func main() {
    fmt.Fprintf(os.Stdout, "Web Server started. Listening on 0.0.0.0:80\n")
    http.HandleFunc("/", handler)
    http.ListenAndServe(":80", nil)
}

You can run the application to ensure works correctly:

go run main.com

If the application its works then we can prepare docker image to "Dokcerize" the application.

Build docker image from scrath

Firstly, we create a file named infrastucture/docker/Dockerfile to contain all config. We use the alpine image as a builder to compile de Go binary and then use image from scratch to copy binary with SSL certificates and define de binary as entrypoint.

This way we get a minimal docker image that contain your binary application than only around 5 MB!

# Use latest Go alpine image as builder
FROM golang:alpine AS build

# Define app as working directory
WORKDIR /app

# Copy all project into docker image
COPY . .

# Compile the Go Application
RUN CGO_ENABLED=0 GOOS=linux GOARCH=amd64 go build -o /app/bin/demo main.go

# Use minimum image with binary
FROM scratch AS prod

# Copy SSL Certificates from build stage
COPY --from=build /etc/ssl/certs/ca-certificates.crt /etc/ssl/certs/

#Copy the application binary from build stage
COPY --from=build /app/bin/demo /app/bin/demo

# Define your binary as entrypoint
ENTRYPOINT ["/app/bin/demo"]

And we can make the build of the image:

docker build -t demo-app:1 . -f ./infrastucture/docker/Dockerfile

-t: Name and optionally a tag in the name:tag format
-f: Name of the Dockerfile (Default is PATH/Dockerfile)

You can found more options on the oficial documentation: https://docs.docker.com/engine/reference/commandline/build/#options

Connect your K3s cluster with Github

For Github Actions can access to your k3s cluster you need import de kubeconfig into secrets actions.

In k3s you can found the kubeconfig on /etc/rancher/k3s/k3s.yaml. You kubeconfig will like somehow like this:

apiVersion: v1
clusters:
- cluster:
      certificate-authority-data: LS0...
      server: https://your-server.com:6443
  name: default
contexts:
- context:
    cluster: default
    user: default
  name: default
current-context: default
kind: Config
preferences: {}
users:
- name: default
  user:
    client-certificate-data: LS0...
    client-key-data: LS0...

Then you go to your project from Github and add the secret named KUBE_CONFIG with the k3s.yaml content.

Project ⮕ Settings ⮕ Secret and Variables ⮕ Actions

Github action for Continuous Delivery (CD)

Finally we come to the long awaited section. We will see how to create a flow with two steps.

Create a file into root of main project: .github/workflow/cd.yaml where we create a workflow triggered on a push/merge on your principal branch, in this casemain branch.

on:
    push:
        branches: [ "main" ]

First step

In the first step build the docker image and publish on the container registry, in this case use the ghcr (GitHub Container Registry).

You can found more information about github packages: https://docs.github.com/en/packages/learn-github-packages

build:
    runs-on: ubuntu-latest
    steps:
      - name: Checkout
        uses: actions/checkout@v3

      - name: Log in to the Container registry
        uses: docker/login-action@v2
        with:
          registry: ghcr.io
          username: ${{ github.actor }}
          password: ${{ secrets.GITHUB_TOKEN }}

      - name: Build and push
        uses: docker/build-push-action@v3
        with:
          context: .
          platforms: linux/amd64
          file: ./infrastructure/docker/Dockerfile
          push: true
          tags: ghcr.io/${{ github.repository }}/demo-app:${{ github.sha }}

Second step

Publish your docker image from registry to k8s cluster.

First of all need a create a token on github with read:package scope and create secret in json format encoded as base64 that you publish on your k8s secrets.

Let's Base64 encode it first:

echo -n "username:your_token" | base64
#example of output: dXNlcm5hbWU6eW91cl90b2tlbg==

Change username for your GitHub username and your_token with the previous generated token

Create a json and encode to Base64 again:

echo -n  '{"auths":{"ghcr.io":{"auth":"dXNlcm5hbWU6eW91cl90b2tlbg=="}}}' | base64
#example of output: eyJhdXRocyI6eyJnaGNyLmlvIjp7ImF1dGgiOiJkWE5sY201aGJXVTZlVzkxY2w5MGIydGxiZz09In19fQ==

Create a manifest to deploy your application on kuberntes deployment, service, ingress

#infrastructure/kubernetes/demo-app-deployment.yml

apiVersion: apps/v1
kind: Deployment
metadata:
    name: demo-app
spec:
    selector:
        matchLabels:
            app: demo-app
    replicas: 2
    template:
        metadata:
            labels:
            app: demo-app
    spec:
        containers:
            - name: demo-app
              image: ghcr.io/path-to-image/demo-app
              env:
                - name: GIN_MODE
                  value: "release"
              ports:
                - containerPort: 80
              livenessProbe:
                httpGet:
                    path: /
                    port: 80
                initialDelaySeconds: 5
                periodSeconds: 3
              readinessProbe:
                httpGet:
                    path: /
                    port: 80
                initialDelaySeconds: 5
                periodSeconds: 3

#infrastructure/kubernetes/demo-app-service.yml

apiVersion: v1
kind: Service
metadata:
    name: demo-app
spec:
    ports:
        - port: 80
          protocol: TCP
          targetPort: 80
    type: NodePort
    selector:
        app: demo-app

#infrastructure/kubernetes/demo-app-ingress.yml

apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
    annotations:
    name: demo-app
spec:
    ingressClassName: nginx
    rules:
        - host: domain.com
          http:
            paths:
                  - path: /
                    pathType: Prefix
                    backend:
                      service:
                          name: demo-app
                          port:
                            number: 80

Finally, once we have the manifests and secrets, we can create the last step of the workflow.

deploy:
  needs: [build]
  runs-on: ubuntu-latest
  steps:
      - name: Checkout
        uses: actions/checkout@v3

      - name: Log in to the Container registry
        uses: docker/login-action@v2
        with:
          registry: ghcr.io
          username: ${{ github.actor }}
          password: ${{ secrets.GITHUB_TOKEN }}

      - name: Kubernetes set context
        uses: Azure/k8s-set-context@v3
        with:
          method: kubeconfig
          kubeconfig: ${{ secrets.KUBE_CONFIG }}

      - name: Deploy
        uses: Azure/k8s-deploy@v4.4
        with:
          action: deploy
          strategy: basic
          imagepullsecrets: |
            dockerconfigjson-github-com
          manifests: |
            ./infrastructure/kubernetes/demo-app-deployment.yml
            ./infrastructure/kubernetes/demo-app-service.yml
            ./infrastructure/kubernetes/demo-app-ingress.yml
          images: ghcr.io/${{ github.repository }}/demo-app:${{ github.sha }}

And just commit in your project the complete CD workflow config .github/workflow/cd.yaml.

name: CD

on:
  push:
    branches: [ "main" ]

jobs:
  build:
    runs-on: ubuntu-latest
    steps:
      - name: Checkout
        uses: actions/checkout@v3

      - name: Log in to the Container registry
        uses: docker/login-action@v2
        with:
          registry: ghcr.io
          username: ${{ github.actor }}
          password: ${{ secrets.GITHUB_TOKEN }}

      - name: Build and push
        uses: docker/build-push-action@v3
        with:
          context: .
          platforms: linux/amd64
          file: ./infrastructure/docker/Dockerfile
          push: true
          target: prod
          tags: ghcr.io/${{ github.repository }}/demo-app:${{ github.sha }}

  deploy:
    needs: [build]
    runs-on: ubuntu-latest
    steps:
      - name: Checkout
        uses: actions/checkout@v3

      - name: Log in to the Container registry
        uses: docker/login-action@v2
        with:
          registry: ghcr.io
          username: ${{ github.actor }}
          password: ${{ secrets.GITHUB_TOKEN }}

      - name: Kubernetes set context
        uses: Azure/k8s-set-context@v3
        with:
          method: kubeconfig
          kubeconfig: ${{ secrets.KUBE_CONFIG }}

      - name: Deploy
        uses: Azure/k8s-deploy@v4.4
        with:
          action: deploy
          strategy: basic
          imagepullsecrets: |
            dockerconfigjson-github-com
          manifests: |
            ./infrastructure/kubernetes/demo-app-deployment.yml
            ./infrastructure/kubernetes/demo-app-service.yml
            ./infrastructure/kubernetes/demo-app-ingress.yml
          images: ghcr.io/${{ github.repository }}/demo-app:${{ github.sha }}

Then you can go to Actions section in your repository and show all workflows and you can see the summary.

And that's all just have fun and deploy :-)

Original published at: albertcolom.com

Use custom PHP Collection instead Array

Albert Colom — Tue, 15 Aug 2023 19:04:17 +0000

In this article we will see how to define a custom PHP collection

An Array is a basic data structure that stores values key/value without any constrictions or any OOP method and it is very difficult to control and maintain the data they contain.

To solve this problem some modern frameworks like Laravel (Illuminate Collection) or Symfony (Doctrine ArrayCollection) use their own OOP wrapper with a lot of functions.

But if we want to decouple from the framework or we use another framework and do not want to install the dependency we can create our own collection.

In this article, we will see how to build our own collection with some methods to work with objects. In the example, it is focused on a CQRS architecture and we place the collection in the reading layer.

NOTE: In the code examples we have used PHP 8.1 but the code is fully adaptable to any other version.

Custom collection:

Firstly create a custom collection without type validation and add some callable functions fromMap, reduce, map, each, some, filter and other OOP functions like first, last, count, isEmpty, add, values, items, getIterator . This is the base of OOP wrapper for working with arrays.

src/Shared/Domain/Read/Collection.php

<?php

declare(strict_types=1);

namespace App\Shared\Domain\Read;

use ArrayIterator;
use IteratorAggregate;
use Traversable;

abstract class Collection implements IteratorAggregate
{
    public function __construct(private array $elements)
    {
    }

    public static function createEmpty(): static
    {
        return new static([]);
    }

    public static function fromMap(array $items, callable $fn): static
    {
        return new static(array_map($fn, $items));
    }

    public function reduce(callable $fn, mixed $initial): mixed
    {
        return array_reduce($this->elements, $fn, $initial);
    }

    public function map(callable $fn): array
    {
        return array_map($fn, $this->elements);
    }

    public function each(callable $fn): void
    {
        array_walk($this->elements, $fn);
    }

    public function some(callable $fn): bool
    {
        foreach ($this->elements as $index => $element) {
            if ($fn($element, $index, $this->elements)) {
                return true;
            }
        }

        return false;
    }

    public function filter(callable $fn): static
    {
        return new static(array_filter($this->elements, $fn, ARRAY_FILTER_USE_BOTH));
    }

    public function first(): mixed
    {
        return reset($this->elements);
    }

    public function last(): mixed
    {
        return end($this->elements);
    }

    public function count(): int
    {
        return count($this->elements);
    }

    public function isEmpty(): bool
    {
        return empty($this->elements);
    }

    public function add(mixed $element): void
    {
        $this->elements[] = $element;
    }

    public function values(): array
    {
        return array_values($this->elements);
    }

    public function items(): array
    {
        return $this->elements;
    }

    public function getIterator(): Traversable
    {
        return new ArrayIterator($this->elements);
    }
}

Custom typed Collection:

The typed collection extends the above collection and implements the constraints.

NOTE: To avoid creating more code than necessary I have used the webmozzart/assert (pakagist) library but feel free to implement your validations.

src/Shared/Domain/Read/TypedCollection.php

<?php

declare(strict_types=1);

namespace App\Shared\Domain\Read;

use Webmozart\Assert\Assert;

abstract class TypedCollection extends Collection
{
    public function __construct(array $elements = [])
    {
        Assert::allIsInstanceOf($elements, $this->type());

        parent::__construct($elements);
    }

    abstract protected function type(): string;

    public function add(mixed $element): void
    {
        Assert::isInstanceOf($element, $this->type());

        parent::add($element);
    }
}

Example implement Typed Collection:

First of all, we need a create a basic Entity Class, in this case create a simple class called Foo.

<?php

declare(strict_types=1);

namespace App\Context\Foo\Domain\Read\View\Foo;

final readonly class Foo
{
    public function __construct(
        public string $id,
        public string $name
    ) {
    }

    public function toArray(): return
    {
        return [
          'id' => $this->id,
          'name' => $this->name,
        ];
    }

    public function equals(self $other): bool
    {
        return $this->id === $other->id && $this->name === $other->name;
    }
}

Then can create a typed collection that contain a Foo inside

src/Context/Foo/Domain/Read/View/FooCollection.php

<?php

declare(strict_types=1);

namespace App\Context\Foo\Domain\Read\View;

use App\Shared\Domain\Read\TypedCollection;

final class FooCollection extends TypedCollection
{
    protected function type(): string
    {
        return Foo::class;
    }
}

Some examples of how to use it

Create an empty collection, then add two Foo elements and filter elements with other name:

$collection = FooCollection::createEmpty();
$collection->add(new Foo('4dae0971-ac81-43f1-b7e1-952df598af5a', 'name'));
$collection->add(new Foo('42deac29-9661-47e8-8746-062fc784ae1b', 'other'));

# Filter by name "other"

$filteredCollection = $collection->filter(function(Foo $foo) {
    return $foo->mame === 'other';
});

# Or if you prefer you can use Arrow function version

$filteredCollection = $collection->filter(fn (Foo $foo) => $foo->mame === 'other');

Create collection from map function using array :

$array = [
  [
    'id' => '4dae0971-ac81-43f1-b7e1-952df598af5a',
    'name' => 'name',
    'surname' => 'surname',
  ],
  [
    'id' => '42deac29-9661-47e8-8746-062fc784ae1b',
    'name' => 'other',
  ],
];

# Create fromMap

$collection = FooCollection::fromMap($array, function(array $data){
    return new Foo($data['id'], $data['name'])
});


# Or if you prefer you can use Arrow function version

$collection = FooCollection::fromMap(
    $array,
    fn(array $data): Foo => new Foo($data['id'], $data['name'])
);

Concat different function with arrow functions:

$array = [
  [
    'id' => '4dae0971-ac81-43f1-b7e1-952df598af5a',
    'name' => 'name',
    'surname' => 'surname',
  ],
  [
    'id' => '42deac29-9661-47e8-8746-062fc784ae1b',
    'name' => 'other',
  ],
];

# Create a collection from map, then filter by name and finally return array

$collection = FooCollection::fromMap(
    $array,
    fn(array $data): Foo => new Foo($data['id'], $data['name'])
)
->filter(fn (Foo $foo) => $foo->mame === 'other')
->map(fn (Foo $foo) => $foo->toArray());

Original published at: albertcolom.com

How to publish domain events with doctrine listener

Albert Colom — Tue, 01 Aug 2023 12:28:52 +0000

Publish domain events on the Queue system when Doctrine flush entity using clean architectures.

In this example, we use:

Symfony as PHP framework
Symfony Messenger as EventBus
Doctrine as ORM
RabbitMQ as Message broker

Create DomainEvent Entity

First of all, create an abstract Entity of which they will inherit all domain events. This entity will be placed on a shared domain as it will be used in different contexts and of this example it will be placed in the writing layer as we implement CQRS.

src/Shared/Domain/Write/Event/DomainEvent.php

<?php

declare(strict_types=1);

namespace App\Shared\Domain\Write\Event;

use DateTimeImmutable;

abstract class DomainEvent
{
    private const DATE_FORMAT = 'Y-m-d H:i:s';

    public function __construct(private ?string $occurredOn = null)
    {
        $this->occurredOn = $occurredOn ?? (new DateTimeImmutable())->format(self::DATE_FORMAT);
    }

    public function occurredOn(): string
    {
        return $this->occurredOn;
    }
}

Create AggregateRoot Entity

Create an abstract class called AggregateRoot witch inherit all root entities.
The AggregateRoot is used to define the main entity from aggregate and can store on memory and publish the domain events from the entity.

src/Shared/Domain/Write/Aggregate/AggregateRoot.php

<?php

declare(strict_types=1);

namespace App\Shared\Domain\Write\Aggregate;

use App\Shared\Domain\Write\Event\DomainEvent;

abstract class AggregateRoot
{
    private array $domainEvents = [];

    final protected function recordEvent(DomainEvent $domainEvent): void
    {
        $this->domainEvents[] = $domainEvent;
    }

    final public function domainEventsEmpty(): bool
    {
        return empty($this->domainEvents);
    }

    final public function pullDomainEvents(): array
    {
        $recordedEvents = $this->domainEvents;
        $this->domainEvents = [];

        return $recordedEvents;
    }
}

And create a simple root entity to use which uses the above class and record an event when construct.
Firstly create a simple Event like a DTO called FooWasCreated.

src/Context/Foo/Domain/Write/Event/FooWasCreated.php

<?php

declare(strict_types=1);

namespace App\Context\Foo\Domain\Write\Event;

use App\Shared\Domain\Write\Event\DomainEvent;

final class FooWasCreated extends DomainEvent
{
    public function __construct(
        public readonly string $id,
        public readonly string $name,
        public readonly string $createdAt,
        ?string $occurredOn = null
    ) {
        parent::__construct($occurredOn);
    }
}

Then can create a root entity to record the event and store it in memory when construct the entity.

src/Context/Foo/Domain/Write/Foo.php

<?php

declare(strict_types=1);

namespace App\Context\Foo\Domain\Write;

use App\Context\Foo\Domain\Write\Event\FooWasCreated;
use App\Shared\Domain\Write\Aggregate\AggregateRoot;
use DateTimeImmutable;

final class Foo extends AggregateRoot
{
    public function __construct(
        private string $id,
        private string $name,
        private DateTimeImmutable $createdAt
    ) {
        $this->recordEvent(
            new FooWasCreated($id, $name, $createdAt->format('Y-m-d H:i:s'))
        );
    }

    public function id(): string
    {
        return $this->id;
    }

    public function name(): string
    {
        return $this->name;
    }

    public function createdAt(): DateTimeImmutable
    {
        return $this->createdAt;
    }
}

NOTE: This entity must define as Doctrine mapping because when persist this entity trigger the events. In this example not define doctrine infrastructure layer to persist the entity, I think you can find a lot of tutorials about that and I don't want to make this tutorial longer.

Define bus for events with messenger

Define the interface on a shared domain in the write layer.

src/Shared/Domain/Write/Bus/EventBus.php

<?php

declare(strict_types=1);

namespace App\Shared\Domain\Write\Bus\Event;

use App\Shared\Domain\Write\Event\DomainEvent;

interface EventBus
{
    public function publish(DomainEvent ...$domainEvents): void;
}

Implement the interface with the concrete Symfony Messenger on the infrastructure layer. This bus is quite simple just has one method publish is responsible for dispatching messages on MessageBus.

src/Shared/Infrastructure/Bus/MessengerEventBus.php

<?php

declare(strict_types=1);

namespace App\Shared\Infrastructure\Bus;

use App\Shared\Domain\Write\Bus\EventBus;
use App\Shared\Domain\Write\Event\DomainEvent;
use Symfony\Component\Messenger\Envelope;
use Symfony\Component\Messenger\MessageBusInterface;
use Symfony\Component\Messenger\Stamp\DispatchAfterCurrentBusStamp;

final readonly class MessengerEventBus implements EventBus
{
    public function __construct(private MessageBusInterface $messageBus)
    {
    }

    public function publish(DomainEvent ...$domainEvents): void
    {
        foreach ($domainEvents as $currentEvent) {
            $this->messageBus->dispatch(
                (new Envelope($currentEvent))->with(new DispatchAfterCurrentBusStamp())
            );
        }
    }
}

Define Bus service messenger on messenger.yaml in this example define async bus to publish events on the ampqp transport with RabbitMQ.

In this case, we use ampqp because no need other dependences than Symfony Messenger but you can define other transports like a kafka.

NOTE: You can find more information on: https://symfony.com/doc/current/messenger.html#transports-async-queued-messages

.env

MESSENGER_TRANSPORT_DSN=amqp://guest:guest@rabbitmq:5672/%2f/messag

config/packages/messenger.yaml

To simplify the example I just define de basic configuration of the event bus but you should define the other bus like a query or command and the correct retry policy.

framework:
    messenger:
        transports:
            ampqp:
                dsn: '%env(MESSENGER_TRANSPORT_DSN)%'
        buses:
            async.event.bus:
                default_middleware: allow_no_handlers
        routing:
            'App\Shared\Domain\Write\Event\DomainEvent': ampqp

config/services.yaml

To simplify this example I just defined the minimum services definition for this case but you need to define your different services need for your application.

parameters:

services:
    _defaults:
        autowire: true
        autoconfigure: true

    App\:
        resource: '../src/'

    App\Shared\Infrastructure\Bus\MessengerEventBus:
        arguments:
            - '@async.event.bus'

Create Doctrine Listener to publish DomainEvents when flush

The idea is quite simple when flush the entity the listener gets the entities to update from Doctrine UnitOfWork and publish the domain events if they have.

Once the events are published Symfony Messenger takes care of sending the Queue system.

<?php

declare(strict_types=1);

namespace App\Shared\Infrastructure\Listener;

use App\Shared\Domain\Write\Bus\Event\EventBus;
use App\Shared\Domain\Write\Aggregate\AggregateRoot;
use Doctrine\ORM\Event\OnFlushEventArgs;

final readonly class DoctrinePublishDomainEventsOnFlushListener
{
    public function __construct(private EventBus $eventBus)
    {
    }

    public function onFlush(OnFlushEventArgs $eventArgs): void
    {
        $unitOfWork = $eventArgs->getObjectManager()->getUnitOfWork();

        foreach ($unitOfWork->getScheduledEntityInsertions() as $entity) {
            $this->publishDomainEvent($entity);
        }

        foreach ($unitOfWork->getScheduledEntityUpdates() as $entity) {
            $this->publishDomainEvent($entity);
        }

        foreach ($unitOfWork->getScheduledEntityDeletions() as $entity) {
            $this->publishDomainEvent($entity);
        }

        foreach ($unitOfWork->getScheduledCollectionDeletions() as $collection) {
            foreach ($collection as $entity) {
                $this->publishDomainEvent($entity);
            }
        }

        foreach ($unitOfWork->getScheduledCollectionUpdates() as $collection) {
            foreach ($collection as $entity) {
                $this->publishDomainEvent($entity);
            }
        }
    }

    private function publishDomainEvent(object $entity): void
    {
        if ($entity instanceof AggregateRoot && !$entity->domainEventsEmpty()) {
            $this->eventBus->publish(...$entity->pullDomainEvents());
        }
    }
}

And define the listener on a service definition.

config/services.yaml

parameters:

services:
    _defaults:
        autowire: true
        autoconfigure: true

    App\:
        resource: '../src/'

    App\Shared\Infrastructure\Bus\MessengerEventBus:
        arguments:
            - '@async.event.bus'

    App\Shared\Infrastructure\Listener\DoctrinePublishDomainEventsOnFlushListener:
            tags:
                - { name: doctrine.event_listener, event: onFlush }

NOTE: You can find more doctrine events on: https://www.doctrine-project.org/projects/doctrine-orm/en/2.15/reference/events.html#events-overview

Conclusion

In this example, you can see how to implement an EventBus with Symfony Messenger and implement doctrine listener to publish domain events on flush at RabbitMQ following clean architectures.

Original published at: albertcolom.com