DEV Community: Jérôme TAMARELLE

Suppose you need to store documents in MongoDB from an external source such as a RabbitMQ or Kafka queue. The order in which these messages are processed is not guaranteed, especially if there are several processes running in parallel.

To ensure that the latest version of a document is always available, and that it is not overwritten by an older version from another message that was processed later, a version number can be used. The higher the version number, the more recent the document.

The external data source must then include the document version number with each message. This version number can be an increment or simply the timestamp at which the data source generated the message.

In MongoDB, we use the updateOne method to update a document, but to ensure that we don't overwrite a more recent version, we add a condition that checks that the version number of the document in the database is lower than that of the document we wish to insert.

use MongoDB\Collection;

/**
 * @param Collection $collection The MongoDB collection
 * @param int $id                The ID of the document to update
 * @param int $version           The version of the new document
 * @param array $document        The new document
 * @return bool                  True if the new version have been saved, false if it was outdated
 */
function optimisticUpsert(Collection $collection, int $id, int $version, array $document): bool
{
    // The document id and version are stored in the document itself
    $document['_id'] = $id;
    $document['_version'] = $version;

    $result = $collection->updateOne(
        ['_id' => $id],
        [
            ['$replaceWith' => [
                '$cond' => [
                    'if' => ['$lt' => ['$_version', $version]],
                    'then' => $document,
                    'else' => '$$ROOT',
                ],
            ]],
        ],
        ['upsert' => true],
    );

    return $result->getUpsertedCount() > 0 || $result->getModifiedCount() > 0;
}

The MongoDB server is smart enough to know that there are no changes
when the stored version was already more recent than the one you wanted to insert.

If $result->getUpsertedCount() is greater than 0, this means that the document did not exist and has been inserted. If $result->getModifiedCount() is greater than 0, this means that the document already existed and has been updated. Otherwise, it means
that the document version was already more recent than the one to be inserted.

More information on update operations can be found in the documentation :

Expression $replaceWith.
Operator $cond.
Method updateOne.

Mise à jour optimiste des documents versionnés dans MongoDB

Jérôme TAMARELLE — Thu, 13 Mar 2025 19:03:11 +0000

Read this article in English

Supposons que vous ayez besoin de stocker dans MongoDB des documents provenant d'une source externe comme une file d'attente RabbitMQ ou Kafka. L'ordre dans lequel ces messages sont traités n'est pas garanti, surtout s'il y a plusieurs processus de traitement en parallèle.

Afin de garantir de toujours disposer de la dernière version d'un document, et que celle-ci ne soit pas écrasée par une version plus ancienne qui proviendrait d'un autre message ayant été traité plus tard, il est possible d'utiliser un numéro de version. Plus le numéro de version est élevé, plus le document est récent.

La source de données externe doit alors inclure le numéro de version du document à chaque message. Ce numéro de version peut être un incrément ou simplement le timestamp auquel la source de données a généré le message.

Dans MongoDB, on utilise la méthode updateOne pour mettre à jour un document, mais pour s'assurer qu'on n'écrase pas une version plus récente, on ajoute une condition qui vérifie que le numéro de version du document dans la base de données est inférieur à celui du document que l'on souhaite insérer.

use MongoDB\Collection;

/**
 * @param Collection $collection The MongoDB collection
 * @param int $id                The ID of the document to update
 * @param int $version           The version of the new document
 * @param array $document        The new document
 * @return bool                  True if the new version have been saved, false if it was outdated
 */
function optimisticUpsert(Collection $collection, int $id, int $version, array $document): bool
{
    // The document id and version are stored in the document itself
    $document['_id'] = $id;
    $document['_version'] = $version;

    $result = $collection->updateOne(
        ['_id' => $id],
        [
            ['$replaceWith' => [
                '$cond' => [
                    'if' => ['$lt' => ['$_version', $version]],
                    'then' => $document,
                    'else' => '$$ROOT',
                ],
            ]],
        ],
        ['upsert' => true],
    );

    return $result->getUpsertedCount() > 0 || $result->getModifiedCount() > 0;
}

Le serveur MongoDB est assez intelligent pour savoir qu'il n'y a pas de changement lorsque la version stockée était déjà plus récente que celle que l'on souhaitait insérer.

Si $result->getUpsertedCount() est supérieur à 0, cela signifie que le document n'existait pas et a été inséré. Si $result->getModifiedCount() est supérieur à 0, cela signifie que le document existait déjà et a été mis à jour. Sinon, cela signifie
que la version du document était déjà plus récente que celle que l'on souhaitait insérer.

Plus d'informations sur les opérations de mise à jour dans la documentation :

Expression $replaceWith.
Opérateur $cond.
Méthode updateOne.

PHP Closures and Generators can hold circular references

Jérôme TAMARELLE — Fri, 17 Jan 2025 21:34:11 +0000

Circular references are a common cause of memory leaks in PHP. They occur when objects reference each other, directly or indirectly. Thankfully, PHP has a garbage collector that can detect and clean up circular references. However, this consumes CPU cycles and can slow down your application.

The garbage collector is triggered whenever 10,000 possible cyclic objects or arrays are currently in memory, and one of them falls out of scope.

The garbage collection is never triggered if you have a small number of objects using a lot of memory. You can reach the memory limit even if the memory is used by orphaned objects that should be collected by the garbage collector.

That's why you should identify situations where you create circular references and avoid them.

In this article, we explore how closures and generators can hold circular references, and how to prevent them.

About circular references
- Typical example of a circular reference
- Preventing circular references with weak references
Closures and circular references
Generators and circular references
Conclusion

About circular references

Typical example of a circular reference

class A {
    public B $b;

    public function __construct()
    {
        $this->b = new B($this);
    }
}

class B {
    public function __construct(public A $a) {}
}

In this example, A and B reference each other. When you create an instance of A, it creates an instance of B that references A. This creates a circular reference.

To detect circular references, we can trigger the garbage collector manually using gc_collect_cycles() and read the number of collected references using gc_status().

// Object created but not assigned to a variable
new A();

gc_collect_cycles();
print_r(gc_status());

This will output:

Array
(
    ...
    [collected] => 2
    ...
)

This example shows that the garbage collector has detected and removed 2 objects with circular references.

You can also see the number of references to an object using the xdebug_debug_zval() function.

Preventing circular references with weak references

When confronted to a circular reference, an easy solution is to use a weak reference. A weak reference is an object that holds a reference that does not prevent the garbage collector from collecting the object it references. In PHP, you can create a weak reference using the WeakReference class.

This requires some changes to the code. The B class now stores a WeakReference object instead of an A object. You must access the A object using the get() method of the WeakReference object.

class A {
    public B $b;

    public function __construct()
    {
        $this->b = new B($this);
    }
}

class B {
    /** @var WeakReference<A> $a */
    public WeakReference $a;

    public function __construct(A $a)
    {
        $this->a = WeakReference::create($a);    
    }
}

// Object created but not assigned to a variable
new A();

gc_collect_cycles();
print_r(gc_status());
// [collected] => 0

In the output, you will see that the number of collected references is now 0.

Tip 1: Use weak references to prevent circular references only when necessary.

Closures and circular references

The concept of closures in PHP is to create a function that can access variables from the parent scope. This can lead to circular references if you are not careful.

function createCircularReference()
{
    $a = new stdClass();
    $a->b = function () use ($a) {
        return $a;
    };

    return $a;
}

In this example, the closure $a->b references the variable $a from the parent scope. The circular reference is easy to spot as the reference is explicit.

But the same issue can occur in a more subtle way if you use the short syntax for closures. Using an arrow function, the variable $a is not explicitly referenced in the closure, but it is still captured by reference.

function createCircularReference()
{
    $a = new stdClass();
    $a->b = fn() => $a;

    return $a;
}

createCircularReference();

gc_collect_cycles();
print_r(gc_status());
// [collected] => 2

In this example, the number of collected references is 2, indicating a circular reference.

Reference to `$this` in closures

Any non-static closure that is created within a class method will have a reference to the object instance ($this), even if $this is not accessed.

class A {
    public Closure $closure;
    public function __construct()
    {
        $this->closure = function () {};
    }
}

new A();

gc_collect_cycles();
print_r(gc_status());
// [collected] => 2

This is because the $this reference is always captured by reference in the closure. It can be accessed using Reflection::getClosureThis().

class A {
    public Closure $closure;
    public function __construct()
    {
        $this->closure = static function () {};
    }
}

new A();

gc_collect_cycles();
print_r(gc_status());
// [collected] => 0

If the closure is created from the global scope or inside a static method, the $this reference is null.

Tip 2: Always use static function () {} or static fn () => when creating a closure if you don't need $this.

Generators and circular references

We come to the reason for this article. Something I discovered recently:
Generators keep references, as long as they are not exhausted.

In this example, the class stores the generator in a property, but the generator has $this reference to the object instance.
The generator behaves like a closure and keeps a reference to the object instance.

class A
{
    public Iterator $iterator;

    public function __construct()
    {
        $this->iterator = $this->generator();
    }

    private function generator(): Generator
    {
        yield;
    }
}

new A();

gc_collect_cycles();
print_r(gc_status());
// [collected] => 1

The class instance was collected by the garbage collector because it has a reference to the generator, which has a reference to the object instance.

As soon as the generator is exhausted, the reference is released, and the object instance is removed from the memory.

iterator_to_array((new A())->iterator);

gc_collect_cycles();
print_r(gc_status());
// [collected] => 0

Tip 3: Ensure you always exhaust the generator by iterating over it.

Tip 4: Use static methods or closure to create generators to not keep references to the object instance.

Conclusion

Circular references are a common cause of memory leaks in PHP. Even if the garbage collector can detect and clean up circular references, it consumes CPU cycles and can slow down your application. You must detect situations where you create such circular reference and adapt your code to prevent them. Using weak references can prevent circular references, but some simple tips can help you to prevent circular references in the first place:

Use static function () {} or static fn () => when creating a closure if you don't need $this.
Ensure you always exhaust the generator by iterating over it.
Use static methods or closure to create generators to not keep references to the object instance.

Fix PHP 8.4 deprecation: Implicitly marking parameter as nullable is deprecated, the explicit nullable type must be used instead

Jérôme TAMARELLE — Mon, 29 Apr 2024 23:18:55 +0000

PHP 8.4 will be released in November 2024, but the list of new features and deprecated features is already very established. It is clearly detailed on php.watch/versions/8.4.

The most prominent change is the deprecation of implicit nullable parameter declarations:

function test(string $test = null) {}

test('PHP'); // Allowed
test(null); // Allowed

which is easy to fix using the explicit nullable type introduced in PHP 7.1:

- function test(string $test = null) {}
+ function test(?string $test = null) {}

You can detect this deprecation in your PHP files using PHP's built-in linter. This requires a local installation of PHP 8.4.

php -l src/**/*.php

Now that you know the affected files, you can fix them by yourself, or automatically using PHP-CS-Fixer or Rector.

Fix your codebase using PHP-CS-Fixer

PHP Coding Standards Fixer is a tool to automatically apply formatting rules on your code. In our case, we will use only the nullable_type_declaration_for_default_null_value rule.

1 - Install php-cs-fixer:

composer require --dev friendsofphp/php-cs-fixer

2 - Copy the minimal configuration in .php-cs-fixer.php, at the root of your project. We assume your source files are located in src but you can change it to set the path of all your PHP files.

<?php

$finder = PhpCsFixer\Finder::create()
    ->in([__DIR__ . '/src'])
    ->name('*.php');

return (new PhpCsFixer\Config())
    ->setFinder($finder)
    ->setRules([
        'nullable_type_declaration_for_default_null_value' => true,
    ]);

3 - Run php-cs-fixer

vendor/bin/php-cs-fixer fix

Note: For versions before v3.80.0, seting the environment variable PHP_CS_FIXER_IGNORE_ENV=1 is necessary to run on PHP 8.4.

Fix your codebase using Rector

Rector is an excellent tool for refactoring and modernizing code. In our case, we will only use the ExplicitNullableParamTypeRector rule.

1 - Install rector:

composer require --dev rector/rector

2 - Copy this minimal configuration in rector.php, at the root of your project. We assume your source files are located in src but you can change it to set the path of all your PHP files.

<?php

return Rector\Config\RectorConfig::configure()
    ->withPaths([__DIR__ . '/src'])
    ->withPhpVersion(Rector\ValueObject\PhpVersion::PHP_84)
    ->withRules([
         Rector\Php84\Rector\Param\ExplicitNullableParamTypeRector::class,
    ]);

3 - Run rector:

vendor/bin/rector

I hope this quick tutorial has saved you some time. There are hundreds of projects to patch on GitHub, have fun!

Spread the world

Deprecating the implicitly nullable parameter in #PHP 8.4 had a huge cost for the maintainers. Even if there are solutions to automate the fixes, that breaks almost all old code with very low value for developers. My article about fixing this is by far the most popular I have ever published.
— GromNaN (@jerome.tamarelle.net) 2025-04-04T08:00:03.746Z

How to overcome PHP's naming constraints to model MongoDB operators

Jérôme TAMARELLE — Mon, 16 Oct 2023 11:42:39 +0000

TL;TR: I research how to modelize MongoDB operators in PHP. From namespaced function names, to static method on classes or enums, to closures in variables. I'm looking for the best developer experience.

About MongoDB aggregation pipeline operators

As developer in the MongoDB PHP driver team, my goal is to provide the best developer experience to PHP developers using the document database.

MongoDB provides drivers for various languages including PHP. In order to ease create of aggregation pipelines in PHP, we need to modeling all the stages and operators as functions that can be combined.

An aggregation pipeline is a list of "stage" documents. We will take an example doing a query with $match and a join with $lookup:

db.orders.aggregate([
    {
        $match: {
            $or: [
                { status: "shipped" },
                { created_at: { $gte: ISODate("2023-01-01T00:00:00Z") } }
            ]
        }
    },
    {
        $lookup: {
            from: "inventory",
            localField: "product_id",
            foreignField: "product_id",
            as: "inventory_docs"
        }
    }
])

Each key with a dollar prefix is an operator for which we want to provide a factory method.

Namespaced functions

The most obvious solution is to create namespaced functions, like: MongoDB\Operator\eq for the $eq operator.

namespace MongoDB\Operator;

function eq(mixed $value): array {
    return ['$eq' => $value];
}

function lookup(string $from, string $localField, string $foreignField, string $as): array {
    return ['$lookup' => [
        'from' => $from,
        'localField' => $localField,
        'foreignField' => $foreignField,
        'as' => $as,
    ]];
}

Using this functions with named arguments, the pipeline would be written in PHP:

pipeline(
    match(
        or(
            query(status: eq('shipped')),
            query(date: gte(new UTCDateTime())),
        ),
    ),
    lookup(
        from: 'inventory',
        localField: 'product_id',
        foreignField: 'product_id',
        as: 'inventory_docs',
    ),
);

But, some operator names conflict with the reserved keywords in PHP. We cannot create functions (global or namespaced) with the following operator names: $and, $or, $match, and $unset.

Adding suffix to function names

To avoid this problem of reserved names, we can add a prefix or a suffix to the function names.

With the type of operator as suffix:

function andQuery(...) { /* ... */ }
function matchStage(...) { /* ... */ }

With an underscore:

function _and(...) { /* ... */ }
function _match(...) { /* ... */ }

Or with an emoji. Beautiful, but not practical:

function 🔍and(...) { /* ... */ }
function 💲match(...) { /* ... */ }

Static class methods

By chance, the list of reserved keywords is shorter for method names. We can create static methods on classes.

final class Stage {
    public static function lookup(...) { /* ... */ }
    public static function match(...) { /* ... */ }
}
final class Query {
    public static function and(...) { /* ... */ }
    public static function eq(...) { /* ... */ }
}

Writing got a little bit longer, but it's still readable.

new Pipeline(
    Stage::match(
        Query::or(
            Query::query(status: Query::eq('shipped')),
            Query::query(date: Query::gte(new UTCDateTime())),
        ),
    ),
    Stage::lookup(
        from: 'inventory',
        localField: 'product_id',
        foreignField: 'product_id',
        as: 'inventory_docs',
    ),
);

To prevent anyone from creating instances of this class, we can make the constructor private.

final class Operator {
    // ...
    // This constructor can't be called
    private function __construct() {}
}

We can also use an enum without case. Enum accepts static methods and cannot be instantiated.

enum Query {
    public static function and() { /* ... */ }
    public static function eq() { /* ... */ }
}

Both the class and the enum static method can be called the same way.

The downside with enum is that there is a static cases method that is always available and will pollute the API.

Closures in variables

By not finding the ideal solution, we start raving about improbable solutions.

An idea that comes if we want a short syntax that looks very similar to the MongoDB syntax, without name restrictions, is to use variables to store the closures. Note that (...) is a new syntax to create closures in PHP 8.1.

$eq = Operator::eq(...);
$and = Operator::and(...);

Don't trust the syntax hightlighter.

It's so wonderful that PHP uses the same dollar $ sign to prefix variables that MongoDB uses for operators.

pipeline(
    $match(
        $or(
            $query(status: $eq('shipped')),
            $query(date: $gte(new UTCDateTime())),
        ),
    ),
    $lookup(
        from: 'inventory',
        localField: 'product_id',
        foreignField: 'product_id',
        as: 'inventory_docs',
    ),
);

These closures could be made available by the library as an array.

enum Query {
    public static function and(array ...$queries) { /* ... */ }
    public static function eq(mixed $value) { /* ... */ }
    public static function query(mixed ...$query) { /* ... */ }

    /** @return array{and:callable,eq:callable,query:callable} */
    public static function functions(): array {
        return [
            'and' => self::and(...),
            'eq' => self::eq(...),
            'query' => self::query(...),
        ];
    }
}

The syntax to get all the variables is a little bit verbose, but it's still readable.

['and' => $and, 'eq' => $eq, 'query' => $query] = Query::functions();

We can import all the variables into the current scope with the magical extract function that is so much used in Laravel, but so much hated by PHPStorm and static analysis tools.

extract(Query::functions());

var_dump($and(
    $query(foo: $eq(5)),
    $query(bar: $eq(10))
));

// INFO: MixedFunctionCall - Cannot call function on mixed

See psalm analysis: https://psalm.dev/r/3bbbafd469

Conclusion

As you can see, naming functions in PHP isn't all that simple when it comes to using reserved keywords. We haven't yet decided how we're going to proceed. If you have any ideas or remarks, please leave a comment.

Please note that the code has been simplified, the functions will be more complex to offer more features and type safety.

Rendering Twig templates in Storybook

Jérôme TAMARELLE — Sat, 11 Mar 2023 21:54:59 +0000

Storybook is a frontend workshop for building UI components and pages in isolation. Thousands of teams use it for UI development, testing, and documentation. It’s open source and free.

Update: this is now easier to use Storybook with Twig Components. Read this article from Mathéo.

Historically, it requires a frontend framework like React, Vue or Angular to render components. But I build websites with Twig, so I needed a way to render the templates in my backend application. The package @storybook/server is a solution for this use-case. It was released in March 2022 and is exactly what its name suggests: a server-side rendering solution for Storybook.

There are very few things to do to connect Storybook to a Symfony application, this is a step-by-step guide to get you started. I'm assuming that you already have a Symfony application with Twig installed, and that you're using webpack encore. But that's not a requirement, you can use any other frontend tool you like.

Creating the first component

In the websites I build, a components is a single Twig file that can be included in any other template of the project. This can also be a Twig Component if you are using Symfony UX. For this example, we'll create a simple button component, in a templates/components folder. This button can have a label, a size and can be primary or secondary.

{# templates/components/button.html.twig #}

<button
    type="button"
    class="{{ size }} {{ primary|default(false) ? 'primary' : 'secondary' }}"
    style="{{ style|default('')|escape('html_attr') }}"
>
    {{- label -}}
</button>

Our goal is to provide a preview of this component in storybook, with the possibility to change its properties.

Creating the story

In StorybookJS, stories are individual states of components that can be displayed in isolation for preview and testing. They are typically written in JavaScript when rendering is done on the client-side with a javascript function. For server-side rendering, stories can be written in JSON. An additional key parameters.server.id is required to specify the path to the preview.

// stories/button.json
{
  "title": "Components/Button",
  "parameters": {
    "server": { "id": "button" }
  },
  "args": { "label": "Button" },
  "argTypes": {
    "label": { "control": "text" },
    "primary": { "control": "boolean" }, 
    "backgroundColor": { "control": "color" },
    "size": {
      "control": { "type": "select", "options": ["small", "medium", "large"] }
    }
  },
  "stories": [
    {
      "name": "Primary",
      "args": { "primary": true }
    },
    {
      "name": "Secondary"
    },
    {
      "name": "Large",
      "args": { "size": "large" }
    }
  ]
}

The Storybook will look like this:

Installing Storybook server

The @storybook/server documentation recommends using of npx to initialize the project with the server type:

npx sb init -t server

Symfony Webpack Encore uses Webpack5 while Storybook is uses Webpack4 by default, so we need to install the @storybook/builder-webpack5 package:

npm install @storybook/builder-webpack5

This will install npm packages and create a .storybook folder with main.js and preview.js.

The file .storybook/main.js defines the location of the stories and the addons to be used. We need to add the @storybook/server framework and the @storybook/builder-webpack5 builder because webpack encore uses webpack5.

// .storybook/main.js
module.exports = {
    "stories": [
        "stories/**/*.stories.mdx",
        "stories/**/*.stories.@(json)"
    ],
    "addons": [/* optional addons */],
    "framework": "@storybook/server",
    "core": {
        // Webpack Encore uses webpack5
        "builder": "@storybook/builder-webpack5"
    }
}

Last configuration step, we need to configure the URL where the application is served (by default symfony-cli serves on port 8000). The url is arbitrary, it can be anything you want, but it must match the route defined in the Symfony application. Storybook server will concatenate the url with the parameters.server.id value to get the full url to the component. It must be a valid absolute url.

// .storybook/preview.js
export const parameters = {
  server: {
    url: `https://localhost:8000/storybook/component`,
  },
};

Creating a generic Symfony controller

Back to the Symfony application, we need to create a controller behind the url defined in the Storybook configuration. It gets the id from the url and the args from the query string.

<?php # src/Controller/StorybookController.php

namespace App\Controller;

use Symfony\Component\HttpFoundation\Request;
use Symfony\Component\HttpFoundation\Response;
use Symfony\Component\HttpKernel\Attribute\AsController;
use Symfony\Component\Routing\Annotation\Route;
use Twig\Environment;

#[AsController]
readonly class StorybookController
{
    public function __construct(private Environment $twig) {}

    #[Route(
        '/storybook/component/{id}',
        // The id can contain slashes, so we need to use a regex
        requirements: ['id' => '.+'],
    )]
    public function __invoke(Request $request, string $id): Response
    {
        // $id is the path to the Twig template in the storybook/ directory
        // Args are read from the query parameters and sent to the template
        $template = sprintf('storybook/%s.html.twig', $id);
        $context = ['args' => $request->query->all(), 'id' => $id];
        $content = $this->twig->render($template, $context);

        // During development, storybook is served from a different port than the Symfony app
        // You can use nelmio/cors-bundle to set the Access-Control-Allow-Origin header correctly
        $headers = ['Access-Control-Allow-Origin' => 'http://localhost:6006'];

        return new Response($content, Response::HTTP_OK, $headers);
    }
}

Creating a Twig template to render the component

We have created our component in templates/components/button.html.twig and we configured the stories in stories/button.json. Now we need to create a Twig template that will render the component in the context of Storybook. This template will be used by the controller we created in the previous step.

{# templates/storybook/button.html.twig #}

{{ include('components/button.html.twig', {
    label: args.label|default('Button'),
    size: args.size|default('medium'),
    primary: args.primary|default(false) == 'true',
    style: args.backgroundColor is defined ? "background-color: #{args.backgroundColor}" : '',
}) }}

Running Storybook

Finally, we can run Storybook and see our component in action.

npm run storybook

> storybook
> start-storybook -p 6006
...
╭───────────────────────────────────────────────────╮
│                                                   │
│   Storybook 6.5 for Server started                │
│   4.56 s for preview                              │
│                                                   │
│    Local:            http://localhost:6006/       │
│    On your network:  http://192.168.1.11:6006/    │
│                                                   │
╰───────────────────────────────────────────────────╯

Time to play with the component and see how it behaves in different states.

Conclusion

This was my first experience with Storybook, and I'm very satisfied with the result. I hope this article helps you to get started and if you need to convince your frontend team that you don't need to switch to Node to use Storybook. Symfony is an amazing framework for building rich web applications and the need for a frontend framework to build the rich UI is being challenged by the Symfony UX initiative.

If you have any questions or feedback, feel free to comment below or reach out to me on Twitter.

Configure Symfony Secrets with Hashicorp Vault

Jérôme TAMARELLE — Thu, 20 Jan 2022 12:56:43 +0000

tl;dr: Integrating Vault and Symfony does not require any PHP code. Using vault-agent secrets variables can be dumped into .env file. Dynamic secrets can even be used as feature flags.

Prisma Media's websites and applications are mostly developed using Symfony framework. Many of them require secrets values: API keys, database credentials, private certificats… they needs to be treated carefully.

Why we choose Vault to store secrets

If you are there, you may know that storing secrets in your Git repository is a terrible practice that could lead to severe security issues.

A cryptographic "Vault" mecanism is proposed by Symfony. It uses encrypted secrets that can be spread in repositories and artifacts. This is a simple solution for basic needs. That has the advantage of being independent from any external system (simplicity, scalability).

We preferred HashiCorp Vault, a server solution that we deployed on our infrastructure. It is centralized, auditable and can handle dynamic secrets. This open-source product is not tied to a cloud provider. Easy to run locally for dev.

Vault works as a key-value store where secret variables are read and written using a REST API.

The need for frequent reload of secrets

Even if we generate the most secure password for the database, it will leak somewhere: logs, APM, error pages... To be safe, credentials needs to change, change all the time.

To get short-living secrets, Vault has a concept of Dynamic Secrets. Unlike key/value secrets where you had to put data into the store yourself, dynamic secrets are generated when they are accessed.

Not only we have to load the secrets from Vault, but also they have to be reloaded every time they change.

Dynamic config of Symfony with `.env`

// Detect dark theme var iframe = document.getElementById('tweet-1343842728547901441-25'); if (document.body.className.includes('dark-theme')) { iframe.src = "https://platform.twitter.com/embed/Tweet.html?id=1343842728547901441&theme=dark" }

Thank you Fabien, this is exactly what I need: changing the values without redeploying the application.

Unlike the YAML/PHP files in config/ directory, that are read only when the container cache is built, the .env file are read on every HTTP request. Updating this file while the application is running is a good way to update its configuration without impacting performance.

Vault Agent can write secrets into a file

vault-agent is a small utility that can be used as a cache proxy for the Vault server; or as a schedule to write secrets into a file using a template (every 5 minutes, configurable).

Example of vault-agent configuration for an app running on AWS EC2 instance.

# vault.conf
vault {
  address = "https://vault.example.com"
  retry {
    num_retries = 5
  }
}

auto_auth {
  method "aws" {
    config = {
      type = "iam"
      role = "<iam role>"
      region = "eu-west-1"
      header_value = "vault.example.com"
    }
  }
}

template {
  source = "./.env.local.ctmpl"
  destination = "./.env.local"
}

The template maps Vault secrets to environment variables. The templating syntax allows some flexibility, but it looks very primitive for a developer with Twig practice.

# .env.local.ctmpl
# This will generate a regular .env file
APP_ENV=prod

{{ with secret "secret/example/app" }}
APP_SECRET={{ .Data.data.APP_SECRET }}
{{ end }}

# Real environment variables can be read
{{ $env := (env "ENVIRONMENT") }}
{{ with secret (printf "secret/example/%s/database" $env) }}
DB_HOST={{ .Data.data.DB_HOST }}
DB_NAME={{ .Data.data.DB_NAME }}
DB_USER={{ .Data.data.DB_USER }}
DB_PASSWORD={{ .Data.data.DB_PASSWORD }}
{{ end }}

Finally, vault agent can be launched with any process manager (supervisord or systemd) or scheduler (crond).

# Daemon for prod server
vault agent -config=vault.conf

# Single run for testing
vault agent -config=vault.conf -exit-after-aut

The last security recommendation is to not write secrets on disks. We can create a tmpfs volume and symlink from the project root to that volume.

For kubernetes, vault-agent runs in a sidecar container that renders Vault secrets to a shared memory volume.

Use dynamic configuration feature flag

Feature flags are a benefit of using Vault and supporting dynamic configuration. Even if they are not secrets, flags can be stored in Vault. With fine tuned policies, product managers could manage feature flags and being rejected from other secrets.

Example of feature flag to render a block in Twig

In this example, we create a feature flag in Vault, with is a boolean to show or hide a "sales" block on a page.

Create a variable in Vault KV:

# secret/example/app/
{
  "FEATURE_FLAG_SALES": "true"
}

Read this secret in the vault agent template

# .env.local.ctmpl
{{ with secret "secret/example/app" }}
FEATURE_FLAG_SALES={{ .Data.data.FEATURE_FLAG_SALES }}
{{ end }}

Share the value of the variable with Twig context.

# app/config/packages/twig.yaml
twig:
  variables:
    feature_flag_sales: "%env(bool:FEATURE_FLAG_SALES)%"

Use the variable to render the block conditionally:

{% if feature_flag_sales %}
    <div>My conditional sales block</div>
{% endif %}

Run the vault-agent.
When the variable updated, the block is shown or hidden after few minutes.

DEV Community: Jérôme TAMARELLE

Andreas turned massive analytical tasks into lightning-fast reads using the Bucket Pattern on a giant MongoDB dataset.

How to Scale to a Billion Documents in Symfony

Rishabh Bisht for MongoDB ・ Jan 14

Searching in a database from image and text, made easy thanks to Symfony AI, MongoDB and Voyage AI.

Building an Archaeology Matcher: A (Literal) Deep Dive Into Multimodal Vector Search

[Boost]

Symfony and MongoDB: Our Commitment to a Stronger Integration

[Boost]

Better Aggregation Pipeline Support in the MongoDB PHP Driver

MongoDB Guests for MongoDB ・ Mar 17

MongoDB optimistic update with versioned documents

Mise à jour optimiste des documents versionnés dans MongoDB

PHP Closures and Generators can hold circular references

About circular references

Typical example of a circular reference

Preventing circular references with weak references

Closures and circular references

Reference to $this in closures

Generators and circular references

Conclusion

Read more

Fix PHP 8.4 deprecation: Implicitly marking parameter as nullable is deprecated, the explicit nullable type must be used instead

Fix your codebase using PHP-CS-Fixer

Fix your codebase using Rector

Spread the world

How to overcome PHP's naming constraints to model MongoDB operators

About MongoDB aggregation pipeline operators

Namespaced functions

Adding suffix to function names

Static class methods

Closures in variables

Conclusion

Rendering Twig templates in Storybook

Creating the first component

Creating the story

Installing Storybook server

Creating a generic Symfony controller

Creating a Twig template to render the component

Running Storybook

Conclusion

Configure Symfony Secrets with Hashicorp Vault

Why we choose Vault to store secrets

The need for frequent reload of secrets

Dynamic config of Symfony with .env

Vault Agent can write secrets into a file

Use dynamic configuration feature flag

Example of feature flag to render a block in Twig

Read more...

Reference to `$this` in closures

Dynamic config of Symfony with `.env`