DEV Community: TomvdPeet

From PHP to Xan to Rust: speeding up large CSV imports in Symfony

TomvdPeet — Thu, 24 Sep 2026 13:25:17 +0000

The goal of a price comparison website is obviously to compare prices across as many webshops as possible. So far Vindle has mainly integrated relatively smaller and more specialized webshops, offering one or a few product types, and each with no more than 40,000 total products.

Recently, however, we started integrating with Bol.com (one of the largest Dutch webshops). This was very exciting since Bol is a much more general webshop covering almost all product types, thus allowing an extra price to be compared for almost all products. But with a more general and larger scale webshop you also get many, many more products—roughly three orders of magnitude more, in fact. Bol.com offers over 73 million products, spread across 26 gzipped CSV feeds totaling roughly 27 GB.

As a price comparison website, there is something else that is very important: the prices shown need to be up to date. To achieve this, Vindle updates all product feeds once an hour, storing all the product feed entries in our database so that further operations can be done based on that stored data. Operations like updating the prices and availability of offers, or finding new offers for existing products.

This is where our journey starts. As you can imagine, suddenly increasing the scale of one merchant integration by roughly three orders of magnitude is going to require some optimizations. Now, since the new Bol.com integration is the main culprit behind this increase, our optimization efforts will be focused here. During this process, we will hopefully also learn some things that will help us optimize future feeds if needed (spoiler: we will—plenty)

Before we even get into parsing the CSV files there is an optimization we can already make, and one we made at the start. The indisputably cheapest and fastest way to parse CSVs is to not parse them at all. One nice thing Bol.com does, as I mentioned before, is split the 73 million products into 26 gzipped CSV files that are about 27 GB in total. The sizes of the individual files, however, range from around 1.5 MB to 4.8 GB, so there is quite a large size difference between them. But this still gives us two nice advantages: we only need to process the files we need, and we can include them in our hourly loop one at a time instead of all at once.

Where we started: PHP was fine

Vindle being a PHP/Symfony application, it made the most sense to just write the initial version in PHP.

The original implementation was about as exciting as you would expect: relying on fgetcsv(). A reasonably well-optimized standard PHP function for reading CSV.

Our first implementation was roughly like this:

merchant feed
    ↓
download and store the file
    ↓
gzip decompression stream
    ↓
PHP CSV reader
    ↓
filtering
    ↓
normalization
    ↓
database

Now normally in an article about optimizing a process of this kind, we would now take some time and talk about how obviously terrible this initial version was in hindsight. We would talk about how building something like this in PHP is super inefficient, the wasted I/O with all this disk writing and reading going on, and various other problems.

But we will not do this. In fact, I'm quite happy with this as an initial version. And here is why:

It was quick and easy to implement, so we got off the ground fast
It did exactly what it had to, generating the right data and keeping filtering statistics
There were no major speed problems at all when running the initial few files
We did not fall for premature optimization
The parts of this that we kept in PHP ended up remaining basically the same to this day

So why then does this article exist?

Because we needed to start parsing more and larger Bol.com files to make proper use of their wide product coverage. And this version simply didn't allow for that.

The shortcomings of PHP

The initial PHP-only implementation had one primary bottleneck, and it might not be what you would initially expect: the filtering step.

See, during the filtering step we get rid of the vast majority of products for one of the following reasons:

There is no valid price
It does not ship to the Netherlands
It is not in a new condition
It has no or an invalid brand
Or it does not fall under one of our configured allowed categories

Now this last one is the most important since it ends up filtering out most products. As I mentioned, the feed is split up in multiple files but each file contains multiple sub-categories. For example, the digital-reading file contains both digital books as well as e-readers. But since Vindle does not currently offer digital books we just get rid of all of them and only keep the explicitly allowed sub-categories, in this case e-readers.

Because of all this filtering, depending on the file we keep anywhere from about 9% down to only 0.0023% of the products. The most extreme example is a file containing ~7.2 million products, of which we keep about ~170.

Doing all this filtering and getting rid of the majority of the rows was quickly becoming expensive in PHP, in part because of the fgetcsv() function. Fortunately, there was quite a clean seam in the application: we could move this simple pre-processing to an external program and keep the more integrated work in PHP.

First optimization: Xan

Introducing Xan, a command-line CSV processing tool written in Rust. Xan was (almost) exactly the tool that we needed for the job: it allows for basic conditional filtering and selecting specific columns. And it works directly with compressed CSVs.

Symfony made it easy to keep the application logic where it was, while delegating the expensive pre-processing step to an external tool.

Equipped with Xan our pipeline became roughly:

download and store the file
    ↓
Xan
├── filter unwanted rows
└── select required columns
    ↓ (stream)
PHP
    ↓
normalization
    ↓
database

This created a significant speed-up without introducing much complexity. We simply introduced the pre-filtering using Xan and then streaming those filtered rows directly into PHP.

If you are reading this article because you're dealing with a comparable problem, I would highly recommend this sort of solution using Xan. This avoids having to write things in Rust and keeps everything quite contained.

But our pre-processing had one additional requirement. We don't only want the filtered rows. We also keep statistics about why rows were filtered.

For example, after processing a feed it is useful to know how many source rows were rejected because they had no valid offer, because they were not deliverable, because the brand was missing, or because they did not belong to a relevant product group.

Sadly, this means that this relatively simple solution using Xan was not enough.

The hidden cost of multiple passes

Now how does one solve this issue? It's simple, just use more Xan.

So, to circumvent this problem of not being able to retrieve the filtering statistics from Xan directly, we implemented an extra pre-pass that resolves them separately.

Simplified, the flow was:

feed
 ├── Xan pass 1 → exclusion statistics
 └── Xan pass 2 → filtered rows → PHP

This worked, but it also felt like we took one step forward and one step backward, since we were now running Xan twice per feed. Though this was still considerably faster than using plain PHP, this was not going to allow us to comfortably go through all the files we'd need to every hour. This extra pass cost us roughly 20–40 seconds per feed.

From here, a new idea was born: run both passes in parallel. The idea was simple: we centrally read and decompress the feed and then use tee to pipe the result into our two separate Xan processes.

Something like:

               ┌→ Xan statistics
gzip → tee ────┤
               └→ Xan filter → PHP

The main improvements here were:

removing one disk read cycle
removing the need to decompress twice

Parallelization doesn't remove work

This idea looked attractive on paper, but in practice it introduced a new problem: backpressure.

Our two Xan branches did not do equal work. The statistics branch only had to count results, while the filtering branch also had to filter rows, stream them into PHP, and wait for PHP to continue processing them. Because Unix pipes have bounded buffers, the slower branch would eventually fill its pipe. Once that happened, tee blocked, which blocked the decompressor, which in turn blocked the upstream producer.

So while the pipeline looked more parallel, its throughput was still determined by the slowest consumer.

On top of that, both branches still had to fully parse the CSV. So although the pipeline removed one disk read and one decompression pass, it still duplicated the expensive row-by-row parsing work.

There was also a bonus problem: implementing clean error handling for this type of split process is much more complicated.

So our supposedly more efficient streaming pipeline now had:

two CSV parser processes;
duplicated filtering work;
additional pipes;
difficult error handling;
more process scheduling;
more context switching;
and a slowest-consumer bottleneck.

And if all of that was not enough, the performance was also lacking. In general, this implementation was not faster than the serial one. In fact it leaned more toward being a few percent slower, getting worse with larger feeds.

So at this point we realized that this type of pipe contraption would probably not get us to where we wanted. What we needed was much simpler.

One row, one decision

For each row, we already knew exactly what needed to happen:

read row
    ↓
check exclusion rules
    ↓
if excluded:
    update statistics
otherwise:
    write selected columns

The statistics and filtering were not fundamentally incompatible operations. In fact, initially in PHP they were done in the same pass; we were just forced to separate them by the limitations of command-line tools.

Basically, what we needed was:

An external program more optimized for these types of operations than PHP
The ability to collect statistics and filter at the same time so we only had to parse each CSV row once

This is when we started to experiment with Rust. In fairness, we considered Rust as an option from the very beginning but we wanted to see what was possible with off-the-shelf tools. Adding Rust to a project like this is not without its own downsides.

The goal here was now very narrow: write this hot loop of filtering and collecting statistics into a small native Rust program. That's all. The only other requirement was that it must allow streaming of input and output data through STDIN and STDOUT. Lastly, it must be as dumb as possible, depending on as little context and business logic as we can make it.

Streaming here is also of particular importance, since at this stage we'd made another realization. The current implementation had a non-deliberate dependency: empty disk space. When dealing with a 5 GB file our current implementation would have to first download that whole file to disk, so the machine doing this process would need 5+ GB of free disk space. Another thing is the fact that writing 5 GB to disk to then read it once and delete it also seemed wasteful. For these reasons we wanted the next iterations to directly stream the incoming download into our preprocessor instead of downloading the entire file to disk. We wanted to make the process diskless.

The boring Rust program

There are a few reasons we chose Rust here, as opposed to another low-level language. But one of the reasons is that Xan is written in Rust and uses a well-optimized CSV library called simd-csv. In fact, simd-csv was designed around Xan's requirements, so we can use it as our foundation and build our relatively simple filtering and statistics collection on top of it.

Its original contract was essentially:

stdin
  ↓
Rust
├── parse CSV
├── apply source filters
├── count exclusions
└── project required columns
  ↓
stdout: filtered CSV

stderr: statistics

The existing PHP ingestion logic could continue consuming the resulting CSV exactly as it did before. There, it could perform the operations that are more integrated with our business logic. That compatibility was very useful: we were replacing one expensive stage of the pipeline rather than redesigning everything around it.

The processing loop itself is super simple:

for each record {
    statistics.total_rows += 1;

    if let Some(reason) = exclusion_reason(record) {
        statistics.exclude(reason);
        continue;
    }

    statistics.retained_rows += 1;
    write_projected_record(record);
}

Yet this achieved the same thing as two passes through Xan and a bunch of piping, at a faster rate. In other words, we had roughly regained the speed of the original single-pass Xan setup, while also getting the filtering statistics that had previously required an additional pass.

Symfony remained in charge

What I like about this architecture is that Symfony never stopped being the core of the application.

Rust only took over the narrow part of the pipeline that consisted of reading a lot of data and performing cheap, deterministic checks. Everything that depends on application context still lives in Symfony: managing and downloading feeds, credentials, domain logic, normalization, persistence and the rest of the actual ingestion workflow.

That boundary ended up being quite useful. We did not need to redesign the application around Rust; Symfony could simply orchestrate another specialized component, while the existing PHP code continued handling the parts it was already well suited for.

We could have continued moving all of that functionality into Rust. This would probably have been even faster in the end, assuming we were not limited by internet speed. But we chose not to.

Why? Because it was simply not worth it. The potential improvement would not have outweighed the extra work and maintenance debt this would have created, especially the latter.

The Rust program was implemented as a simple CLI. We chose that over FFI because FFI would have been more work. In fact, running this as a separate operating-system process gave us isolation, and a wonderfully boring interface.

There are some interesting complications around managing those streams safely from PHP, but they deserve an article of their own.

The next frontier

After we got the Rust preprocessor up and running we ran into a new bottleneck.

The setup with the Rust process was now roughly like this:

compressed stream
    ↓
gzip -dc
    ↓
Rust
    ↓
PHP

Compared to the previous iterations, this was already a much cleaner architecture: no more downloading the whole file to disk, no more multiple passes and no more parallel processes.

But now the processing of the CSV became fast enough to expose a new problem.

Gzip. It was often consuming an entire CPU core while the other parts of this pipeline were using much less.

In practice this meant the Rust code was waiting for more data and therefore even our PHP code was waiting. The external decompressing process was now determining how quickly data would arrive.

This is one of those things that shows how performance optimization often goes:

A is slow
↓
optimize A
↓
B is now slow
↓
optimize B
↓
C becomes visible

So it's not that the decompression suddenly became slower, the opposite happened. The rest of the system was now fast enough that our current decompression method could not keep up.

Moving decompression into Rust

One nice thing about gzip decompression is that there are a ton of libraries for it. So now it was just about finding the best one for this job.

I won't go into the benchmarking process here, but eventually we arrived at the Rust flate2 crate, and found that it performed substantially better for this workload. Something I only realized later is that Xan uses the same library for compressed input handling.

We deliberately hadn't included gzip decompression in the first Rust implementation. gzip -dc already worked, and until it started limiting throughput there was no reason to replace it. But at this point it made a lot of sense to also move the decompression to Rust, instead of a separate process.

Internally, the interesting part is very small. The gzip decoder implements Rust's Read interface, and our feed processor was already generic over anything implementing Read.

That left us with the final hot path:

Symfony/PHP
└── Bol.com
    ↓
compressed network stream
    ↓
Rust
├── gzip decompression
├── CSV parsing
├── filtering
├── statistics
└── column projection
    ↓
filtered CSV
    ↓
Symfony/PHP
├── validation
├── normalization
├── affiliate logic
└── persistence

No feed-sized temporary source file is required for the normal ingestion path.

No second CSV parsing pass is required for statistics.

And PHP only receives the rows that survived the cheap source-level filtering.

What changed in practice?

The result of these optimizations has been a material improvement to our hourly ingestion cycle, ultimately allowing us to go through all the Bol.com feeds we need without causing any problems or delays.

Test	Before	After	Improvement
Mobile feed – PHP → Xan (1.73M rows)	357.3 s	17.7–18.5 s	~95% less time / ~20× faster
Computer feed – external gzip → Rust gzip	24.10 s	9.48 s	60.7% less time / 2.54× faster
Computer – full production snapshot	60.6 s	39.2 s	35.3% faster
Daily-care – full production snapshot	64.3 s	53.6 s	16.6% faster

The first two rows are controlled processing benchmarks, while the last two are production snapshot observations on live inputs.

The main gains here are not in isolated microbenchmarks, but in the end-to-end ingestion process. Across representative feeds and production snapshots, the improvements ranged from meaningful to dramatic, and together they gave us enough headroom to process the Bol.com feeds we needed comfortably within our hourly update cycle.

And there is another practical benefit, enabled in part by making this process diskless. The larger feeds, which previously were unattractive to support, are now much less problematic because we no longer have to worry about their disk footprint. And even though we still have to download and parse every byte of them, the speed improvements have enabled us to support them hourly.

What we learned from the process

Should we start rewriting all CSV processing in Rust? No.

In fact, I think that for most Symfony developers, moving processing to an external system or writing a process in another language for a speed-up should not be done lightly, and definitely not straight away. It also creates its own overhead.

The progression is the main takeaway here:

PHP was the right implementation at the beginning, and large parts of that code still remain
Xan was the right move when we realized pre-processing was becoming expensive
Rust became attractive when our processing requirements became sufficiently application-specific that general-purpose tools were not effectively able to do what we needed.

I think this progression also maps quite well to a rough decision tree for dealing with performance problems in PHP:

PHP performance problem
        ↓
Isolate the hot path
        ↓
Can PHP handle it with reasonable optimization?
      ↓                         ↓
    yes                         no
    ↓                           ↓
Stay in PHP             Is there a good external tool?
                              ↓              ↓
                            yes             no
                            ↓               ↓
                     Use the tool      Write a targeted low-level component

Removing disk I/O does not automatically make a pipeline faster

Streaming in this process had some important benefits, mainly that we no longer needed to fully download these large files. But if the machine is CPU-bound, replacing disk access with more simultaneous CPU work can actually make throughput worse.

Measure the pipeline rather than assuming that “streaming” or “parallel” means “faster.”

Parallel work is still work

Our tee experiment looked elegant:

one decompression
two parallel consumers

But both consumers still had to parse every row, parallelizing duplicate work doesn't magically make it less expensive. On a small VPS with limited CPU capacity, that distinction matters a lot.

Backpressure is useful

In a streaming architecture, a blocking or waiting step can seem like something is going wrong. But in reality it is what makes the architecture work, by keeping it bounded. If any part of the chain slows down, the section before it just waits once its buffer fills up. Memory usage remains controlled because each stage naturally slows the stage before it when needed.

Optimize the operation, not the language

The biggest structural improvement came from turning:

parse for statistics
+
parse for filtering

into:

parse once
→ statistics + filtering

Rust made implementing that efficient loop straightforward, but changing the algorithm mattered just as much as changing the runtime.

The best language boundary can be a pipe

We initially considered whether integrating Rust with PHP should be done through FFI. However, for this workload and type of problem that would have added unnecessary complexity. Since we're not calling a tiny Rust function thousands of times, we have a single long-running native process parsing a large stream of data.

Because of this, running as a subprocess gives us a clean boundary, natural streaming, crash isolation and almost no coupling between the two ecosystems.

Sometimes stdin, stdout and an exit code are all the integration layer you need.

Where it leaves us

The final architecture is more complicated than the original implementation in PHP.

But the complexity is narrowly contained.

Symfony still behaves like a Symfony application.

Rust behaves like a fast Unix-style preprocessing tool.

And the interface between them consists primarily of streams.

The result is a feed ingestion pipeline that can deal much more comfortably with the scale we're starting to encounter, without requiring the rest of the application to care how those source rows were reduced.

And multiple new possibilities have opened up to us. The Rust component could eventually become a reusable core for other heavy data-processing operations. Other feeds could move onto the same preprocessing infrastructure if it's shown that it is worthwhile. And we have the option to eventually implement FFI functions if they're justified.

But for now, the most useful result is also the simplest one:

We found one expensive loop in a Symfony application, moved exactly that loop somewhere better suited to running it, and left everything else alone.

Originally published on Vindle.

Getting Agent-Ready with Symfony

TomvdPeet — Thu, 28 May 2026 13:26:23 +0000

The ongoing large scale adoption of AI is rapidly changing the way many people interact with the web. Therefore websites and web-apps must adapt so they can continue to thrive. In this article i will go over some improvements you can make to your Symfony application to start making it agent-ready.

Since i want to cover multiple things in this article i will not be going into great detail on each of them, so consider this article more as an introduction to agent-ready-ness and these concepts. I did however write a small bundle for one of the concepts and for the last one I wrote a basic implementation prompt which can be found in the last chapter.

A great resource that will likely help you on this journey and going forward is isitagentready.com. This tool gives a good overview of the improvements you can make to your site and is in part what this article is based on.

SEO Fundamentals

Before getting into any agent specific improvements, it is best we first glance over some SEO fundamentals.

The reason for this is that most things we consider SEO improvements are generally machine- readability and discoverability improvements, and it just so happens that agents are machines too. That is why i would suggest having your SEO fundamentals on point first, since they will not only help with SEO but they're a nice baseline from which to start getting agent-ready

Now, there's a good chance you already have (most) these in place but if you don't i'd highly recommend you look them first before focusing on agents. I will not go into detail for these, if you do want more information on them you can find plenty online, or ask your agent.

Security make sure your site is secure and served over https
Pages must be crawlable requests should return the correct status codes and not be noindex
robots.txt You should have this setup including the sitemap direcive
Sitemap A proper XML sitemap that contains all the links you want to be crawled / indexed.
Inform indexers Setup google search console and Bing webmaster tools and submit your sitemap. Also check these tools occasionally for any indexing warnings
Canonical URLs make sure each page has a specified canonical URL using rel="canonical"
Proper titles and descriptions Make sure each page has proper title's and descriptions
Good semantic HTML using elements like: nav, main, footer,h1 etc. Where appropriate
Structured data implementing structured data where it fits things like: BreadcrumbList, Product, FAQPage
Fast, mobile-friendly and accessible your site should be fast, accessible and work on all devices.

Content Signals

Starting with content signals this is the simplest improvement you can make. content-signal is a proposed robots.txt directive, meant to communicate what you permit AI to do with content from your site. A simple example:

User-Agent: *
Content-Signal: ai-train=no, search=yes, ai-input=yes
Allow: /

What each of the categories mean:

ai-train Your content may be used for training or fine-tuning AI models
search Your content may be used for building a search index and providing search results. This is more so related to traditional search indexes
ai-input Your content may be used as input for AI models. This mainly means things like AI web search tools and does not include training

Generally you'll want search and ai-input to be yes for the purpose of indexing in search and for use by AI through web search tools. I would consider these two standard for most websites, even if you don't want AI to train on your content.

ai-train is more personal and project dependent. The obvious con is that AI may copy your idea's, content or style. But in the case your site is more commercial and features your brand name in the text having AI train on it could be a positive.

It is also possible to apply specific content signals for specific URLs, more information on this is best found on contentsignals.org

Markdown Negotiation

Markdown negotiation is the ability for agents to request your pages as text/markdown instead of HTML, using the Accept: text/markdown request header.

The reason we'd want to implement this is that agents speak Markdown very well, far beter than HTML and it is also much more token efficient. In a lot of cases the HTML you would otherwise return gets turned into Markdown anyway before an AI sees it. Therefore if you create the Markdown yourself you have a lot more control over it.

There are two general ways to implement this:

Explicit markdown responses: checking in each route if the request prefers markdown and returning an appropriate response.
Centralized HTML-to-Markdown conversion: turning the HTML returned by routes into Markdown centrally.

These two complement each other well, generally for the most control you'll want explicit Markdown responses and then fall back to passive HTML-to-Markdown conversion for pages that are less important.

Initially in this article i was going to give a high level overview of how to implement both of these options. Instead I decided that this would be far better to just turn into a small bundle, so i present: tomvdpeet/markdown-negotiation-bundle. The bundle is mainly focused on passive HTML-to-markdown conversion, here is an example:

#[Route('/docs', name: 'docs', options: ['markdown' => true])]
public function docs(): Response
{
    return $this->render('docs/index.html.twig');
}

This example will turn the rendered html in to clean Markdown when requested, simply by adding options: ['markdown' => true] to the Route attribute. The resulting Markdown will also be stripped of stuff like footer, nav, head and comparable layout/decorative HTML content.

The bundle also has some other features like supporting a dev-only ?_markdown GET parameter that will handle the request as if it prefers Markdown for easier debugging. And some tools for explicitly specified Markdown responses as well. Though the main focus is a DX-friendly and minimal overhead implementation of the centralized conversion.

API Exposure

The previous two changes we looked at were purely about machine- readability and discoverability, API Exposure on the other hand opens the door to machine-usability. So before going into API Exposure itself I want to touch on the change in perspective this goes hand in hand with.

Most websites, web-apps, platforms, etc. are user facing and support a user interface, UI. This makes sense since humans interact with them, right now. However for our systems to be agent-ready, we will also need to make them agent-usable and thus machine-usable. This change requires us to look at our systems increasingly more as an API based system rather then just a UI based system.

I believe API Exposure is a nice entry-point into making a Symfony application more agent-ready and machine-usable, since it is the way an agents discover the capabilities and API's your application provides.

API Exposure (RFC 9727) defines a standard URL, API catalog (/.well-known/api-catalog) that lists the available API's and documentation for them in a standard JSON based format.

Since i didn't want to add a full implementation tutorial in this article i did write a prompt to have an agent implement a basic setup of this for you. To help you get started with this the prompt will also setup OpenApi docs for your search page, if you have one. It does rely on the nelmio/api-doc-bundle to do this.

As mentioned API exposure is about making your API's and functionalities discoverable. I am not going into further detail on this because it it very application depended. But you should now have a stable and expandable foundation for making any API's and functionalities you add discoverable.

Conclusion

The shift to a more agentic internet will mean that we need to shift the perception of our own applications to be more and more API like. Since all of this is still very new, standards and conventions are still uncertain and changing quickly. However i believe that what I've covered here is the right direction and what internet will be moving to sooner or later. So getting a head start on implementing or at least learning about these standards will help you and your application continue to thrive in the .

original article: https://vindle.nl/insights/getting-agent-ready-with-symfony

Prompts

Api exposure prompt

Make this Symfony app more agent-ready by adding RFC 8288 Link response headers, an RFC 9727 API catalog, and OpenAPI docs with NelmioApiDocBundle.

Use the smallest pragmatic change that fits the existing codebase. Read the existing routing/controller structure first and preserve current behavior.

Tasks:

1. Install and enable NelmioApiDocBundle
- If not already installed, run:
  composer require nelmio/api-doc-bundle
- Verify the bundle is registered in config/bundles.php.
- Add or update:
  config/packages/nelmio_api_doc.yaml
  config/routes/nelmio_api_doc.yaml

2. Expose OpenAPI docs
- Add JSON docs at:
  /api/doc.json
  using nelmio_api_doc.controller.swagger
- Add Swagger UI at:
  /api/doc
  using nelmio_api_doc.controller.swagger_ui
- Configure Nelmio metadata with the project name, description, version, and server URL.
- Keep operation details on controller route methods using OpenAPI attributes, not hard-coded YAML paths.

3. Document the search page if the project has one
- Find the existing public search route, for example /search or /zoeken.
- Add OpenAPI route-level attributes to that controller method:
  - tag: Search
  - query parameter: q, optional string
  - 200 response with text/html
  - if the app supports markdown content negotiation, also add text/markdown
- Update nelmio_api_doc.areas.path_patterns so the search route is included in generated docs, while excluding /api/doc itself.

Example attributes:

use OpenApi\Attributes as OA;

#[OA\Tag(name: 'Search')]
#[OA\Parameter(
    name: 'q',
    description: 'Search term.',
    in: 'query',
    required: false,
    schema: new OA\Schema(type: 'string'),
)]
#[OA\Response(
    response: 200,
    description: 'Search results page.',
    content: [
        new OA\MediaType(
            mediaType: 'text/html',
            schema: new OA\Schema(type: 'string'),
        ),
        new OA\MediaType(
            mediaType: 'text/markdown',
            schema: new OA\Schema(type: 'string'),
        ),
    ],
)]

4. Add an RFC 9727 API catalog
- Add a route:
  /.well-known/api-catalog
- Return application/linkset+json.
- Include links to:
  - the OpenAPI JSON document as service-desc
  - the Swagger UI as service-doc
  - the public search route as item/search-related discovery if available
- Add a Link response header on the catalog response:
  Link: </.well-known/api-catalog>; rel="api-catalog"; type="application/linkset+json"

Example linkset shape:

{
  "linkset": [
    {
      "anchor": "https://example.com/.well-known/api-catalog",
      "item": [
        {
          "href": "https://example.com/search{?q}",
          "type": "text/html",
          "title": "Product search"
        }
      ],
      "service-desc": [
        {
          "href": "https://example.com/api/doc.json",
          "type": "application/vnd.oai.openapi+json",
          "title": "OpenAPI description"
        }
      ],
      "service-doc": [
        {
          "href": "https://example.com/api/doc",
          "type": "text/html",
          "title": "API documentation"
        }
      ]
    }
  ]
}

5. Add homepage Link discovery headers
- Add a Link response header to the homepage response.
- Prefer putting this directly in the homepage controller if it is homepage-only.
- Use one combined RFC 8288 Link header value.

Example:

Link: </.well-known/api-catalog>; rel="api-catalog"; type="application/linkset+json", </search>; rel="search"; type="text/html"; title="Site search", </api/doc.json>; rel="service-desc"; type="application/vnd.oai.openapi+json"; title="OpenAPI description", </api/doc>; rel="service-doc"; type="text/html"; title="API documentation"

6. Validate
Run the relevant checks for the project, at minimum:
- php bin/console lint:yaml config/packages/nelmio_api_doc.yaml config/routes/nelmio_api_doc.yaml
- php bin/console debug:router
- php bin/console lint:container
- php bin/console nelmio:apidoc:dump --format=json
- project tests, or targeted tests for the new API catalog/controller behavior

7. Documentation
- Update project docs to mention:
  - /api/doc.json
  - /api/doc
  - /.well-known/api-catalog
  - homepage Link headers
  - OpenAPI operation details live as route-level attributes

Symfony as JWT provider

TomvdPeet — Sat, 30 Aug 2025 12:11:48 +0000

In the process of writing this guide Ryan Weaver passed away.
This news was truly shocking to me and my thoughts are with his family and loved ones. I hope this guide can do some justice to his spirit.

So, in my previous article I talked about why I decided to switch from Auth0 to Symfony as my authentication provider. In that article I also promised to write a follow up guide on how I actually implemented it.

For some context this guide is not about authenticating to Symfony with JWT rather it is about using Symfony to authenticate users to other systems. The setup used in this guide is quite simple.

The user interacts through the browser extension.
These interactions (e.g. LLM streaming and such) go through Node.js.
Symfony is setup as a back-end API for handling data.

Now Symfony also houses the user and authentication system and that flow goes as follows:

The user logs into the Symfony application
The extension requests a JWT and Refresh token (RT)
The extension calls to Node.js with the JWT and Node.js validates it
When the JWT expires the RT is exchanged for a new RT and JWT

In this guide I will only go over the Symfony side of this system. so it will require you to implement the other areas yourself, however for most of these you can follow Auth0's guides since its basically the same set up.

Before we begin

This is a bit more of an advanced guide and its quite loose in the sense I expect you to apply it to your situation, so blindly copy and pasting this code and commands will not solve your problem.

I also presume you have / know the following:

An up and running symfony application with users and login
Some knowledge of how JWT works and what it is
Enough knowledge of security to apply this in a safe way

Configuration

To start something to mention is that we will be using web-token/jwt-bundle, all the classes in this bundle are in the Jose\ namespace, and i'm not sure why, I was also confused. I will however refer to this bundle as Jose since its shorter.

To begin we must install the aforementioned bundle:

composer require web-token/jwt-bundle web-token/jwt-signature-algorithm-rsa

This Jose bundle is a framework for working with JWT and its the foundation for this setup.

Now that we have installed the bundle we will begin by configuring it, and to do this we must generate the keys. The keys will live in the config/jwt/ folder but you can change this if you want.

Note: you should not commit these key's to git

Run the following commands:

mkdir config/jwt
php bin/console keyset:generate:rsa --random_id --use=sig --alg=RS256 3 4096 > config/jwt/signature.jwkset
php bin/console key:generate:rsa 2048 --use=sig --alg=RS256 --random_id > config/jwt/signature.jwk
php bin/console keyset:rotate `cat config/jwt/signature.jwkset` `cat config/jwt/signature.jwk` > config/jwt/signature.jwkset
php bin/console keyset:convert:public `cat config/jwt/signature.jwkset` > config/jwt/public.jwkset

Note:
If you have read the docs, they use ./jose.phar however the commands should just start with php bin/console this goes for all jose commands. So no need to mess with ./jose.phar

I will quickly go over what each of these commands does, to give some context.

creates the config/jwt folder
generates a private keyset of 3 keys (none of which we'll use)
generates a single private key that we will use for signing our JWT's
rotates our newly generated real key into the key set
creates a public keyset that other systems will use to verify our signed JWT's

Now one nice thing about this JWT setup is that we can change our private key without breaking everything. so if its compromised you simply run the last 3 commands and the whole system will use a new private key, while the existing signed JWT's are still valid since their public keys remain in the public key set.

To finish off the configuration of Jose we must set up the config file if you followed the steps so far you can simply copy and paste it.

# /config/packages/jose.yaml
parameters:  
    env(JWT_PRIV): '%kernel.project_dir%/config/jwt/signature.jwk'  
    env(JWTSET_PRIV): '%kernel.project_dir%/config/jwt/signature.jwkset'  
    env(JWTSET_PUB): '%kernel.project_dir%/config/jwt/public.jwkset'  

jose:  
    keys:  
        user_sig_priv:  
            jwk:  
                value: '%env(file:JWT_PRIV)%'  
                is_public: false  

    key_sets:  
        user_sig_priv:  
            jwkset:  
                value: '%env(file:JWTSET_PRIV)%'  
                is_public: false  
        user_sig_pub:  
            jwkset:  
                value: '%env(file:JWTSET_PUB)%'  
                is_public: false  

    jws:  
        builders:  
            builder:  
                signature_algorithms: [ 'RS256' ]  
                is_public: true  

        verifiers:  
            verifier:  
                signature_algorithms: [ 'RS256' ]  
                is_public: true

All this config basically does is allow us to use the keys we just created more easily within our Symfony project. it also sets up a builder and a verifier service which we'll use later.
After doing this the JWT icon should appear in the debug toolbar (on any page) and it will show 1 key and 2 keysets.

Refresh tokens

Now this guide includes the use of refresh tokens (RT). These are tokens stored in the database that the user can exchange for a new JWT and RT. The reason we want this is so we can keep the TTL of the refresh token very short. which has various advantages (which I won't list), and as long as the RT doesn't expire or is removed from the db the user remains logged in.

To store the RT in the database we obviously need to create an entity, here is a stub of the one I use:

    # src/Entity/RefreshToken.php

use App\Repository\RefreshTokenRepository;  
use Doctrine\ORM\Mapping as ORM;

#[ORM\Entity(repositoryClass: RefreshTokenRepository::class)]  
class RefreshToken  
{  
    #[ORM\Id]  
    #[ORM\GeneratedValue]  
    #[ORM\Column]  
    private ?int $id = null;  

    #[ORM\Column(length: 255)]  
    private ?string $token = null;  

    #[ORM\Column]  
    private ?\DateTime $expires = null; 

    #[ORM\ManyToOne(inversedBy: 'refreshTokens')]  
    #[ORM\JoinColumn(nullable: false)]  
    private ?User $user = null;  

    #[ORM\Column]  
    private ?\DateTime $issued = null;

    //GETTERS AND SETTERS NOT INCLUDED
}

If you want to you can change this entity, add whatever fields you'd like. You can change the $user property to something else so that the token doesn't have to be associated with a User object . this offers more options but also more complications. Also the $issued property is technically not needed, but I like having it.

Generating JWT's

Now with all that setup out of the way I introduce to you the JWTService. This service will do all that you need for most JWT setups:

# src/Service/JWTService.php 

namespace App\Service;

use App\Entity\RefreshToken;  
use App\Entity\User;  
use Jose\Component\Core\JWK;  
use Jose\Component\Core\JWKSet;  
use Jose\Component\Signature\JWS;  
use Jose\Component\Signature\JWSBuilder;  
use Jose\Component\Signature\JWSVerifier;  
use Jose\Component\Signature\Serializer\CompactSerializer;  

class JWTService  
{  
    private const ALG = 'RS256';  
    private readonly JWSBuilder $builder;  
    private readonly JWSVerifier $verifier;  
    private readonly CompactSerializer $serializer;  

    public function __construct(  
        private readonly JWK    $userSigPrivKey,  
        private readonly JWKSet $userSigPubKeySet,  
        JWSBuilder              $builderJwsBuilder,  
        JWSVerifier             $verifierJwsVerifier  
    )  
    {  
        $this->builder = $builderJwsBuilder;  
        $this->verifier = $verifierJwsVerifier;  
        $this->serializer = new CompactSerializer();  
    }  

    public function generateJWS(User $user): JWS  
    {  
        $claims = [  
            'sub' => $user->getEmail(),  
            'data' => [  
                //This array is where you can store some signed information about the user 
                'email' => $user->getEmail(),  
            ],  
            'iat' => time(),  
            'exp' => time() + 3600 //This decides the JWT TTL,  
        ];  

        $jws = $this->builder->create()  
            ->withPayload(json_encode($claims))  
            ->addSignature($this->userSigPrivKey, [  
                'alg' => self::ALG,  
                'kid' => $this->userSigPrivKey->get('kid')  
            ])  
            ->build();  

        return $jws;  
    }  

    public function createRT(User $user)  
    {  
        $token = bin2hex(random_bytes(64));  

        $rt = (new RefreshToken())  
            ->setUser($user)  
            ->setToken($token)  
            ->setExpires((new \DateTime())->modify('+30 days')) //This decides the RT TTL 
            ->setIssued(new \DateTime());  

        return $rt;  
    }  

    public function generateJWT(User $user): string  
    {  
        $jws = $this->generateJWS($user);  
        return $this->serializeJWS($jws);  
    }  

    public function verifyJWS(JWS $jws): bool  
    {  
        $valid = $this->verifier->verifyWithKeySet($jws, $this->userSigPubKeySet, 0);  
        return $valid;  
    }  

    public function JWSExpires(JWS $jws): \DateTime  
    {  
        $payload = json_decode($jws->getPayload(), true);  
        return date_create_from_format('U', $payload['exp']);  

    }  

    public function serializeJWS(JWS $jws): string  
    {  
        return $this->serializer->serialize($jws);  
    }  

    public function unserializeJWT(string $JWT): JWS  
    {  
        return $this->serializer->unserialize($JWT);  
    }
}

The only things you might want to take a look at are the createRT and generateJWS functions these two might need adjusting depending on your situation. Primarily the generateJWS function's data array. in this array you can store any information that you want to be signed. However this information can be decoded by anyone since its just base64. So only put things in there which are not super critical i.e. passwords or api keys.

In this code the TTL of both the RT and JWT are defined, change them however you like, I personaly found JWT 10 min and RT 1 month to work well for my situation.

Providing JWT's

Now that we've got our JWTService setup we need to create 3 endpoints

to share our public key set (JWKS)
for the user to login and request an initial JWT and RT
for exchanging an RT for a new JWT and RT

Here is a simple controller that does these 3 things:

<?php  

namespace App\Controller;  

use App\Entity\RefreshToken;  
use App\Service\JWTService;  
use Doctrine\ORM\EntityManagerInterface;  
use Jose\Component\Core\JWKSet;  
use Symfony\Bundle\FrameworkBundle\Controller\AbstractController;  
use Symfony\Component\HttpFoundation\Request;  
use Symfony\Component\HttpFoundation\Response;  
use Symfony\Component\Routing\Attribute\Route;  
use Symfony\Component\Security\Core\User\UserInterface;  

class JWTController extends AbstractController  
{  
    public function __construct(  
        private readonly JWTService             $jwtService,  
        private readonly EntityManagerInterface $entityManager  
    )  
    {  
    }  

    #[Route('/.well-known/jwks.json', name: 'jwt_jwks')]  
    public function jwks(JWKSet $userSigPubKeySet)  
    {  
        return $this->json($userSigPubKeySet);  
    }  

    #[Route('/jwt/request', name: 'jwt_request')]  
    public function request(UserInterface $user)  
    {  
        $jws = $this->jwtService->generateJWS($user);  
        $rt = $this->jwtService->createRT($user);  

        $this->entityManager->persist($rt);  
        $this->entityManager->flush();  


        return $this->json([  
            'token' => $this->jwtService->serializeJWS($jws),  
            'token_expires' => $this->jwtService->JWSExpires($jws)->format('U'),  
            'refresh_token' => $rt->getToken(),  
            'refresh_expires' => $rt->getExpires()->format('U')  
        ]);  
    }  

    #[Route('/jwt/refresh', name: 'jwt_refresh', methods: ['POST'])]  
    public function refresh(Request $request)  
    {  
        try {  
            $body = json_decode($request->getContent(), true);  
            $token = $body['refresh_token'];  
        } catch (\Throwable $th) {  
            return new Response('Bad request', 400);  
        }  

        $oldRt = $this->entityManager->getRepository(RefreshToken::class)->findOneBy(['token' => $token]);  

        if (!$oldRt) {  
            return new Response('non-existent token', 404);  
        }  

        if ($oldRt->getExpires() < new \DateTime('now')) {  
            return new Response('Token expired', 400);  
        }  

        $user = $oldRt->getUser();  
        $this->entityManager->remove($oldRt);  

        $jws = $this->jwtService->generateJWS($user);  
        $newRt = $this->jwtService->createRT($user);  

        $this->entityManager->persist($newRt);  
        $this->entityManager->flush();  

        return $this->json([  
            'token' => $this->jwtService->serializeJWS($jws),  
            'token_expires' => $this->jwtService->JWSExpires($jws)->format('U'),  
            'refresh_token' => $newRt->getToken(),  
            'refresh_expires' => $newRt->getExpires()->format('U')  
        ]);  
    }  
}

Again, you can and should change this controller in what ever way best suits your situation.

Somethings to note, the jwt_jwks is fully public. This is not a (big) security risk as all this allows is validation of the signature. however it is generally advisable to also limit access to this route if possible.

In closing

Here are some things you should now also do:

rotate your keys, I'd recommend setting up something that rotates the private key very day or so or on deployment, this is especially recommended if you keep the jwt_jwks public.
Cleanup RT's It is also recommended to set up a cron job that deletes any expired RT's since there is no reason to keep them.
Set up testing setup proper testing to make sure your system is well protected.

If you have any questions or find problems in this guide I would love to help clarify / fix them @TomvdPeet. But keep in mind that this guide is supposed to be followed loosely and modified to your needs.

This article is 100% human written

Lastly this guide is provided “as is” for informational purposes. Use at your own risk. No warranties. I accept no liability for any loss or damage arising from its use.

Original article: https://browsely.ai/blog/symfony-as-jwt-provider

Pivoting from Auth0 to PHP Symfony

TomvdPeet — Wed, 20 Aug 2025 14:51:00 +0000

These days when creating a system there are many different ways of setting up authentication. However the main options boil down to: a self made solution, a Aaas solution (authentication as a service) or a combination of the two, Aaas has become more a lot popular recently, with companies like OpenAI opting for this. And in the space of Aaas the most recognizable name is Auth0 (Partly i think because of OpenAI).

The reason we needed authentication was to verify user in node.js. Node being our service for communicating between the user and the LLM. In this article we'll go over: why we went for Auth0, the problems we ran into, and why we pivoted to Symfony

Why we chose for Auth0

Starting out with Browsely we obviously wanted to get the product to the market as fast as possible, so we opted for what seemed like the easiest and quickest solution Aaas and specificly Auth0. Here is a list of perceived pros of Auth0:

Quick to implement, because we wouldn't have to build much
Free to get started
Social login off the bat
No need to setup email and for: password reset and registration

These points seemed very appealing so we got started with implementing it and we very quickly started running into problems.

The problems with Auth0

When implementing Auth0 the first problem we ran into is that its existing JS package is not really meant to be used for a web extension. From some things we came across online we found that supposedly in the past they did have support but it since has been removed. This meant we had to implement quite a lot of the authentication layer our self's. And because the JS package is meant to be used plug-and-play without much customization we had to back engineer it quite a bit to get it to work in a web extension. But eventually we did get it to work and did end up working as expected.

Now the next problems arose when building the subscription layer, which we decided to do with Stripe as payment provider. This created two issues 1. We now needed to store user data in two external systems, both of which needed to be kept up to date. 2. The user would now have to log in to our portal and in to the extension separately which we thaught would feel clunky. We did make a start on trying to solve the first problem but very quickly we felt the effort required to implement this neatly was to great at this stage in the project. And it was at this point we decided to pivot.

Another problem not mentioned before is the pricing of Auth0 as we started using it we found the free packed permits quite a lot, and the payed plans seemingly less by comparison. Overall it was unclear when you'd have to pay and what you'd have to pay for. This problem was less pressing but one that could blow up at a later stage.

A quick summary of the problems we ended up having with Auth0:

Messy to implement in to an extension
Messy to set up with Stripe
The user would have to login manually in two different places
Vague pricing

The pivot to Symfony

Something i have to mention: our website was already built in Symfony. Now i hear you ask, why Symfony? That's simple, we know Symfony, so we can build fast and know what we can expect from it, now and moving forward. So logically we would migrate the whole auth system to it.

One of the things we learned form our initial Auth0 implementation that we really liked was JWT. A protocol that would make centralized authentication much easier. So our setup would be: Symfony creates and signs a JWT and send it to the user, the user would send it with their request to node, node verifies based on the public key.

If you want more technical information about how we setup Symfony as a JWT provider lookout for a future article where we go more in depth. Since we found there currently is not much information on how to do this with Symfony.

Now, this setup is essentially the same way Auth0 works the only difference being that implementing this yourself gives a lot of liberties and options, allowing us to resolve most of the problems we had with Auth0. The implementation is much cleaner and allows for the user to simply log in to our website and they will automatically be signed in to the extension. We dont have to sync user data between both Auth0 and Stripe. And the pricing is very stable and predictable this way. The only downside thus far has been the lack of social login off the bat.

We are now very happy about our decision to pivot and have learnt a lot from the whole experience.

Conclusion

What have we learned. And what can you learn from this.

Stick to what you know. Building the auth system in Symfony ended up taking about as much time as setting up Auth0. Howerver the result is alot more robust and ready for future developments.
Don't outsource to save time. This is something we had already experienced in the past but at a small scale outsourcing usually costs about as much time if not more as doing it your self. There are exceptions like payment providers for example but generally, if you can do it yourself its usually faster.
Control is important. Keeping as much of your product in house as possible especially starting out. Its easier to outsource something at a later point than it is to take something back.
Be ready to pivot If you feel like a solution is causing more problems than its solving its most often time to look for a better solution.

Also i quickly want to mention that this article is not against Auth0 or Aaas in anyway, we think there are very valid use cases, especially for more traditionally structured web-apps.

Original article: https://browsely.ai/blog/pivot-auth0-to-symfony