DEV Community: Rubén Rubio

Wednesday, 20th of December 2023, 10.08 AM
Client: "Our PHP + Symfony application has performance issues. The website is slow; it takes a lot of time to load some pages. Sometimes, it just freezes, making it unusable. We are receiving complaints from our users: they need it to work, but they can not."

Wednesday, 20th of December 2023, 04.57 PM
Developer: "Hello! That is something I can look into. We can solve it, no problem. However, the next two weeks I am on holidays, so I will work on it when I am back. Merry Christmas and Happy New Year!"

Monday, 8th of January 2024, 07.43 AM
Client: "How is the optimization going? My users could not have holidays: they had to make up for the time lost while working with our website. They are starting to get angry. I even got some threats in my physical mailbox."

Monday, 8th of January 2024, 01.19 PM
Developer: "First, Happy New Year! I understand your users: it is normal to become irritable when you do not have time off. They should take their job less seriously; they are not going to inherit the company. And for you, just do not look into your mailbox. Who sends letters today, anyway?"

"But let me take a look at your application. First, we need to have some metrics to know what is going on. We need to observe which endpoints are slow and which ones are called the most... Otherwise, we are just acting on sensations, not on data. To observe the system, I just installed New Relic. We should have enough data to draw conclusions in a week. We can act then."

Monday, 15th of January 2024, 08.56 AM
Client: "How about the metrics of our application? A week has gone by already. Can you see something that can be improved? The users are losing their patience, and I am starting to get afraid. Yesterday, I found the windows of my car broken, with a note inside: my users request response times of less than 1 second. Otherwise, things will become more serious, they wrote."

Monday, 15th of January 2024, 10.25 AM
Developer: "It is a reasonable request, I must say. Any website nowadays should have a response time of less than 1 second on average. Maybe you can consider it a turning point in your life and stop using a car? It is healthier to walk, you will feel better. But I see there are two endpoints that take up to 50% of the load of the system. Both are queries, i.e., they just get data to show it to the user. They heavily use Doctrine, an ORM, so we are facing the famous N+1 problem. But fear not! I will fight back and win: we should be able to cache the data in Redis, so we avoid querying the database every time to get the same data, and get the calculated data straight from memory. Just hang in there."

Friday, 19th of January 2024, 02.05 AM
Client: "Hey! How is that performance improvement going? My family and I are «spending away time» with some of the ~~angriest~~ ~~craziest~~ users of our application. The worst? They keep showing me how slow and unusable it is. I have come to the point where I understand them. Anyway, let me know, how the improvement is going."

Friday, 19th of January 2024, 12.46 PM
Developer: "I am glad to know that you are enjoying some time off. You deserve it, buddy! In the meantime, I optimized both endpoints that were the slowest: they are now using projections, and all the models they require are cached in Redis. However, I cannot see much improvement, the response times are still over 1 second. Well, it is Friday noon, so I will look into it first thing on Monday. Have a nice weekend!"

Saturday, 20th of January 2024, 03.23 AM
Client: "HEY MAN! THEY TOOK MY DAUGHTER AWAY. THEY MADE ME SAY GOOD BYE TO HER. I DO NOT KNOW WHAT IS HAPPENING ANYMORE."

Monday, 22nd of January 2024, 12.28 PM
Developer: "They grow up so fast! It is good to let them have space and make their own mistakes. Well, I found it: it seems we forgot to enable OPcache. I enabled it at 11.45 AM. Look how we reduced the response times:"

"Here is the chart with transactons time after enabling OPcache:"

"And here is the chart with the AppIndex score:"

"Problem solved!"

Monday, 22nd of January 2024, 12.29 PM
Client: "WTF IS AN OPCACHE?"

Monday, 22nd of January 2024, 3.07 PM
Developer: "PHP is an interpreted language. That means every time a script is executed in a request, the code must be converted to bytecode (or machine code), i.e., the code the computer understands. However, in production, the code is always the same amongst each deploy, so that conversion can be cached for all requests. That is what OPcache does. By the way, you should look into your «Caps. lock» key. It seems stuck."

Monday, 29th of January 2024, 11.50 AM
Developer: "Here is the bill. You could have at least thanked me after all the work I put when optimizing your application."

Weekly reading digest #27

Rubén Rubio — Sun, 18 Feb 2024 09:00:00 +0000

Elegant immutable object pattern in PHP

Building Maintainable PHP Applications: Thinking Data vs Thinking Business Processes

AI for PHP: How To Automate Unit Testing Using AI Assistant?

DTO vs VO in PHP

Stop using old-fashioned closures in modern PHP. There are 4* ways to replace them.

How to get URL parameter using jQuery or plain JavaScript?

How I take notes: Structure with Now Next Notes

The PARA Method: The Simple System for Organizing Your Digital Life in Seconds

Beyond toxic positivity

Weekly reading digest #26

Rubén Rubio — Sun, 11 Feb 2024 09:00:00 +0000

Three dots in PHP. First-Class Callable Syntax, Variadic function & Argument unpacking

Common services with PHP applications

Code to an interface!

Understanding Automated Testing

DbToolsBundle. A set of Symfony Console Commands to interact with your database

Discover Symfony UX. UI with Stimulus = fewer JS headaches for you

Wars of Symfony Runtimes: A Performance Odyssey

Composer 2.7 and CVE-2024-24821: Code execution and possible privilege escalation

Supercharging GitHub Actions with Job Summaries

Weekly reading digest #25

Rubén Rubio — Sun, 04 Feb 2024 09:00:00 +0000

Generating Large ZIP Exports From Files in S3

Avoid mocking repositories by using in-memory implementations

YAGNI exceptions

PAGNIs: Probably Are Gonna Need Its

Why You Should Never Use MongoDB

Readme Driven Development

Weekly reading digest #24

Rubén Rubio — Sun, 28 Jan 2024 09:00:00 +0000

All the recursive Functions in PHP

GitAttributes for PHP Composer Projects

The infamous Repository Pattern in PHP

Designing fine-grained independent features using Symfony and Strategy Pattern

How to ensure that all the routes on my Symfony app have access control

How Standard Ebooks serves millions of requests per month with a 2gb vps; or, a paean to the classic web

Testing tips

Weekly reading digest #23

Rubén Rubio — Sun, 21 Jan 2024 09:00:00 +0000

Performance benchmark of PHP runtimes

Adding feature flags to your Symfony app

Performance Improvement Methods

Understanding Kolmogorov Complexity

Weekly reading digest #22

Rubén Rubio — Sun, 14 Jan 2024 09:00:00 +0000

Deploying in Production | FrankenPHP

Introducing tabular assertions

Tabular assertions with PHPUnit

Building Maintainable PHP Applications: Framework Decoupling vs Framework Coupling

Solving the N+1 problem in ORMs

What is a Command Bus? (vídeo)

Optimising a web application (III): projecting

Rubén Rubio — Wed, 10 Jan 2024 09:00:00 +0000

Introduction

With this post, we finish the series of optimising a web application. As a reminder: the application consists of a Symfony application to manage the entrance and exit to buildings using validation of previously purchased tickets.

In the previous post, we performed some optimisations in an application by indexing the database. We improved the performance by a 57 %. However, the absolute response times were of around 300 ms, that we consider too high. In this post, we will investigate further issues and solve them.

Issue

As in the previous case, we relied on New Relic to identify the problem. After reviewing both the endpoints and the database metrics, we found the queries that were causing the issue. The application provides some real-time data for its users: the number of people inside a building and the names of the people inside a building (only for administrators). To provide this information, the application uses tables with logs of all the check-ins and check-outs of users.

So, in order to calculate the number of people there are now inside a building, the application counts the total of check-ins and checkouts along its whole existence, i.e., it scans data from years ago to give the numbers of the present moment. The query it uses is:

SELECT COUNT(event_checkin.id)
FROM event_checkin
WHERE (event_checkin.building_id = :building_id)
  AND (DATE(event_checkin.created_on) = DATE(:date)
  /*...*/

The query is quite complex and used in several paths, affecting business rules, so we decided not to change it and look for alternatives.

Solution

We want to improve the performance for end-users and administrators, as they are using the application to check in/checkout: they need it to be fast to access/leave the building. The rest of paths are not critical in terms of response time.

We actually only want two pieces of data for these critical paths:

The number of people inside the building.
The names and some extra data, like the email, of the people inside the building (only available for the administrators).

What we can do is have this data prepared: we can act when the user checks-in and checks-out. We will use events to denormalize the data we need and generate projections: we will save together all data we need, so we only have to perform simple queries to get it.

The schema for the check-in will be like this:

When a user checks in, an event is fired to a queue service. Afterwards, two consumers process the event and execute the required action: updating the number of people inside a building or generating a projection with the user data. The checkout case is analogous.

We will only perform the changes to these paths, and leave the rest of paths untouched, so we do not affect the domain logic.

Implementation

The application uses hexagonal architecture, so these two commands, for check-in and checkout, are isolated and well identified in code. We had to perform the following changes:

Fire the events from the domain entities when a ticket is checked-in/checked-out.
Consume the events and generate/update the projections:
- Update the number of people inside a building, by increasing or decreasing a value, depending on if the command is a check-in or a checkout, respectively.
- Insert or delete a row with the data of the use inside a building, depending on if the command is a check-in or a checkout, respectively.
Change the paths from the external layer (the controllers) to the domain layer, so it reads the projected data instead of querying the database.

We could use Redis or another persistence system, but we chose to use MySQL to avoid performing more changes to the project. As the project uses hexagonal architecture, it is easy to swap the implementation in the future if required. We used Symfony Messenger with RabbitMQ as broker for messaging. How we set it up is beyond the scope of this post.

In the end, the queries we had to perform were simple and fast. To get the data of the people inside a building, the query becomes:

SELECT total_in
FROM view_building_data
WHERE building_id = :building_id

And the query to get the information of people inside a building:

SELECT *
FROM view_users_inside_building
WHERE building_id = :building_id

Both queries are using primary keys, so they are optimal.

Results

Unfortunately, the application is seasonal, i.e., it is only used in Summer, and we deployed these changes in Autumn. Therefore, we can not validate the results in production until next Summer.

We did, nonetheless, some checks ourselves. Both endpoints' response time was of around 1.1 seconds (with the possibility to get higher as the data grows up). After deploying the changes, the response time decreased to 150ms. As both queries are simple and use primary indexes, we are sure that this response time will always be similar.

Conclusions

We can allegedly affirm that we improved the response time. We will know for sure next Summer (there will be a post about it).

As for future improvements, we could stop using the logs tables everywhere they are used, and analyse that data differently, instead. It will be done if needed.

Summary

Again, using New Relic we identified performance issues in the queries of the application.
We proposed a solution based on events to decouple the application and project the data we need.
We implemented that solution by only changes those paths of the application that are most important for end-users.
We could not assert that the changes work in production, as the application is seasonal.