DEV Community: Magecom

Profiling and Debugging for Magento and What to Consider When Choosing

Magecom — Tue, 28 Feb 2023 13:35:12 +0000

This article is based on experience and research by Nezbrytskyi Serhii, CTO at Magecom.

Profiling and Debugging

Profiling involves measuring and analyzing the performance of a program, such as how much time it takes to execute specific tasks or how much memory it uses. This information can help developers optimize the program's performance and identify bottlenecks or inefficiencies.

Debugging involves finding and fixing errors or bugs in a program. When a program doesn't work as expected, it can be helpful to use debugging tools and techniques to identify the cause of the problem and fix it. This can involve using a debugger to step through the code line by line, inspecting variables and data, and logging or other forms of output to understand what the program is doing and where it is going wrong.

Both profiling and debugging are important parts of the software development process and can help ensure that a program is reliable, efficient, and effective.

Magento 2 Profiler

To use the Magento 2 Profiler on the shop page, you will need first to enable it in the Magento 2 configuration. You can do this by going to the Magento 2 backend and navigating to "Stores > Configuration > Advanced > Developer > Debug". From here, you can set the "Profiler" setting to "Yes" to enable the profiler.

Once the profiler is enabled, you can access the profiler output for the shop page by adding a query string parameter to the URL of the page. For example, you might go to the shop page URL and add "?profiling=1" to the end of the URL. This will enable the profiler for that page and display the profiler output in the browser console.

The profiler output will provide detailed information about the performance of the shop page, including how long each part of the page takes to load and the amount of memory used. You can use this information to identify bottlenecks or inefficiencies in your Magento 2 application and optimize the performance of the shop page.

In addition to the profiler output, you can also use other tools and techniques, such as debugging and logging, to further understand and optimize the performance of the shop page in Magento 2.

Magento 2 Profiler is quite easy to use and easy to apply, but there are some nuances. If, for example, you have some kind of complex query, it is quite inconvenient to use since you will have to put a lot of labels in the code. We recommend the Xdebug profiler.

Xdebug Profiler

Xdebug is a PHP extension that provides a range of debugging and profiling tools for PHP applications. The Xdebug profiler is a feature of Xdebug that can be used to profile the performance of a PHP application.

To use the Xdebug profiler, you will need to have Xdebug installed and configured on your server. You can then enable the profiler by setting the "xdebug.profiler_enable" PHP configuration option to "1" in your PHP configuration file.

Xdebug2
xdebug.profiler_enable_trigger = 0
xdebug.profiler_enable=1
xdebug.profiler_output_dir = /var/www/profiler

Once the profiler is enabled, Xdebug will automatically collect profiling data for each PHP script that is executed. This data includes information about the functions and methods called, the time taken to perform each function, and the memory usage of the script.

To access the profiler data, you can use a tool such as Webgrind or KCacheGrind to visualize and analyze the data. These tools provide a graphical interface that can help you understand the performance of your PHP application and identify bottlenecks or inefficiencies.

So, the profiler, Xdebug, also provides other debugging tools, such as the ability to pause the execution of a script and inspect variables and support for remote debugging. These tools can be useful for identifying and fixing problems in PHP applications.

Xdebug3 Profiler

Xdebug3 is the latest version of Xdebug and is compatible with PHP 8.0 and newer. Like previous versions of Xdebug, Xdebug3 provides a range of tools for debugging and profiling PHP applications, including

The ability to pause the execution of a script and inspect variables, stack traces, and other information about the state of the script.
Support for remote debugging, which allows you to debug a PHP script running on a remote server from your local development environment.
The Xdebug profiler, which can be used to collect and analyze profiling data about the performance of a PHP script.
Support for code coverage analysis, which can be used to determine which parts of a PHP script are being executed and which are not.

Xdebug3
xdebug.mode = profile
xdebug.output_dir = /var/www/profiler

Xdebug3 also supports some new features and improvements, such as:

Improved performance and reduced memory usage.
Support for asynchronous trace files, which can be used to debug asynchronous PHP code.
Improved support for PHP 8.0, including support for attributes and union types.
Improved support for debugging and profiling PHP scripts run via the command line. Overall, Xdebug3 is a powerful and useful tool for debugging and profiling PHP applications and can help developers identify and fix problems and optimize the performance of their code. ##Blackfire Profiler Blackfire Profiler is a paid tool for profiling and optimizing the performance of PHP applications, especially when you have recursive or cyclical calls.

This Profiler provides detailed information about the performance of your PHP code, including the time taken to execute each function or method, the number of calls made to each function, and the memory usage of each function. It also provides recommendations for optimizing the performance of your code based on this data.

Blackfire Profiler can be used in a variety of environments, including local development, staging, and production, and can be integrated with a range of tools and platforms, such as continuous integration systems and cloud platforms. It can be an effective way to identify and fix performance issues and improve the efficiency and reliability of your code.

XHprof Profiler

Some time ago, xhprof was very popular, and Facebook developers created it for their own needs and then put it Open-Source.

This tool provides a graphical interface that can help you understand the performance of your PHP application and identify bottlenecks or inefficiencies. XHprof also provides other tools, such as the ability to generate flame graphs, which can help visualize the performance of PHP scripts, and support for sampling, which can be used to reduce the overhead of profiling.

Conclusion

In conclusion, profiling, and debugging are essential techniques in research to help identify and fix problems in software systems. Profiling involves measuring the performance of a program or system, while debugging consists of finding and correcting errors in a program or system. Various tools and techniques are available for profiling and debugging, such as performance profilers, memory profilers, debuggers, trace viewers, and log analyzers.

The profiling and debugging investigation typically involve reproducing the problem, collecting data, analyzing the data, identifying the root cause, fixing the problem, and testing and validating the fix.

It is important to remember that profiling and debugging can be challenging, time-consuming tasks and requires a good understanding of the system, and the problem is solved. If you need advice or help, you can always contact us.

Google Page Speed Insights and Magento 2

Magecom — Mon, 07 Nov 2022 15:38:12 +0000

My name is Sergey. I am CTO at Magecom. My article is based on a talk I gave at Magento Meetup #11, where I talked about the results of the work of our entire Magecom team.

Let's say you have a task to improve the performance of a site about which you still know almost nothing. For example, if this is a new client who came with a specific problem since the website on Magento 2 is working slowly.

The task that the team and I solved was assembling a set of universal solutions suitable for most projects. The main project requirement was to make solutions be with a minimum estimate and as automated as possible to save the estimate. The conditions under which solutions can be applied include a minimum set of project knowledge so that an engineer outside the project context can use these solutions.

Google PageSpeed Insights

To evaluate the results of our work, we use Google PageSpeed Insights.

It is a set of scripts for measuring metrics (but not only). First of all, I would say that this is a tool a client uses to confirm their feelings that their site is somehow slow. It means that the same tool can be used to show the client the effectiveness of your work in improving performance. There may be purely subjective feelings of the website speed, but the numbers are better.

In fact, the benefits of Google PageSpeed Insights are not over. Besides the fact that it can measure and visually demonstrate the performance of the site, but also gives recommendations on how to improve this performance.

How we see this process

Quite often, you can see such a life flow of a client’s project.

At the start, when Magento is configured, it has a small performance out of the box. Next, when the Nth number of modules and a theme are introduced, the performance decreases even more. At the go-live stage, the volume of content increases since there’re many products, pages, widgets on the homepage, and numerous categories in the main navigation menu ( nodes in DOM). As a result, the performance fails. After the go-live, we can see that the website works slowly, so we spend a couple of days on doing something with that, slightly improve the performance, but not even reaching the indicators at the beginning of the development process.

The second line is the hours spent on improving performance based on this approach.

How we would like to see this process

We would like to have everything to the maximum at the start of the project and during development. Without doubt, a slight drawdown is allowed, especially at the go-live stage, but after relatively little effort, we should return the indicators to normal.

But for this, you need to make a small investment (up to 8 hours) at the start of the project and during the development period.

My article is about what can be done during these 8 hours at the start of the project. And it doesn’t matter if you are starting a project from scratch or if it has just entered your company. Our task is to improve the standardized performance as much as possible.

How did we develop this procedure? We just took the path of least resistance and started checking everything to make it meet Google PageSpeed Insights and Lighthouse.

Prerequisites: measure performance

npm install -g lighthouse-ci
npx lighthouse-ci https://som.e/plp.html --filename=plp.html

The main rule here is that you should measure performance.

It's great that there are tools that allow you to do this automatically. It’s also good if you already have CI/CD configured. You can install Lighthouse CI, an npm package that can be run during pushes and deployments. It will give you a report during every pull request so you can always see which pull request is breaking performance.

Let's say we installed a module and added 150K UI scripts to the front of all pages, although it’s just a store locator. It is better to avoid the situation when you introduce a store locator and, without measuring anything, implement an in-store pick up on the checkout based on this store locator. Then it turns out that it is very slow, and you need to remove this whole thing, but the checkout is already built on it. Of course, it is better when you immediately receive notifications about this and can instantly fix it.

As a result, the first thing to start is setting up the performance measurement.

What Google PageSpeed recommends us

To evaluate performance, Google Page Speed Insights measures several metrics and also evaluates some site parameters that can significantly affect these metrics. For each such parameter, there are recommendations on how to improve it if it suddenly turns out to be in the red (and not only in the red) zone. Sometimes these are general recommendations; sometimes, they are personalized for a specific platform.

Do not load images that are not visible on the first screen

Do not load images into the browser that users won’t see on the first screen. You don’t need to load them until they scroll to them or until your slider starts scrolling through these images, . It is possible that they won't scroll to them, and there is no point in wasting network and browser resources to load them.
There are several solutions to this issue:

The old way: go to the marketplace, find the extension with lazy load and install it.
New way: since HTML5 supports the lazy loading of images, you need to add the loading=lazy attribute to the img tag.

At Magecom, we created an extension for this, which looks something like this if you skip details:

/**
 * @see \Magento\Framework\Controller\ResultInterface::renderResult
 */
public function afterRenderResult(
    \Magento\Framework\Controller\ResultInterface $subject,
    \Magento\Framework\Controller\ResultInterface $result,
    \Magento\Framework\App\ResponseInterface $response
): \Magento\Framework\Controller\ResultInterface
{
    $content = $response->getBody();
    $content = $this->imageProcessor->process($content);
    $response->setBody($content);
    return $result;
}
public function process(string $content): string
{
    $closure = function (array $match) {
         return str_replace('<img', '<img loading="lazy"', $match[0]);
    };

    return preg_replace_callback('/(<\s*img[^>]+>)/i', $closure, $content);
}

Before the server returns the html to the browser, we go through this html regularly and add the loading=”lazy” attribute to all IMG tags. That's it.

Reduce server response time

In fact, and it’s the most painful topic. If we skip the details, we can state that Magento is slow.

The main reason that is often mentioned in the context of this problem is the significant number of modules that are not the fact that will be used in the application. As a result, one way to improve this indicator is to turn off unused modules. You can find several reports and talks on this topic on various sites, and it was even mentioned on Mageconf. There are many solutions on how to turn off modules if you are not using MSI or GraphQL, for example. Lots of solutions.

The effect is obvious since the more modules we remove, the fewer configs will be collected, and the whole thing will work faster. The effect will be maximum if you remove the maximum number of modules.

To do this, we developed a module that does a little trick like this.

public function getUnusedModules(): array
{
    $allModules = $this->getAllModules();
    $requiredModules = $this->getRequiredModules();
    $unusedModules = array_diff_key($allModules, $requiredModules);
    ksort($unusedModules);
    return $unusedModules;
}

private function getRequiredModules(): array
{
    $requiredModules = [];
    $modules = $this->configProvider->getConfig();

    foreach ($modules as $moduleName) {
        $this->addRequiredModules($requiredModules, $moduleName);
    }
    ksort($requiredModules);
    return $requiredModules;
}

private function addRequiredModules(array &$requiredModules, string $moduleName): void
{
    if (array_key_exists($moduleName, $requiredModules)) {
        return;
    }

    $module = $this->moduleRepository->getModuleByModuleName($moduleName);
    $requiredModules[$moduleName] = $module;
    foreach ($module->getDependencies() as $dependency) {
        $type = $dependency['type'];
        $dependencyName = $dependency['module'];
        if ($type === 'hard') {
            $this->addRequiredModules($requiredModules, $dependencyModule->getFullName());
        }
    }
}

We create a whitelist of modules that are used on the project, for example, Magento_ConfigurableProduct, Magento_Checkout, Magento_Sitemap, add to this list all the dependencies of these modules (see the composer.json require section) and dependencies of those dependencies, and so on recursively, until we get the complete list of modules used on the project. And then, we remove all the remaining modules by listing them in the replace section in the composer.json of the project.

If we skip the details, the main methods here are getUnusedModules, which returns us a list of modules that are not used in the project. It takes all the modules, and those that are necessary, calculate the difference and returns a list of modules that are not required.

An interesting point is how getRequiredModules is calculated. It just implements the main algorithm of the module, which I described above.

It is a fairly simple script, but it allows us to get the most extensive possible list of modules that can be removed.

Preload Key Request

There are resources on the site that you may need from the very beginning of the page loading. It would be great if they were already loaded from the start and not delayed until the moment they appear in the DOM.

The solution may be as follows: you need to find these resources and add them to the xml layout through the link and specify the relation preload.

On web.dev, where this algorithm is described, you can see how to read it. Or you can start Google PageSpeed lnsights in the report to see which resources are fundamental to add to preload.

Minify CSS

This point is already implemented in Magento. All we have to do is enable minification. I suggest doing this at the stage when we launch the project.

bin/magento config:set dev/css/minify_files 1 --lock-config

For example, in our company, the configuration of the deployment project is automated, so it is added to scripts with builds. We have a project from the start in config.php that we minify scripts. If someone needs them to be non-minified locally in developer mode, they override the settings in env.php.

Minify JavaScript

If you don't have minification for JavaScript enabled, Lighthouse will not be satisfied and recommend you turn off default minification and use Terser. At the same time, if you allow minification in Magento, then it will be satisfied and will not notice that you are using Magento minification and not Terser.

Terser works better, but there is not much profit, so I don’t recommend using it just for this. Terser will be helpful to us a little later, and you can see it on Github.com.

There are other options if you do not want to install and configure Terser.

Eliminate render-blocking resources

The task is to display the first screen to users as soon as possible so that they start interacting with the site as quickly as possible. All we do not need on the first screen, we place in the footer and do not load directly from the start. The primary resources are CSS and JavaScript.

Google Page Speed Insights recommends doing the following steps.

For CSS, we need to enable critical CSS.

php bin/magento config:set dev/css/use_css_critical_path 1 --lock-config

There is a tool that allows generating critical CSS and determining which CSS is critical and which is not. It helps to separate it into files, and there is also an npm package for that.

As for JavaScript, moving them to the footer is recommended, so they do not slow down page loading.

php bin/magento config:set dev/js/move_script_to_bottom 1 --lock-config

As it turned out, the recommendations are pretty controversial.

Applying these recommendations slightly improves the First Contentful Paint score but significantly degrades the Cumulative Layout Shift. And therefore, if we measure the performance BEFORE and AFTER these settings, then we will see performance degradation at the end.
As a result, we do not apply these recommendations by default.

Keep Request Counts Low

The most well-known moment is Keep Request Counts Low. The number of requests from the frontend is huge, they need to be reduced, and everything is quite simple here.

We include merge CSS. Instead of hundreds of CSS files, one merged is loaded.

CSS:

php bin/magento config:set dev/css/merge_css_files 1 --lock-config

As for JavaScript, Magepack is an advanced bundler, has proven itself very well on projects, and is easy to configure.

JavaScript:

composer require creativestyle/magesuite-magepack
npm install -g magepack

bin/magento config:set dev/js/merge_files 0 --lock-config
bin/magento config:set dev/js/enable_magepack_js_bundling 1 --lock-config

magepack generate --cms-url="{HOMEPAGE_URL}" --category-url="{PLP_URL}" --product-url="{PDP_URL}"
magepack bundle

By the way, here we still need Terser because if we use the Magepack bundler, we first need to generate statics, and then, based on a specific config, we assemble bundles from these statics. And only after that do we already minify the bundles. And if we enable Magento minification, then we force Magepack to work with minified files.

Avoid excessive DOM size

If you have more than 3000 nodes on the page, then Lighthouse will definitely be dissatisfied with that. It cannot be fixed quickly, and there is no general rule for how to do this.

There are several indirect ways of.

Disable Page Builder. Disabling it can reduce the number of nodes. But Page Builder is an important element of the online store, so this step should be discussed with the client. Perhaps the client paid for a Magento license to have a Page Builder here or hired a content manager with no knowledge of HTML just because there is a Page Builder. If there are no such strict requirements, you can try to stop using the Page Builder to reduce the DOM size.
Load some content via Ajax. For example, you have 1000 categories in your store, and they are all on the menu. You load them, and inside, you have
- , and also in each one, for example, or other elements. By using Ajax you can leave only the top-level menu and pull up the lower levels.

Show images in next-gen formats

A new image format, WebP, has appeared, which is not inferior in quality to JPEG and also takes up less space. You can directly install any module from the marketplace that will convert on the fly.

There is an option to use the resources of various CDNs, such as Fastly and Cloudflare. They can convert your images to the WebP format and work better than modules. The CDN determines whether the browser supports the WebP format and if so, it converts JPEG to WebP on the fly.

Serve static resources with an efficient caching policy

Everything is good here by default because the default settings nginx have already configured the correct cache policy for all types of resources. The only point is if we start using the nginx config in WebP. Magento does not know about WebP, and you will need to add it to the appropriate section.

locator /media/{
     try_files  $uri  $uri/ /get.php?$args;
location ~ ^/media/theme_customization/.*\.xml {
      deny all;
}
location ~* \. (ico|jpg|jpeg|png|gif|svg|js|css|swf|eot|ttf|otf|woff|woff2|webp)$  {
           add_header  Cache-Control “public”;
           add_header X-Frame-Options “SAMEORIGIN” ;
          expires +1y;
          try_files $uri $uri/ /get.php?$args;
}
location ~* \. (zip|gz|gzip|bz2|csv|xml)$ {
         add_header Cache-Control “no-store”;
         add_header X-Frame-Options “SAMEORIGIN” ;
         expires         offf:
         try_files  $uri $uri/ /get.php?$args;
}
add_header X-Frame-Options “SAMEORIGIN” ;
}

Remove unused CSS

As soon as we enable Merge CSS, Lighthouse starts complaining about it. We have one large file with all the CSS from the entire site, and on each specific CSS page, it starts diagnosing that Merge CSS files need to be turned off.

But with Merge CSS being turned off, the indicators are lower. Therefore, as with render-blocking resources, we ignore this recommendation.

Enable text compression

The server should give scripts and all text files in a compressed way (gzip). Everything is fine here — the nginx config gzips them by default.

There is also a small micro-adjustment, which is to enable the minification of html files. This setting will remove all extra spaces and hyphens. Of course, in this case, you won’t save much, but we’ll save an extra byte:

php bin/magento config:set dev/template/minify_html 1 --lock-config

Efficient image encoding

If the CDN has an image optimization function, and you load it with good quality, then it raises them out to web quality and gives the browser a minimum of content. There are also a bunch of extensions that do the same thing, and we recommend Apptrian. We use it often and have no problems with it.

Core Web Vitals

Google Page Speed Insights measures metrics in two conceptually different contexts.

Lab Data are measurements made in laboratory conditions on some standardized hardware, with particular characteristics, and the latest version of the browser installed, which is Google Chrome. Based on these metrics, you are given recommendations for improving performance.

Field Data are almost the same parameters but are measured on the devices of specific users.

Without a doubt, these data can be very different from each other. And the effect of applying these recommendations can be very different.

For example, suppose most customers use the site through the Internet Explorer browser (because these are some government agencies where outdated software can be used). In that case, they may not notice that you added the loading=lazy attribute to the images because this browser does not support this attribute.

By default, we consider that these recommendations apply only to lab data. It may take more time to analyze customer devices to improve performance for customers.

What's next

It is not a complete list of recommendations. Some we still haven't tested firsthand, and they're in our TODO backlog. But the effect of their corrections should not be significant because, as a rule, Google Page Speed Insights does not swear at them when we are dealing with Magento.

There are also some metrics (for example, reduce JavaScript execution time) that cannot be improved quickly, and you will have to dive deep into the project to understand how you can fix it.
Now TODO list looks like this:

Conclusion

As a result, we looked at some simple and reliable ways to improve the site's performance so that Google Page Speed Insights will also like it.

If you have other ways to solve the problem raised here, write in the comments.

Note: Google Page Speed Insights is just a web service. And the set of metrics is another tool called Lighthouse for simplicity. Google Page Speed Insights just uses this tool. But for simplicity, the article only uses the name Google Page Speed Insights, even when it's more about Lighthouse.