Webpack stats file

Francisco Diaz Paccot — Mon, 18 Mar 2024 17:20:28 +0000

Analyze the amount of JavaScript we are adding in a pull request (PR) in a Next.js application using the Webpack stats file.

To begin with, it is important to say that when we refer to performance it is crucial to understand that as we incorporate more JavaScript in our frontend applications -especially if we use React as a base- we are more likely to experience performance issues. For this reason, I consider it critical to be able to exercise some control over the amount of JavaScript we add as we move forward with pull requests in our application.

To address this challenge (since most applications that use React as a library also use Webpack as a packager) we can take advantage of a very powerful tool provided by this bundle. This tool consists in generating the stats file of our application when creating the build. You can find more information about this in the official Webpack documentation.

With these tools in mind, a highly recommended practice before adding code to our main branch is to detect what we are adding to the application. This involves being clear about the amount of JavaScript, CSS, images and other resources we are adding.

This file is not generated automatically when building our application. In this blog we will explore how we can generate this file and also how to evaluate it to better understand what we are adding each time we push our code.

1- Prerequisites

Have an application that uses Webpack. For this example, we will create an application using Next.js which, by default, uses Webpack.
Install the webpack-stats-plugin plugin, which will allow us to generate Webpack statistics files.
Create a script (in this case, using Node.js) that evaluates the generated statistics file and provides us with useful conclusions about the resources added to our application.

2- Application to be measured

The first thing to do is to install the webpack-stats-plugin plugin or any other plugin that allows us to generate Webpack statistics files.

npm install -–save webpack-stats-file

Once the library is installed, it is necessary to adjust the Webpack configuration to integrate the plugin. Since, we are working with a Next.js application, this involves making modifications to the next.config.js file specifically.

next.config.js

const { StatsWriterPlugin } = require('webpack-stats-plugin')


const nextConfig = {
 …
 webpack: (config, _options) => {
   config.plugins.push(
     new StatsWriterPlugin({
       filename: '../webpack-stats-base.json',
       stats: {
         assets: true,
       }
     })
   );


   return config;
 }
};


module.exports = nextConfig;

With the configuration set up, when running the npm run build command, we should notice a new JSON file in the root of our application. This file will have the name we specified in the configuration, in this case, webpack-stats-base.json.

Note: In the example provided, within the configurations we are including assets: true. However, to further expand the content of the statistics file, we can include other attributes such as entrypoints, chunks, among others. To understand the purpose of each attribute and how it works, we recommend that you consult the official Webpack documentation at the following link: https://webpack.js.org/api/stats/

Ideally at this point we should generate this control file from our main branch. Once we have created the CLI and it is ready for use, we should save the generated statistics file in the repository. This will ensure that we always have access to information about the resources added to our application in the main branch of the project.

3- The CLI

To install commander and shelljs, you can run the following command:

npm install --save commander shelljs

Once these dependencies are installed, the CLI is instantiated using commander.

And then, add the handling of this option.

#! /usr/bin/env node
const { Command } = require("commander");

const program = new Command();
program
 .version("1.0.0")
 .description("An example CLI Analyze webpack bundle")
 .option("-c, --compare  [value]", "Compare both stats")
 .parse(process.argv);

const options = program.opts();

...

if (options.compare) {
 const baseStats = typeof options.compare === "string" ? options.compare : "webpack-stats-base.json";
 const baseStatsFilePath = path.join(process.cwd(), baseStats)
 compare(baseStatsFilePath, 'webpack-stats.json');
}

The implementation of the compare() function is missing, which is the one that will be in charge of doing all the comparison.

The next thing to do -once we run the CLI inside an application- is to install the dependencies and perform the build of the application. With these two steps completed, the new statistics file should have been generated (since we added the plugin in the Webpack settings in the previous step). Finally, we can compare both files to identify the differences.

Next, let’s see how to implement this in code. First, use ShellJS to execute the npm install and then npm run build commands. And the last step is to find both files that have been generated and compare them with each other.

#! /usr/bin/env node
const { Command } = require("commander");

async function compare() {
  console.log("Installing dependencies...")
  const installWorks = shell.exec("npm install").code

  console.log("Building...")
  shell.exec("npm run build")

  if (installWorks !== 0) shell.exit(1)

  // Comparation
  const filePath = path.join(folderPath, filename);


 try {
   const newData = fs.readFileSync(filePath, "utf8");
   const { assets } = JSON.parse(newData);


   let jsSize: number = 0;


   assets.forEach(({ name, size }: { name: string; size: number }) => {
     if (name.includes(".js") && !name.includes(".json"))
       return (jsSize += size);
   });


   const prevStats = fs.readFileSync(baseStatsLocation, "utf8");
   const { assets: prevStatsAssets } = JSON.parse(prevStats);


   let prevjsSize: number = 0;


   prevStatsAssets.forEach(
     ({ name, size }: { name: string; size: number }) => {
       if (name.includes(".js") && !name.includes(".json"))
         return (prevjsSize += size);
     }
   );


   const difference = [
     {
       type: "JAVASCRIPT",
       "base size (Kb)": prevjsSize / 1000,
       "PR size (Kb)": jsSize / 1000,
       "Difference (Kb)": jsSize - prevjsSize,
     },
   ];


   console.table(difference);
  } catch (err) {
    console.error(err)
  }
}

With this we should obtain as a result, a table that compares the amount of JavaScript that was there before with what we are adding or subtracting with this Pull Request (PR).

Up to this point we have developed the CLI to run in any of our applications, but this is limited to a local environment only. We can take it a step further and integrate it directly into our PRs, which will allow us to control every Pull Request that is sent to our main branch.

4- CLI from a Github Action

To run our CLI from a GitHub Action, we must first publish it. An efficient way to do this automatically (every time we update our CLI) is to create a GitHub Action specifically for this task.

To accomplish this, we would need to generate a token from our npm profile and then add that token as a secret variable within GitHub. In this example, let’s call it NPM_AUTH_TOKEN. Then, we can have a file called publish.yml inside the .github/workflows folder in our repository, where we will configure the workflow to publish our package in npm.

.github/workflows/publish.yml within the CLI

name: "Publish package to npm"


on:
 push:
   branches: [ main ]


jobs:
 publish:
   runs-on: ubuntu-latest
   steps:
     - name: checkout
       uses: actions/checkout@v2
     - name: node
       uses: actions/setup-node@v2
       with:
         node-version: 16
         registry-url: https://registry.npmjs.org
     - name: publish
       run: npm publish --access public
       env:
         NODE_AUTH_TOKEN: ${{secrets.NPM_AUTH_TOKEN}}

Now, we simply create the action in our Next.js application to execute our CLI every time we generate a Pull Request against our main branch.

.github/workflows/analize.yml within the application to measure

name: "Analyze webpack stats"


on:
 pull_request:
   branches: [ main ]


jobs:
 build:
   runs-on: ubuntu-latest
   strategy:
     matrix:
       node: [ 20 ]


   name: Node ${{ matrix.node }} sample
   steps:
     - uses: actions/checkout@v3
     - name: Run linting rules and tests
       uses: actions/setup-node@v3
       with:
         node-version: ${{ matrix.node }}
     - run: npx @fdiazpaccot/webpack-js-difference -c
       env:
         GH_TOKEN: ${{secrets.GH_TOKEN}}
         GH_USER: ${{secrets.GH_USER}}
         GH_REPO: ${{secrets.GH_REPO}}

When running it in a Continuous Integration (CI) environment we would no longer be able to see the outputs of the console.table command that we perform directly in the console, without going to see the action itself. Therefore, in order to simplify and quickly understand the results, it would be necessary to add a comment in the Pull Request with the result of the action. To accomplish this, we can use @octokit/rest to add the comment.

.github/workflows/analize.yml within the application to measure

async function addComment(values: any[]) {
  const token = process.env.GH_TOKEN;
  const user = process.env.GH_USER
  const repository = process.env.GH_REPO

  let comments = ''

  values.forEach((value: any) => {
    comments += `| **${value.type}** | ${value['base size (Kb)']} | ${value['PR size (Kb)']} | ${value['Difference (Kb)']} | \n`
  })

  if (!token) return

  const octokit = new Octokit({ auth: token });

  const eventData = JSON.parse(fs.readFileSync(process.env.GITHUB_EVENT_PATH, 'utf8'));

  const pullRequestId = eventData.pull_request.number;

  let comment = `**These are the bundle size changes in this PR.**

| Type | Base size (Kb) | PR size (Kb) | Difference (Kb) | 
| :--- | :----- | :------ | :------- |
${comments}`

  octokit.issues.createComment({
    owner: user,
    repo: repository,
    issue_number: pullRequestId,
    body: comment
  })
}

When the github action finishes, we will see a result similar to the following:

What has been developed up to this point is only the basis. As much as we want to extend this solution, it is subject to how much we want to go deeper into this solution.

We could analyze the chunks that each asset consumes when adding more JavaScript, and even identify the exact entry point where this occurs.

I hope you find this information useful to avoid uncontrolled injection of excessive amounts of JavaScript into your applications!

Thanks for reading :)

Python and Selenium, Unleashed without Docker

Francisco Diaz Paccot — Thu, 01 Feb 2024 14:45:21 +0000

Deploying Python + Selenium applications without the need for dockerize it.

In recent days, I have been dedicated to the development of a Python-based Application Programming Interface (API), requiring the integration of Selenium. When it came time to implement the solution on a server, I faced various limitations in efficiently running Selenium in that environment. I found that many of the proposed solutions to overcome this challenge involved dockerizing the application; however, given the peculiarities of my use case, I chose to forgo this option and instead sought a solution that would allow deployment in a native Python environment.

In this way, I discovered that by using render.com as a hosting platform, it is feasible to execute bash scripts on the server intended for the implementation of our application. This capability provides us with the opportunity to install Chrome directly on the mentioned server.

For this purpose, through this simple bash script, we can install Chrome on our server if it is not already present.

#!/usr/bin/env bash
# exit on error

set -o errexit

STORAGE_DIR=/opt/render/project/.render

if [[ ! -d $STORAGE_DIR/chrome ]]; then
  echo "...Downloading Chrome"
  mkdir -p $STORAGE_DIR/chrome
  cd $STORAGE_DIR/chrome
  wget -P ./ https://dl.google.com/linux/direct/google-chrome-stable_current_amd64.deb
  dpkg -x ./google-chrome-stable_current_amd64.deb $STORAGE_DIR/chrome
  rm ./google-chrome-stable_current_amd64.deb
  cd $HOME/project/src # Make sure we return to where we were
else
  echo "...Using Chrome from cache"
fi

pip install -r requirements.txt

# be sure to add Chromes location to the PATH as part of your Start Command
# export PATH="${PATH}:/opt/render/project/.render/chrome/opt/google/chrome"

Upon concluding the script, we include the installation of the necessary dependencies.

In the final stage, it is crucial to navigate to the settings in the "Build & Deploy"section of render.com. Specifically, look for the option labeled "Build Command", where the execution of our bash script is specified using the command:

./render-build.sh

An additional relevant aspect involves adding the following command in the "Start Command" section:

export PATH="${PATH}:/opt/render/project/.render/chrome/opt/google/chrome" && gunicorn app:app

In this instruction, the first part establishes the path where the Chrome executable is located, and we use gunicorn to run the application.

By implementing these steps, it is expected that our Python application can run Selenium seamlessly, having properly configured the render.com environment, installed Chrome, and defined the necessary dependencies in the build script and start command. This set of measures provides a solid framework for the smooth operation of the application in the desired environment.

Github: https://github.com/FranciscoDiazPaccot73/selenium-python-example

Deploy: https://python-selenium-example.onrender.com/test

DEV Community: Francisco Diaz Paccot

Webpack stats file

1- Prerequisites

2- Application to be measured

3- The CLI

4- CLI from a Github Action

Thanks for reading :)

Python and Selenium, Unleashed without Docker

Deploying Python + Selenium applications without the need for dockerize it.

Thanks for reading :)