DEV Community: Ali Sherief

How to avoid "Too many open files" error in NodeJS

Ali Sherief — Tue, 02 Sep 2025 08:30:35 +0000

Hi guys, I know it's been quite a long time since I last posted here, but I have had this one development problem that had been bugging me for several months, until I finally found a solution for it this morning.

It specifically affects Linux desktops, but might also apply to Macs.

You might notice when you have a lot of browser tabs open like Chrome, Firefox, Opera and so on, when you try to use npm run dev on your Node project to run it locally, you get a hundred errors that all look like this:

Watchpack Error (watcher): Error: EMFILE: too many open files, watch '/home/zenulabidin'
Watchpack Error (watcher): Error: EMFILE: too many open files, watch '/home'
Watchpack Error (watcher): Error: EMFILE: too many open files, watch '/home/zenulabidin/Documents/talksearch.io/app'
Watchpack Error (watcher): Error: EMFILE: too many open files, watch '/home/zenulabidin/Documents/talksearch.io'
Watchpack Error (watcher): Error: EMFILE: too many open files, watch '/home/zenulabidin/Documents'
Watchpack Error (watcher): Error: EMFILE: too many open files, watch '/home/zenulabidin'
Watchpack Error (watcher): Error: EMFILE: too many open files, watch '/home'
Watchpack Error (watcher): Error: EMFILE: too many open files, watch '/home/zenulabidin/Documents/talksearch.io/app'
Watchpack Error (watcher): Error: EMFILE: too many open files, watch '/home/zenulabidin/Documents/talksearch.io'
Watchpack Error (watcher): Error: EMFILE: too many open files, watch '/home/zenulabidin/Documents'
Watchpack Error (watcher): Error: EMFILE: too many open files, watch '/home/zenulabidin'
Watchpack Error (watcher): Error: EMFILE: too many open files, watch '/home'

And it just keeps printing this continuously forever. Alternatively, if you are not using NextJS, it will simply terminate npm.

This can be a very frustrating error to deal with if you have many open programs at once and don't want to close most of them, especially the browser.

The conventional solution that you will find on Stack Overflow and blogs is to change the maximum file limit of the current user, using ulimit programs. I will not cover these here, because they did not work for me. The problem was still the same.

What is the cause of Webpack EMFILE errors?

The "too many open files" error is almost always caused by having too many inodes open on your system. An inode is just a file in Linux. It is the internal representation of the file.

Sometimes if you use VScode, you see this error very frequently and then it gives you some instructions on how to resolve the error by running sysctl commands in your terminal. It is the same type of error, and it is especially prevalent when using Webpack inside React projects.

How to fix Webpack EMFILE errors on Linux

This is an approach I have only found when I was using Claude Code. I specifically instructed it to 'find a way to work around too many open files errors when running npm'.

What I have found is that it used set the environment variable: WATCHPACK_POLLING=true inside the terminal before running npm.

If you only want to run it once, use the following command: WATCHPACK_POLLING=true npm run dev.

Otherwise, if you want it to apply to all runs of npm run dev inside this terminal, run the command export WATCHPACK_POLLING=true in the shalle. Or place it in your .bashrc file.

That is all. I hope you found this helpful. Don't forget to follow me on X (@Zenul_Abidin) for more posts like this.

Formatting HTML and CSS For Beginners

Ali Sherief — Fri, 11 Oct 2024 09:13:26 +0000

Well, I see it's been a long time since I last posted here. A lot of interesting stuff have happened over the past few years! But let's get back to the subject - you want to know how to format your HTML and CSS correctly, and this is how you're going to do it.

Some Benefits of Styling with CSS

Many of these benefits also translate to HTML too, when you organize your markup using HTML5 elements.

By keeping style separate from structure, CSS allows for cleaner, more maintainable code. This separation improves content accessibility and makes styling changes across a website much more manageable.
CSS provides extensive control over the presentation, from fonts to layout, enabling developers to create responsive designs that adapt across devices.
External CSS files can be cached, improving load times for subsequent page views.

Techniques for Applying CSS With Your HTML

CSS can be applied in three primary ways:

Inline Styling: Using the style attribute directly within HTML elements. While this method is straightforward, it's generally discouraged for its lack of maintainability.

   <p style="color: blue;">This text is blue.</p>

Internal Styling: Using a <style> tag in the HTML <head>. This method is useful for small projects or unique styling for a single page.

   <style>
   p {
       color: red;
   }
   </style>

External Styling: Linking to a separate .css file, which is the most recommended for extensive projects due to its maintainability and reusability.

   <link rel="stylesheet" href="styles.css">

In styles.css:

   p {
       color: green;
   }

Best Practices

Use Semantic HTML, as I wrote above.
Avoid Inline Styles.
Prefer Classes Over IDs when creating selectors.
Consider organizing your CSS properties alphabetically.
Use class names that are descriptive but avoid overly specific selectors that might conflict or be hard to override.
Use Media Queries. They're good for responsive design. I cannot stress this enough.

On a different note...

Did you know that I am running a free eBook giveaway on X/Twitter? You will learn how Google formats their own HTML and CSS and their own advice. If you like free stuff, then check out my tweet.

Electron vs. The Native Windows Frameworks

Ali Sherief — Mon, 13 Sep 2021 05:38:03 +0000

Electron is starting to be used in many different popular applications, such as VSCode and WhatsApp, simply because of its cross-platform feature. But is that the only advantage that Electron provides?

As we will see by the end of this article, it is not the only benefit. We will look at how simpler it is to write a Windows app in Electron vs. native frameworks such as C++/C# WindowsForms and WPF. Why just Windows? Because the vast majority of programs are written for it, so for many categories of programs (e.g. games) there is a standardized library stack that's used.

Background (tray) apps that display modal dialogs

Electron is probably overkill for this category. Modal dialogs are simple to create using WindowsForms, and C# also makes minimizing windows and displaying their notification (tray) icon very simple. This can also be done in Electron, but with a little more work.

Apps with a custom theme

Because WindowsForms does not support making custom window themes, developers use the WPF framework when they want to create a custom theme for their C# app. This process could take a lot of time, depending on the complexity of the theme.

Of course, theme-designing in Electron also takes time, but it's helped by the abundance of HTML UI components that can be installed and inserted directly from npm, the NodeJS package manager. Only some WPF UI controls are free to use, while the vast majority remain closed-source, internal, and private, away from the public eye. While most HTML UI components are open-source and easy to obtain from Github.

Apps with heavy API/web service integration

Messaging clients such as Discord, Telegram, and Slack fall into this category. On C#, networking is done using the System.Net namespace, or more usually, with the ASP.NET framework. In C++, more low-level socket functions are used, hence why complex networking is traditionally done in C#.

System.Net has classes to manage cookies, network credentials, requests and responses, and HTTP listeners, among others. When used correctly, formidable network-enabled C# programs can be developed.

A positive trait of C# networking classes is that the compiler will detect most of the incorrect usages of these classes, which can save you hours of runtime debugging. This is not the case for Node.js and Electron apps, where all code is evaluated at runtime, and could silently fail then.

Worse (for Electron), as the bundled code is usually minified, it is difficult to get a stack trace that corresponds to your source code.

That is not to say that Electron is not good for building network apps. On the contrary, libraries such as Axios and node-fetch take care of networking very well. Asynchronous requests are also (slightly) easier to perform on NodeJS due to the language architecture. Additionally, NestJS is often used to make entire back-end APIs.

Video games and other GPU-intense apps

While HTML5 games built using WebGL-emitting game engines such as Unity are starting to emerge, C#/C++ and WPF still dominate the video game industry, due to the heavy use of graphics frameworks such as DirectX and OpenGL. In particular, there is no easy way to integrate Electron and DirectX, because the latter is a Windows-specific technology.

CAD & designer programs, and other kinds of graphics-accelerated programs, make use of OpenGL instead of DirectX, and since Electron apps can use WebGL, the level of difficulty of developing these kinds of applications in both platforms is about the same.

Microsoft Store apps

Here is a surprise: You can publish Microsoft Store apps that are built with Electron. They will need at least Windows 10 1607 to run, and they will compile to AppX packages that you can publish on the Store after Microsoft manually verifies the application.

So in this area, it doesn't look like there are advantages or drawbacks to using either language, because the end-product is the same - with the exception that C# can make UWP apps, but Electron can't, but to be fair, most people are not running UWP apps on the alternate platforms such as Xbox anyway.

The conclusion of all this is that whether you choose to write your app in C# and WPF or in NodeJS and Electron depends on whether you plan to port it to browsers eventually, or whether you want to use GPU acceleration. C#/C++ is the clear winner for GPU-accelerated programs, while for deploying your desktop app to the web, Electron is the way to go.

Desktop app deployment to web browsers has some uses. For instance, if you make an open-source code editor using Electron, then services that let you run and test code online can embed your editor inside their service, for people to use on the web. This is actually how Stackblitz is embedding VScode and a Chromium browser window in its web IDE.

Cover image by Envato Elements

Google Chrome Media History: How Does It Work?

Ali Sherief — Mon, 06 Sep 2021 18:52:30 +0000

The first thing you're probably wondering after reading the article title is, "What is Media History?" Indeed, there is no such feature when looking externally at Google Chrome. However, this is just the name of a database that is stored in Chrome's local storage. This database stores a history of URLs on which you played an audio or video stream on, hence the name "Media".

Now, it does not store the media files themselves, but just the length of the time played or watched, and the URL it was playing on.

So, let us see what are the high-level parts that make up the Google Chrome media history feature.

The source code for the Media History feature lies in the chromium/chrome/browser/media/history/ folder of the repository. Here, I have linked the Github mirror of the chromium repo, because it's much easier to browse than Google's own source listing.

The Chromium Media History feature consists of these parts:

Contents Observer
Images Table
Keyed Service
Keyed Service Factory
Origin Table
Playback Table
Session Images Table
Session Table
Store
Table Base

Each of these items resides in its own file and we'll be taking a look at them one by one.

The Elements

Contents Observer

It implements the Observer pattern documented in the famous Design Patterns book by the Gang of Four. As a recap, directly from Wikipedia - Observer pattern:

The observer pattern is a software design pattern in which an object, named the subject, maintains a list of its dependents, called observers, and notifies them automatically of any state changes, usually by calling one of their methods.

Here, the Media History is notifying its subscribers when new events happen. Among the possible events to listen to, are the DidStartNavigation, DidFinishNavigation, and WebContentsDestroyed events, inherited from the WebContentsObserver parent class, which we shall cover on a later date.

Novel event methods include MediaSessionInfoChanged, MediaSessionMetadataChanged, MediaSessionActionsChanged, MediaSessionImagesChanged, and MediaSessionPositionChanged which alert subscribers when those respective fields change. It has fields for whether media has ever been played on this particular WebContents object, and whether a web navigation is occurring right now to pause state updates.

These event callbacks also update any caches that have these old objects inside them.

Images Table

This is an SQL table entity that saves information about images into the database. It contains a method to export images to the database and returns their ID.

Keyed Service

The Keyed Service holds a pointer to the Store object (more on that later), which could either be at a local or remote place. Each Keyed Service also has one pointer to a Google Chrome profile.

Keyed Service objects have methods to save watch times and stats to the Store.

Keyed Service Factory

This is the Factory pattern from the Design Patterns book, applied to keyed services. As you have probably guessed, this class is in charge of creating Keyed Service objects.

Origin Table

It is like the Image Table but it stores URLs in SQL-like tables.

Playback Table

Similarly, the Playback Table exports the timestamp and watch time specifically to the data store.

Session Images Table

This is a supporting table that correlates the image ID with the session ID.

Session Table

Yet another table, this one saves URLs, metadata, and the position of the media being placed.

Store

This object contains the actual pointer to the database, as well as to each of the tables.

Table Base

This is the parent class of all of the other table classes in this section. It is responsible for initializing the database member object inside the table object.

As I briefly hinted in the last section, each Table object has its own database pointer. Consequently, when you save an entry inside the Chromium Media History, it is assigned to one of these tables, and as a result, one of the databases. There is no single Media History database: It is split into several different databases for each table.

Here is a diagram of how these components of the media history connect together.

MSYS2 Build Processes Driving You Crazy?

Ali Sherief — Mon, 30 Aug 2021 16:32:14 +0000

At work, I have the task of making builds for a particular cryptocurrency wallet for multiple platforms. Among them is Windows, and we don't build with Visual Studio - instead, we using CMake with MSYS2 so we don't have to rewrite the whole codebase just for Windows-specific edge cases.

There was just one problem though. When I was assigned to this, the codebase just wouldn't pass the CMake configuring process on Windows according to the build instructions. There didn't seem to be anything wrong at first, but later on, there were a few red herrings of an error that threw off my debugging.

Eventually, I was able to nudge it past CMake. Here's how I did it.

Some background information about MSYS2

You don't actually build applications inside MSYS2. You build them inside MinGW - that's because MSYS2 is just a shell. That's right, MSYS2 only provides you with a bash-like shell, and the rest of the GNU tools are provided by MinGW.

In particular, all the build tools are only available if you run the MinGW version of the MSYS2 shell. So every Windows project's build instructions that don't use Visual Studio will require you to build it under MSYS2.

MSYS2 and MinGW use the pacman package manager to install software. There are two sets of packages (well, there are actually more than that, but I'm not counting the 32-bit or Universal C Runtime or cross-platform packages). The mingw-w64-x86_64-* packages install in MinGW's "namespace", and the unqualified (with no crazy prefixes) packages install in MSYS2's namespace.

Unqualified packages are visible from the MinGW program, but apparently, the mingw* packages are not visible from MSYS2, but please let me know in the comments if this is wrong, because I discovered this from trial and error.

The problem

CMake Error: File <REDACTED>/external/unbound/win32pthread-NOTFOUND does not exist.
-- Could NOT find Readline (missing: Readline_INCLUDE_DIR)
Doxygen: graphviz not found - graphs disabled

Do any of these errors look familiar to you? Well apparently, Google returns a few hits on them, but doesn't really offer answers for any of them.

Some of these errors can be resolved by installing the relevant mingw-* packages. This is the case for Doxygen and graphviz not being found. Others, like the first two, are more subtle and cannot be resolved just by installing the package - there is a little more work you have to do in your CMake files.

Only the pthreads error was fatal, but all of them are equally annoying. Let's start with the Readline error first.

First though, ensure that mingw-w64-x86_64-readline is already installed. The same goes for the Doxygen and graphviz packages. The MinGW packages are necessary, not the unqualified MSYS2 packages.

Your Readline package configuration probably looks something like this:

A call to find_package, followed by a check whether Readline is found (and a diagnostic printing that it will build without Readline support otherwise).

The problem here is, find_package never finds the path to Readline. This is strange, for two reasons. One, find_package has no problem finding other packages, and two, the Readline library is already installed inside MinGW. So what could be the problem?

After doing some inspection, I discovered that find_package does find the Readline library, but not the include path. Apparently, this is because it was told to look for a specific filename (readline/readline.h) that had a forward slash in it, as the path separator.

MinGW programs don't understand UNIX paths because they are programmed to understand only Windows paths. This is to make them work under Windows.

To make things worse, MSYS only works with UNIX paths, not the Windows paths that MinGW programs use.

This causes some confusion in CMake, because the find_package function ultimately just calls the find(1) program, which is a MinGW program and only understands Windows paths.

Worse, CMake itself will only take UNIX paths for file specification no matter whether you install the MSYS or MinGW version of it. This leads to a situation where you have to mix Windows and UNIX paths in the same configuration file, which can get quite messy.

I could've gone that way and modified the file responsible for finding the Readline package, but since that wouldn't be portable, I decided to implement a fallback condition instead, just for MinGW.

As you can see, there is a special condition in the fixed code that detects the case where find_package can't find Readline, and if the program is being built under MinGW. In this case, I just emit a warning, hardcode the path that Readline is supposed to be, and proceed with the rest of the CMake file.

This particular case also has checks for 32- and 64-bit MinGW.

As I mentioned before, the pthreads error was the one that halted CMake execution, and not any of the other errors. In this case, it complained that it could not find the pthread library under Windows.

This can get confusing, because Windows doesn't use pthreads - It uses it's own OS-specific threading library. So MinGW emulates pthreads on Windows by shipping a pthreads .dll library specifically for Windows.

This error is basically saying it can't find that file.
First, ensure that mingw-w64-x86_64-winpthreads-git is installed - this will install the pthreads development files, and will also pull the *-libwinpthread package, which provides the /mingw64(or 32)/bin/libwinpthread-1.dll library that needs to be found.

Here, we have to use Windows paths to specify this file, because the (unshown) copy function that happens immediately after this search will not work with UNIX paths.

The result will look something like this, along with warnings and 64-bit checks:

You should now be able to build your codebase without these errors displayed, and the compiler will automatically be able to find these files itself.

Cover Image by Envato Elements

The 4 Parts Of The WebUI: View Google Chrome’s Code

Ali Sherief — Mon, 23 Aug 2021 00:07:07 +0000

This is the first of a series of posts that will dive deep inside Google Chrome's codebase and understand how it works as much as possible.

I believe that it's a great idea to know what on earth is going on in the software you run, the first and foremost of them being your browser, especially as a flurry of universal security vulnerabilities were discovered inside Chrome recently.

Google Chrome is a slightly modified and reskinned copy of Chromium, an open-source browser. This means that source code is available, which makes this investigation possible.

The objective is not to understand every single function and line of code - it's to get an overview of the high-level components that make up the software.

What is the WebUI?

It is a special web page that has access to the Google Chrome internal machinery itself. For example, the New Tab page, the Settings page, the Extensions page, etc.

The URLs that use the chrome:// prefix (which is a protocol in its own right) are all WebUIs. They are scripted using the same web technologies used in normal web pages such as HTML, CSS, and Javascript, but they have permission to access browser internals.

When I say "internals", I am referring to the renderer, which I will talk about later. For now, it suffices to say that it is the component that generates a bitmap from the HTML and CSS and draws it on the screen.

The renderer is usually written in some form of Javascript. You might have heard of the V8 engine (not to be confused with the car engine of the same name). This is the renderer that powers not only Chrome, but also NodeJS. It has some methods which allow you to run arbitrary Javascript on it [WebUIMessageHandler::CallJavascriptFunction()] and send messages to the window object using chrome.send(). This is used to implement things like service workers.

Google Chrome pages are sandboxed

The vast majority of pages have no access to the filesystem or any part of the world outside the browser whatsoever. Since implementing access controls on each and every tab would be prohibitively slow, this sandbox is enforced on the renderer.

As a result, the renderer (and all the pages under it) can only draw stuff to the screen but not interact with the OS. They cannot access the camera or microphone. They cannot show you popups or use your on-device location services. Chrome even supports USB and serial port interfacing - which is not surprising when you consider it's the basis of ChromeOS - and obviously, a page has no access to those either.

I can make a long list of resources that are off-limit to pages, but you can find many of them by going to the Site Permissions area for that particular page.

The WebUI as an attack vector

Most security vulnerabilities like the one you saw above rely on breaking through the sandbox using maliciously crafted Javascript served in a webpage that bypasses the renderer.

WebUI pages are special because they are not sandboxed like this. This makes them an attractive attack vector, especially when you consider Chrome supports extensions. See that New Tab Page extension in the results for example? It can technically replace the chrome://newtab WebUI with its own page.

Also, since WebUIs receive all the service worker messages, it's possible that a malicious service worker sends something that's designed to gain access to some internal Chrome functionality.

These issues can be mitigated by disallowing WebUIs from loading random HTTP(S) fetches and data URLs. If they need to embed content from an external page for some reason, such as the Google Doodles, they can use iframes for that, which are sandboxed like a standard web page.

Parts of the WebUI

Chrome makes a WebUI by creating a C++ class of the same name, along with a WebUIDataSource, a WebUIMessageHandler, and a WebUIController which is responsible for pulling the strings that make the puppet move (figuratively). All of these are neatly documented consecutively on this page.

The WebUI represents the actual super-privileged Chrome page. The WebUIDataSource adds resources to a WebUI that it can access later, for example, images or translation strings. The WebUIMessageHandler receives Javascript events on behalf of the WebUI.

To summarize what we have learned, I have made this simple diagram of linked concepts. Who ever knew that learning parts of your programs could be so simple?

Image by NotATether.com. Do not share without attribution.

What's all this fuss about spying with light bulbs?

Ali Sherief — Mon, 16 Aug 2021 03:27:46 +0000

At first, hearing this might sound ridiculous. How could a hacker possibly monitor your activity through a light bulb?

Except that's exactly what I read on Forbes the other day. It particularly affects virtual assistants such as Alexa, Google Home, Siri, and any other microphone input (such as meeting apps) and involves you, unwittingly, literally shining a light at the microphone piece.

Now obviously, shining any old light at your home assistant isn't going to do anything, it's a special type of exposure that was found to cause the vulnerability. In the research paper Lamphone: Real-Time Passive Sound Recovery from Lightbulb Variations, the researchers demonstrate how "dumb" light bulbs are sensitive to sound and make different vibrations depending on the audio coming out from the home assistant.

Theoretically, any object in the room could've been used to record vibrations, but light bulbs are the only ones that reflect any light to a long distance, and this attack needs light rays to work properly.

The attack method

The reason this method works is that it emulates how a microphone works. You see, a microphone consists of a diaphragm that converts analog sound into vibrations, a transducer that runs the vibrations through a magnetic field, turning them into an electrical current, and an Analog-to-Digital Converter (ADC) that samples the current using frequencies (e.g. 44.1KHz) that the device can understand. As far as the device is concerned, this is intelligible audio to them.

Obviously, the diaphragm part that makes vibrations can't be replicated by some thieving machinery far away, but those vibrations can be "stolen" from the room with the help of an extremely close light bulb. This technique will not work with light bulbs more than a few centimeters away from the microphone piece because the intercepted sounds will be full of unintelligible noise.

Also, it will not work if the light bulb has an enclosure, as most home lights have, that limits light radiation, since that reverts it to a normal household object that can't be inspected without lots of noise.

So don't worry about this device gleaming audio from your ceiling light. It simply is not possible (yet).

Basically, instead of vibrations, light signals are fed through a makeshift transducer and ADC some close distance away. This can record people's voice input.

To be honest, the attack is only practical if you're somewhere within 25 meters (a little above 80 feet) of the target device. Understandably, this means that you have to be really close to the target building to record a strong enough light signal. So it's only practical if attached to places such as your neighbor's building, or in the case of the researchers, a bridge.

But if you had those high-security requirements, you wouldn't be placing sensitive stuff near a public bridge, now would you?

The setup used to spy on audio in a room.

Fortunately, as fancy as this vulnerability sounds, there's one obvious patch for it. Closing your curtains or blinds. This has been the standard privacy guard for decades for sensitive stuff you don't want neighbors and snoops listening on or seeing. The researchers have also implied that opening the door might produce unintended side effects.

(Also, if you see giant telescopes outside pointing directly at your building, you should definitely go out and enquire about that. It's not like there are scientific planetary objects of interest in your room.)

The power LED to the audio

Yesterday, another article was published on Forbes that describes how a variant of this attack, aptly called "Glowworm", can be carried out using power LEDs instead of light bulbs.

This video linked in the article demonstrates how the "Glowworm" attack works.

The Forbes author also points out in the article the no-brainer precautions also work for power LEDs as they do for light bulbs:

You've probably already jumped to the most obvious mitigation conclusion: as Glowworm requires a clear line of sight to the power LED, closing the curtains, turning speakers around to face away from any window or sticking a piece of, oh the irony, electrical tape over the LED will all kibosh it.

So should you be concerned about these things? Probably not. As well as having ridiculously simple precautions, it needs specialized equipment that you can probably see from your window, and take the appropriate actions: closing blinds, calling the police on that location, and such.

Cover Image by LightFieldStudios from Envato Elements

Pro Tips For Designing Robust React Components Part II: Bundle Size

Ali Sherief — Mon, 09 Aug 2021 03:47:58 +0000

A few weeks ago, I shared some tips for improving React app performance. Let's see how another metric can be improved - the bundle size.

Pro Tips For Designing Robust React Components

Ali Sherief ・ Jul 19 ・ 3 min read

#react #javascript

Why is bundle size important?

Because that affects how quickly your app will load when a user opens your page. It is vital because many users will probably be connecting from dodgy 3G or wireless connections, where the speed is slow, and thus a small bundle size is essential so that users don't leave your site. Users tend to leave a site if a page takes longer than 3 seconds to load. The 2-second threshold is the "danger zone" where most users expect the app to be fully loaded within that time and begin to get impatient if it isn't.

Granted, React app loading is not symmetrical to page loading - generally, you can load a bunch of HTML and CSS much faster than a React.js bundle file. However, load time is still important even though you have a slightly longer amount of time to render the app. So while users will forgive you for your app taking 10 seconds to render, the same cannot be said for 60 seconds, 45, and possibly even 30 seconds.

Nobody's expecting you to render a React app in 2 seconds, though if you can, then your team should throw a pizza and beer celebration. For the rest of you, here are some techniques to shrink the bundle size.

Split your bundles into smaller ones

This is a very powerful technique to make the app load faster because instead of one large bundle, it is now a bunch of smaller ones that Webpack can load on-demand. So, you can package your app's dashboard as a bundle that loads immediately and delay the loading of bundles representing other auxiliary pages. I imagine this is what Facebook, Instagram, and others use to keep the load time of their main sites - which are written in React - manageable.

Splitting bundles is a feature available since Webpack 4. Apps made nowadays likely are not building using Webpack 3 or lower, so there should be no worries about upgrading to a slightly incompatible version.

How does code-splitting work?

The Webpack documentation gives us 3 methods to implement code splitting. The first one uses entry points using entry configuration lines in the Webpack config. This basically means that each component tree you want to split off has some ancestor component in a specific file referenced in the Webpack configuration. The entire tree is bundled down to a single bundle.

This is how you use entry to define different bundles Webpack needs to make:

You must include dependOn: 'shared' for all the bundles, and then list any external libraries you're importing as dependencies and the file name of each component used by the multiple component trees. Otherwise, the shared dependencies are duplicated in both bundles and defeat the purpose of code splitting. The lodash dependency in this example will occupy more than 500KB in each of the created bundles without shared dependencies.

Of course, it is usually not feasible to place all your shared components in one file. Whatever you write in the dependOn: the directive will have a key right below the entry object, such as shared in this example, and is an array of strings if a bundle has multiple dependencies. Creating multiple dependOn names for different bundles allow you to define multiple shared entry points whose paths reflect the structure of your React app.

Refactor long lists of content as separate XHR calls

If you have any long arrays of text strings in your React app, these might be weighing down the load time. Try to make an API endpoint to supply this data, then use node-fetch to retrieve it at runtime, using a progress indicator as a placeholder while the request completes. You can use this alongside code splitting to fetch the content before additional bundles are loaded, which reduces the time to render before a user can interact with the app.

The react-window module has been designed to fetch long lists of content. It has an additional performance optimization, however. Instead of fetching the entire list, it only fetches the amount that fits in the viewport and then issues a DOM update. This is useful if, for some reason, your list, along with all its properties, is several megabytes large. It happens sometimes.

Additionally, you might be able to set up your API endpoint to prefetch the request, which will make the server cache the response by the time you're ready to issue the actual API call. In some cases, this can speed up the time it takes to fetch long lists of content.

Use tree-shaking

Tree-shaking is a process that eliminates dead code from bundles. For this to work, you must only import the functions you need from modules (i.e., don't import the whole thing), and you have to place "sideEffects": false in your package.json on the same level as the name property. You can also add it in the Webpack configuration file under the rules property object.

A side effect is any module that, when imported, runs some background function in addition to importing items from the module. Webpack wants to make sure that removing unused functions from the bundle does not accidentally inhibit important code from running. If there are such modules, you should include their file names as an array of strings in the sideEffects property and Webpack will keep those in the bundle.

Note that for this to work, you have to use ES2015 import syntax in your files.

Use service workers

Applicable to all kinds of web apps, not React apps per se.

A service worker is a Javascript file that a page deploys in the background. It "installs" this file by caching all files specified in the "install" event listener. Then, it communicates with the page by sending a window.postMessage()` call, whose data is then intercepted by the "message" event listener on the webpage.

But how does a service worker know which page to communicate with? It turns out that postMessage() also takes an origin parameter that tells the browser which pages it should broadcast the message to. So tabs in a browser window that have the same origin will all receive the message.

So service workers don't really do one-to-one messaging unless there's only one matching page. Think of it as a publish-subscribe channel where all open pages of the same origin will get the data received in the message. Remember that an origin is a tuple of hostname or domain name, port number, and protocol (HTTP or HTTPS).

Service workers can improve app performance by caching the files specified at install time and then returning them in the "message" payload to open the page. These files are effectively cached in the browser, so apps can use this method to read items such as CSS files, fonts, and other dependencies defined in the HTML, such as <script> tags. It doesn't work well for caching bundles (use the Webpack server instead), Also without specifying the origin, you create security holes in your app.

Google Developers has some terrific code samples for service worker events in their documentation. They also have an older tutorial that explains how service workers work.

I hope this post has benefited you in your quest to make your React app faster. If you have any other performance ideas, let me know about them in the comments below.

Cover Image by @bobbinio2112 via Twenty20

Musings about Database Indexes

Ali Sherief — Mon, 02 Aug 2021 13:13:49 +0000

And not "Indicies"; I just took the Triplebyte backend developer test last night, and this was on the assessment.

Never take a Triplebyte assessment late at night. Your concentration falls if you're not blasting loud music in your headphones and the prolonged drowsiness dulls your comprehension as smooth as an unsharpened №2 pencil.

Notwithstanding all that, I didn't do so badly. In fact, I'm just shy of the 50th percentile of backend devs. But my DB indexes skills need some reflexing.

A caveat: I admit I'm not the best at handling databases, and while I got my relational DB theory and basic types, rows/column/keys, and SQL nomenclature out of the way, indices are a bit advanced for me, so we'll try to journey through it in this post, and hopefully by then we all will get the gist of them.

Also, before we begin, I would like to go through the subject of why they are called database Indexes and not Indices. The topic is apparently so confusing that even Nasdaq wrote an article about it. Technically either is correct since they are both plural forms of the word "index," but I see pages universally saying indexes so we will stick with that throughout this article.

TL;DR: the word "indices" paired with any word ranks higher on search engines. It is generally more ubiquitous on the web but we Americans have gotten used to saying the word "indexes" so often that we wrote them all over technical documentation. Sigh.

So… what exactly is a DB index?

Now that the naming hoo-ha is out of the way, let's turn our attention to the Wikipedia article about the subject:

"A database index is a data structure that improves the speed of data retrieval operations on a database table at the cost of additional writes and storage space to maintain the index data structure. Indexes are used to quickly locate data without having to search every row in a database table every time a database table is accessed. Indexes can be created using one or more columns of a database table, providing the basis for both rapid random lookups and efficient access of ordered records."

Woah. Let's break this down into detail. We have "improves the speed of data retrieval operations," which implies that indexes speed up queries. But how? And why not just improve table lookup speeds instead?

How does DB indexing work?

From what I gathered, an index stores an entire column into an internal structure with faster read times than the database table itself. You can store any column you want inside an index using the following SQL statement:

CREATE INDEX <index_name>
ON <table_name> (column1, column2, ...)

This is not universal syntax - the exact command might vary depending on the database management system used.

It also allows for multiple columns to be indexed simultaneously, potentially speeding up read times for values in those columns.

It's like a mini-table with just those columns but sorted on the first column column1. This sorting feature is the crucial part that makes the whole indexing system work. A DB already knows where a particular primary key is, so having the ability to find the rest of the filtered columns by another "key" allows the entire (filtered) row to be fetched rapidly.

You may not know this, but primary keys are already stored inside indexes without any additional coding on your side.

However, it is worth noting that while indexes are read faster than tables, they are slower at writing than tables. Hence, do not create indexes on tables frequently updated with new rows or new column values.

So now that we know when not to use indexes let's look at an analogy that demonstrates its practical use.

A Book Index

A brilliant example of an index is given by Stack Overflow user Sankarganesh Eswaran:

Classic example "Index in Books"
Consider a "Book" of 1000 pages, divided by 10 Chapters, each section with 100 pages.

Simple, huh?

Now, imagine you want to find a particular Chapter that contains a word "Alchemist". Without an index page, you have no other option than scanning through the entire book/Chapters. i.e: 1000 pages.

This analogy is known as "Full Table Scan" in database world.

But with an index page, you know where to go! And more, to lookup any particular Chapter that matters, you just need to look over the index page, again and again, every time. After finding the matching index you can efficiently jump to that chapter by skipping the rest.

But then, in addition to actual 1000 pages, you will need another ~10 pages to show the indices, so totally 1010 pages.

In the case of a database, the book index could be a table with the entry and page number as columns with some other metadata columns (for example, whether it is a child entry of another entry) and an ID primary key. But the entry and page number can be stored by themselves in an index. And since there will only be one unique entry in a book index, that can be used as the sorting column of the index as well.

Source: https://stackoverflow.com/questions/1108/how-does-database-indexing-work

Most of the time, when people read about a topic, some of it tends to stick in their heads, so at that next Triplebyte test in four months time (which you hopefully are not taking at night!), you'll at least have a cursory idea of what database indexes are.

Cover image by @Zozulinskyi via Twenty20

Who's responsible for securing software?

Ali Sherief — Mon, 26 Jul 2021 18:34:25 +0000

A Venify report showed that the number of companies where DevOps is responsible for the security and the number who assign this to their Infosec department is almost evenly split. But should there really be this ambiguity regarding the responsibility of software security?

When you look at the situation on paper, you might say that Information Security should find holes in the apps because they are normally penetration testing the company's running software anyway. But equally compelling is to claim that because DevOps knows how the software works and is likely one of the few units with access to the codebase, they should be at work making sure hackers cannot exploit the software.

From this perspective, it seems that both of these views are right. DevOps should patch their programs - and make sure there are none of those pesky off-by-one errors - while Infosec has the knowledge necessary to send a running program off the rails.

So for you and me, it seems obvious to us what the roles of those two departments should be. But what about who takes responsibility when a cyberattack happens?

I'm not really the one posing this question. Executives often assign this taking-in-charge to either DevOps or Infosec, provided that both exist. We don't see most businesses sharing collective blame when they are hacked, right? So why don't they just make one department responsible for protecting the company from breaches?

That would be a poor approach to the problem because merely pentesting the program for vulnerabilities ignores the fact that hackers can try a completely new attack vector to enter the organization. And while DevOps can guard against basic flaws such as null dereference, they don't have the time that Infosec has to run different tests to catch new vulnerabilities continuously.

I don't believe that incident response should be a blame game and that all departments which play a part in the application interacting process should take responsibility to protect the software they use at their company and write, not only for their own company but for clients that also use the software in their products as well.

This post was originally published on NotATether.com.

Cover Image by maxxyustas from Envato Elements

Pro Tips For Designing Robust React Components

Ali Sherief — Mon, 19 Jul 2021 06:51:43 +0000

As you already know, React components are written as functions these days, not classes. Among other things, it allows us to dispense with binding methods and the this prop. But with both ways, you ultimately have to write a render method that returns a part of the DOM in the form of JSX.

They return a part of the DOM and do not generate a completely new one because the DOM is quite expensive to update, so developers try to minimize the number of DOM updates as much as possible.

Hence, most web developers reduce the number of components renders to a minimum to reduce the load on both the client's browser and the server.

Another important requirement of React components is that they update their UI fairly quickly. This prevents users from unnecessarily waiting on the app frontend and improves user experience.

Finally, it helps when your components are reusable. Not only do you avoid writing the same code twice, thus satisfying the DRY (Don't Repeat Yourself) principle, you can also be confident that each instance of your reusable, independent components will do a minimal number of re-renders.

In this article, and in the next few articles in this series, I will share with you some tips to reduce the number of renders your web app makes.

Try to partition the app so that each component is independent of others

The reason for this is, if your components are interdependent, then each state update in one component will likely require a state update in the other component. This causes a re-render, so you end up rendering multiple times when you do a higher-level component update. Ideally, you want to update components once per high-level update, but of course, this may not always be possible.

It would help if you tried to make each component partitioned in such a way that represents the UI layout of your app. For example, most apps have a header bar with buttons and links on it. So you should contain your button components in that location inside a header component.

Each component you create adds complexity to the whole application. You have to make sure the parameters are correct, and the returned JSX is what you expected, and, in the case of arrow or lambda functions, that they are defined in an order such that a function does not call another such arrow or lambda function above it in the file.

Try to make the nesting level of components as flat as possible. Although the way React updates the DOM ensures that nested components are not re-rendered if they have not been modified in a parent component, the advantage of making the component tree flat is that it makes it easier for you to debug each component by itself.

When to use prop destructuring

Prop destructuring in methods can greatly cut down the length of your prop variable names - if done properly. For one thing, it is not recommended to destructure multiple layers of props simultaneously (nested destructuring) because you cannot validate the data in the intermediate-level props, so that is a source for semantic errors.

It is not uncommon for components to have a few dozen props, so just the spelling of those props by itself will get messy when you write your functional component.

You should destructre your props, one level at a time, when there are a small number of them like this:

Credits: https://javascript.plainenglish.io/destructure-react-props-with-es6-object-destructuring-for-cleaner-code-3984453e484d

So that you avoid writing functions that continuously reference props like this:

Credits: https://javascript.plainenglish.io/destructure-react-props-with-es6-object-destructuring-for-cleaner-code-3984453e484d

Using another component as an example, we can do two different destructuring assignments to drill down the props, doing the equivalent of nested destructuring:

Credits: https://stackoverflow.com/questions/60589914/destructuring-props-in-react

In addition to that, the spread operator fits nicely on the right side of the assignment involving a destructured variables.

That's all for today folks. Stay tuned for next week's post where I write about pro tips for managing component state. Also, let me know in the comments below if you have any questions.

Cover image by Raphaël Biscaldi on Unsplash

Using Babel to write cross-browser modern ECMAScript

Ali Sherief — Mon, 12 Jul 2021 14:16:30 +0000

This is going to be a short write-up to demonstrate how Babel can make your modern JS more portable.

Here's a familiar situation you might relate to: You're a Javascript developer who wants to use ECMAscript 2015 features in your code, but you can't because you either a) have to support Internet Explorer or b) have to support mobile browsers. And believe it or not, with all the mobile-first craze, that's a very common concern!

So you're probably scratching your head thinking: "Well, how will I ever write ECMAscript 2015 like this?" Enter Babel.

Babel is an open-source JSX to JS compiler. So what it does is that it takes your element-wise JSX markup and converts it to JS that'll work on a particular browser. And what happens is you can insert your ECMAScript 2015 code inside the JSX and that will also get converted using polyfills for older browsers, and this allows you to utilize newer features that would otherwise result in a syntax error on some browsers.

Installation and Running

Babel actually consists of a collection of packages. The library itself is located under babel-core, while the commands are contained in babel-cli and extensions to babel start with babel-plugin-. You usually want to install only babel-cli and this will automatically pull babel-core, and only install one of the babel-plugin- packages if you need that particular functionality.

Since it is a NodeJS package, you'll usually want to install it locally, not globally. This can be done with npm install babel-cli. You also need to install the babel-preset-es2015 package as well because, without it, Babel will just spit out the input files onto the console. Then you can call Babel like this: babel src --preset es2015 --out-dir dir, where src is simply your root directory of source code, and "dir" is the folder where our files will be mirrored in but with the transpiled code, so that it's not sent to the console.

There are many other flags that can be passed to Babel such as -s, for source maps. Source maps are special files that assist in debugging Babel-compiled Javascript. Another useful flag is -w which means "watch the current folder for any changes in the files and automatically compile them". It's an integral part of making webapps automatically load after a code update. All of these settings, including presets, can be placed in a .babelrc file so that you don't have to spell them out on each run. See this page for details.

Also, by using libraries such as ui-logger-dom and services like BrowserStack, you can emulate any browser online to test your app on, and the console errors will be printed in the main (DOM) window instead of in the respective browser's console, which may be difficult to access.