DEV Community: Siddharth Venkatesh

Building a realtime performance monitoring system with Kafka and Go

Siddharth Venkatesh — Mon, 20 Feb 2023 15:04:12 +0000

Recently, I had a chance to try out Apache's Kafka for a monitoring service and I was pleasantly surprised how you could set up a full fledged event streaming system in a few lines of code. I quickly realised we could be building powerful systems with Kafka at the centre of things. Notification systems, distributed database synchronisations, monitoring systems are some of the applications that come to mind when thinking of Kafka's use-cases.
In my quest to understand Kafka a bit more deeply, I tried setting up a system monitoring application which looks for system stats like CPU and RAM usage and visualises them in a dashboard.

The tools I went with for this are

Kafka as a message queue
Golang for a couple of microservices
React for the dashboard
Docker and docker compose for orchestration

Architecture

The architecture I'm going for is this.

Services

Service	Description
System stats	Golang service which captures CPU and RAM usage from host machine
Broker	Kafka broker which facilitates event streaming
Producer	Golang service which receives usage data from system stats service and pushes it into Kafka
Consumer	Golang service which subscribes to Kafka broker and streams data to dashboard
Dashboard	Visualises system stats in real time. Gets system stats data from Consumer through a WebSocket. Built using React

TL;DR

If you understand the architecture and the purpose of each service above, you can find the implementation in the below repository

Realtime system monitoring

If you do decide to stick around, we'll be going over in detail about these services.

Deep dive

Alright! Thanks for sticking around. We'll go over key parts of each service and finally we'll see how this comes together with docker compose.

System Stats

The main purpose of this service is to collect system usage statistics from the host machine and send it to the producer service.
Here, I'm reading memory stats from the /proc/meminfo file which contains information about the current memory usage. This file is available in all linux/unix based systems

We could have used a much more focussed tool like Prometheus or Cadvisor to gather system stats, but that is not the main objective of this article.

We'd like to collect memory stats every second. So, we need a cron job which runs every second and reads memory info from /proc/meminfo file. We'll write a function which reads, parses and returns memory stats.

Next, we need a function which takes in the stats and sends it to the producer. This will be a simple HTTP POST call.

Finally, we need to put these two together in our cron job. Every time the job runs, these two functions need to be called in series. Our main function for this service looks like this.

Awesome, now we have a service which collects memory info from our host machine every second and sends it to the producer

Producer

The purpose of this producer service is to receive system stats and push it into the Kafka broker. To do this, we have a HTTP POST endpoint which get the stats in the body and writes it into a Kafka topic.

Our main function in this service is very simple. There is a single POST endpoint to receive stats. I'm using Gin framework for routing here.

Before we write to Kafka, we need to setup our Kafka Writer. Let's do that.

Now, we can setup our PushToKafka function which is the handler for our POST endpoint.

Great! We have a Producer service which receives system stats via a POST endpoint and writes this data into our Kafka broker.

Consumer

The Consumer service does two things.

Subscribe to Kafka topic to receive system stats data
Relay system stats data to Dashboard via WebSockets

Before our service can listen to Kafka messages, we need to setup a Kafka Reader

Now our service can subscribe and listen to data pushed into Kafka queue by our Producer. We'll setup a listen function which reads messages from Kafka

Note that it takes in a callback function. This callback function gets called whenever there is a message from Kafka. We'll use this callback function to relay systems stats to Dashboard service

The next step is to send system stats data to Dashboard whenever we receive it from Kafka. But before we do this, we need an endpoint, specifically a socket endpoint which our Dashboard service can connect to. We'll define this endpoint in our main function

And our upgrade handler looks like this

To establish a Websocket connection, the HTTP connection between the service and client has to be upgraded to use WebSocket protocol.

The connections.Subscribe(conn) call is to keep track of all the socket connections.

The final step in our Consumer service is to relay the messages from Kafka queue to Dashboard service. For this, we setup a function called SendMessage, which will be our callback function for Kafka Listen function.

Awesome! Now we have a Consumer service which listens to messages in our Kafka queue and relays that message to our Dashboard service through a WebSocket.

Dashboard

Dashboard is a very simple service which connects to the Consumer service via a WebSocket and renders the system stats in a table UI.

Without going into much detail about how to setup a React application or how the markup and styling will be, the important part here is to create a socket connection with Consumer service's socket endpoint and setup an onmessage callback function on our socket.

We have a React component with system stats as our state. We update this state whenever we receive data from the WebSocket.

Kafka Broker

The final service to setup is indeed Kafka Broker which facilitates this whole message queue.

I'm using an example docker compose config from Kafka's Github

Kafka-docker-compose

That's it! We're done with our setup.

Putting it all together

Now if we run all of the services using docker-compose,



docker-compose build
docker-compose up

Navigate to http://localhost:WEB_UI_PORT, we can see our dashboard getting updated every second with our system stats.

Awesome! As you can see, our table gets update with latest value of systems stats every second.

I haven't gone into detail about how to setup build process for our services. Please refer Dockerfile and docker-compose.yml files for more information.

Conclusion

This is a barebones Kafka setup which does nothing more than relay messages in a single topic. I wrote this article just to get a basic understanding of what Kafka is, and to understand what Kafka could be used for. Hope folks reading this got something out of it.

Thanks for coming through, cheers!

Your first Turborepo

Siddharth Venkatesh — Fri, 29 Apr 2022 20:42:22 +0000

Introduction

Monorepos are fantastic. They let you maintain all your projects in a single repository. I use one at my workplace and I see its advantages everyday. If you know anything about monorepos, setting them up can be tricky sometimes. Recently, I've been following the developments over at Turborepo, which attempts to make setting up the tooling for monorepose simpler. The more I look through their docs, the more I get exited about using it. So, I gave it a shot and I have to say, the experience has been fantastic.

Why this article?

If you're wondering you can just go to their docs and set it up yourself, yes, you absolutely can. They have a cli which can help you setup a new project and they have a solid set of examples for most scenarios. But, it is super fun setting things up from scratch, and I wanted to see how much of work it is with Turborepo.

The setup

I'll setting up a new monorepo with a couple of simple apps and a UI library which would be shared by the apps. The goal is not the design and functionalities of these apps, but the tooling and features Turborepo provides. There will be two apps admin and products, both of them will be bundled using Vite. Vite is blazing fast and you should definitely give it a try just for its speed. The UI library, which will contain just a button component, which is written in TypeScript, will be bundled using tsup. tsup uses esbuild underneath, so we can expect blazing fast build times. I'll be using yarn for package management. We will also be using a common eslint configuration which will be shared across all three codebases.

Let's get started!

Turborepo stuff

Let's first create a folder for our project and start initialising our monorepo.
As with any JS project, we start with a package.json.

package.json

This is the initial config I'm using. It has turbo and eslint installed as a devDependency. If you are familiar with monorepos, the workspaces array should make sense. All the projects in your monorepo should be listed as a workspace. Here, we have two directories, apps contain admin and products, and packages, which contains the UI library and the eslint configuration. Anything that can be shared across multiple projects can live in the packages folder.

Next is our turbo.json. This is Turborepo's config file. I browsed through their examples and found the simplest config to get started.

turbo.json

We'll be covering this in a later section.

Setting up apps

Vite has a cli which makes it easier for us to bootstrap a React app.
In our apps folder, run



yarn create vite admin --template react

This will create a new react app named admin. Similarly, we can create products app as well.



yarn create vite products --template react

Now we have two apps named admin and products in our apps directory.

Setting up the library

I've added all the dependencies needed for a TS library with types and eslint packages. Also added are the scripts for build, dev, lint.

packages/ui/package.json

Now, lets simply add a Button component and export it.

packages/ui/Button.tsx

packages/ui/index.tsx

Now, our project looks like this

Now that we have setup our apps and library, we can setup the tooling to link(turbocharge) them.

Add library as a dependency

The next step is to add the library as a dependency to our apps. It is as simple as adding it to devDependecies in both apps/admin/package.json and apps/products/package.json.

Turborepo will use the name field in the library's package.json to resolve it in the apps.

We can now use this Button component in admin and products.

In apps/admin/src/App.jsx

We can do the same thing in apps/products/src/App.jsx as well.

Adding scripts

The final step before we test this is to add scripts for build, lint and dev. In our root package.json, we can add

These commands are directly tied to the pipeline configurations in turbo.json. For example, if we look at the build command, with the "dependsOn": ["^build"], option, we are letting Turborepo know that build commands should only be run after all its dependencies are built. Turborepo is smart enough to realise admin has a dependency ui, which needs to be built before building admin. So, it builds ui first and then bundle admin. Pipelines are a powerful feature in Turborepo and you can read about it here.

Now, there is nothing left but to run our two apps. First, we would need to install our dependencies by running,



yarn install

Then, we start the dev server using



yarn dev

If we inspect the terminal messages, we can see that admin is running in localhost:3000 and products is running in localhost:3001.

(Look at the insane 2.914s start times! Vite FTW!)

Now if we navigate to localhost:3000, we see

We can see our button component is rendering as expected.

Setting up shared lint config

Similar to how we shared a library across apps, we can share config files across apps as well. We'll be using a single eslint config in all our apps and library. For that we shall create a folder called config in our packages directory.
Inside it, we'll create a file eslint-preset.js,

And a package.json.

packages/config/package.json

The package.json contains all the eslint packages we'll be needing, and notice the files property includes the lint config file.

Now, we add config as a dev dependency in admin, products and ui. In each of their package.json, add it as a devDependency

apps/admin/package.json

Also, we would need a .eslintrc.js which simply exports the lint config from config.

apps/admin/.eslintrc.js

Now, we we run yarn lint on our root folder, Turborepo will run the lint command on all of our projects.

Notice that we did not need to install eslint(except in the root) or its corresponding packages anywhere else other than the config folder.

Awesome! We have setup our own monorepo with two apps, a library and a shared eslint config.

Conclusion

This idea of monorepos can be extended and even backend code can be added to the same repo. One awesome use-case I can think of is sharing types between frontend and backend apps using a shared package. We have barely scratched the surface of Turborepo and its features. Remote Caching
is one such feature I'm exited to try out. Meanwhile, this exercise was a great starting point.

The source code for this can be found here

Cheers!

Creating a simple file explorer with recursive components in React

Siddharth Venkatesh — Wed, 16 Feb 2022 19:05:38 +0000

Introduction

Recursion is one of the most common programming constructs there is. Recursion in JavaScript land, is generally implemented via recursive functions, where a function calls itself. A very common example of a recursive function is the factorial function. It goes like this



function factorial(x) {
    if (x === 0) {
        return 1;
    }
    return x * factorial(x - 1);
}

As you can see, the function calls itself until the argument becomes 0. This idea can be extended to a variety of scenarios.

Idea

Where it gets interesting is when you add React into the mix. React components are basically functions. So, it must be possible for a component to render instances of itself in it.

Example

Let's build a simple file explorer to list files and folders. Each folder can in turn have multiple files and folders. When you click on a folder, it should expand to show its contents. It is exactly like the File Explorer sidebar in VSCode/Sublime etc.

Let's create a component that mimics this behaviour and uses recursion in the process.

Implementation

Before we get started on our component, we need a list of files and folders. We'll create a json file with a files and folders from a typical React project.

files.json

Here, each entry will have a name property, which denotes the name of the file/folder, a type property, which denotes if it is a file or a folder, and an items array, which in case of a folder will house all the contents in that folder. Each entry in the items array will again be an entry with the name, type and items properties.
With this, we are ready to create our recursive component

Recursive component

Our Directory component will accept a prop called files which will be the contents from our files.json file. First, let's get the easier part out of the way, displaying a file. If the type property is file, we simply render the file name

Directory.jsx

Now for the folder part, we first render the name of the folder.

To render the items in a folder, all we have to do is loop through the items array and render the <Directory /> component for each item.

Our <Directory /> component now uses recursion to traverse through our file list to render files and folders. One final thing left to do is, when you click on a folder, its contents should show up. We can do this by declaring a state variable in our component and toggling it on clicks.

Great! This should be enough to get our app up and running. We'll import this component and pass in contents from files.json as a prop.

App.jsx

Now, if we run our app, it should give us something like this.

Thats it! We've created a component which recursively calls itself.

The complete source code can be found here

Cheers!

Network aware preloading strategy in Angular

Siddharth Venkatesh — Tue, 05 Oct 2021 17:39:11 +0000

Introduction

Preloading is great. It lets us fetch parts of our application before it is even requested for. By doing this, our content is ready to be served when needed without any delay. There are many ways of doing this, I'll be touching upon some of the ways to do this in Angular and how we can create our own custom preloading strategy based on the user's network state.

Preloading strategies in Angular

Angular provides us with route-based preloading out of the box. There are two strategies namely PreloadAllModules and NoPreloading.

NoPreloading, as the name suggests does not preload any modules. This is the default behaviour.

PreloadAllModules, on the other hand, preloads all modules. A simple example adding a preloading strategy in a routing module file can be seen below

app.routing.module.ts

In the above code, the ProductsModule is lazily loaded, which means, the bundle is downloaded only when the user lands on the /products route. By passing the preloadingStrategy property, we instruct Angular to preload all lazily loaded modules.

Custom preloading strategies

Angular also lets us provide custom preloading strategies where we can determine if a component needs to be preloaded or not. This can be done by providing our own custom class to the preloadingStrategy property.

Our custom class needs to extend PreloadingStrategy class from @angular/router and implement the preload function in it. A simple custom preloading strategy class would look something like this.

customPreloadingStrategy.ts

Here, our CustomPreloadingStrategy class implements PreloadingStrategy class and has the preload function as well. The preload function should either return the load function or an empty observable. Returning the load function means the module can be loaded. As for the shouldPreload function, that is where our custom logic goes in.

We can use this custom strategy in our routing module
app.routing.module.ts

Network aware preloading strategy

Now that we are familiar with setting up a custom preloading strategy, now we can move on to using our user's network connection information to make a decision for us. Let's say our user has a very slow bandwidth, so we wouldn't want to further tax our user by preloading all of our modules in the background. In that case, we can decide not to preload our modules. To do this, we rely on the navigator object in our browser and specifically the connection property in navigator.

If you're reading this in desktop/laptop, open the browser's console and type in navigator.connection, you would see something like this.

We can leverage this information in our shouldPreload function to let Angular know whether to preload the module or not.

customPreloadingStrategy.ts

In the example above, we're using the saveData property to determine if the user has data-saver on, and we're using the effectiveType property to see if the user is on 2g or 3g connections. If the user is on slower internet connections, we are returning false, which would in turn prevent preloading of the particular module

Conclusion

We can extend this idea of custom preloading strategies to make intelligent decisions based on our user's state. I have no idea if this method would even result in a tangible improvement in the user experience, but nevertheless a fun topic to explore.

You can find the source code here

Cheers!

Shared library in yarn workspaces

Siddharth Venkatesh — Tue, 24 Aug 2021 11:03:33 +0000

Introduction

Recently, I wrote and article about setting up yarn workspaces and adding Docker support to it for orchestration.

You can read it here Docker setup for yarn workspaces

I thought we can extend this setup to include a library as well. Shared libraries are fairly common in any organisation, so we'll be adding a component library based on React to our workspace.

To go over our setup, we have an apps folder where all our apps live in. We have two apps called admin and product. We are going to be adding a component library to this setup and this library will be used by both of our apps. Let's get started.

Adding a library

I'm going to be using Vite as our build tool. It has a nice cli which lets us scaffold an application easily.
To create a vite project in our appsdirectory,



cd apps
yarn create vite lib --template react

Our lib folder would look something like this now

The next step is to update our dependencies by running yarn install from the root of our workspace.

Adding components to our library

If we open the src folder in lib, we can see that it is an ideal setup for an application, not a library. So, we'll remove all the files from src and index.html and add our own.

First, we'll add an entry file called index.js in src. This will be the starting point file in our bundling process.

Next, we'll create a folder called components which would house all our components, and add an index.js file to it as well. We'll export all our component from this index file.

Our project should look like this now.

Adding a button component

Great! We have our directory setup nailed down. We can finally start adding components. We can add a simple button component. I'm going to create a directory called Button in components directory, which would contain three files. Button.jsx, Button.css and an index.js.
Let's add the contents of each of these files

Button.jsx

Button.css

index.js

This is a very simple component with not a lot of emphasis on styling and attributes. You can customise this component to your liking.

We have a component exported from our Button directory. We need to export this component from our components directory as well. We'll add this export like this

src/components/index.js

The next step is to export the components from our src.
src/index.js

This would export all our exports from components directory. If we add more components, all of them would be exported from here.

Our lib folder should look like this now

Finetuning Vite config

Before we can start using our library in applications, we need to modify our vite.config.js to let vite know this is a library and should be bundled as one.
The documentation for this can be found here
We'll be adding the following config to vite.config.js

We also need to add some options to our lib's package.json.

The peerDependencies option tells the bundler not to add these dependencies in our final bundle.
The main, module and exports options are needed for the application's bundler to figure out where the files are for umd and esm formats.

Great! Let's now move on to using library in our apps.

Using our library in apps

Adding a local library as a dependency is as simple as adding any other dependency.



yarn workspace admin add lib@0.1.0

This command would add lib as a dependency to admin. Notice we've mentioned the version of lib as well. This version must be the same as the version property in package.json in lib.

Another way is to simply add lib: 0.1.0 entry to the dependencies section of package.json in admin.

Testing it out

We are now in a position to use our Button component from lib. We'll do that in App.jsx in admin

admin/src/App.js

Next step is to alter our scripts in package.json to make sure our library is compiled when we run our applications.
We'll add a few scripts to do this

package.json

Awesome! We're almost done. Only thing left to do is to check our admin app.



yarn start:admin

If we open http://localhost:3000 on our browser, we can see our Button component in red color as per our type prop.

We can repeat the same process to use the library in any other application in the workspace.

Conclusion

Awesome! We have a yarn workspace with two applications and a component library in React. We can extend this idea and even add multiple libraries into this workspace.

If you are interested in setting up a React-Typescrippt component library from scratch, check out my article here Component library setup with React, TypeScript and Rollup

The source code for this can be found here

Cheers!

Docker setup for yarn workspaces

Siddharth Venkatesh — Fri, 20 Aug 2021 18:06:44 +0000

Introduction

As monorepos seem to be having their moment in the developer community right now, we can see quite a bit of new monorepo tools popping up. npm recently announced npm workspaces with version 7, Nx has been gaining a lot popularity and lerna has been around for quite a while now. I use yarn in most of my projects now, and thought it would be fun to explore yarn workspaces with a simple monorepo setup.

In this workspace, I am going to be adding two React applications. Further, we can also add docker support to make it easier for deployments. Let's get started.

Initialising the workspace

Let's start by creating a folder for our project and initialise yarn



mkdir yarn-docker-setup
cd yarn-docker-setup
yarn init -p

If you do not have yarn installed already, you can install by npm install yarn -g.

After filling out basic questions, you would have a package.json file.

To turn this project into a workspace, we need to add workspaces option in our package.json



"workspaces": ["apps/*"]

apps is a directory where all our apps live.
Great! We've initialised our workspace, next step is to add applications.

Adding apps

We're going to be adding two React applications to this project namely admin and product. I'm using Create React App to scaffold our apps.



yarn create react-app apps/admin
yarn create react-app apps/product

This would take a couple of minutes to finish and by the end you would have two folders called admin and product inside the apps folder.

Great! We've added two apps to our workspace. The next step is let yarn know about each app's dependencies, so it can optimise and cache them. In the project root folder, run



yarn install

This goes through the dependencies and moves them to a central node_modules folder in the project's root.

Let's test out our setup to see everything works. Let's add scripts in our package.json to start and build our apps



"scripts": {
    "admin": "yarn workspace admin start",
    "product": "yarn workspace product start",
    "build:admin": "yarn workspace admin build",
    "build:product": "yarn workspace product build"
}

We've also added build scripts to compile our apps into static files.
If we run yarn admin or yarn product, we should see the standard create react app screen

Adding Docker support

Docker provides us with a simple and effective way to package our apps into images that could be run anywhere without any dependence on the environment or operating system. With docker-compose, we can orchestrate multiple services(apps) with a simple configuration. Going too much into docker and docker-compose maybe a bit out of reach for this article, so let's dive into the docker setup.

First step is add a Dockerfile. We can add individual Dockerfiles for each app, but since the build process is same for both the apps, we can use a single Dockerfile for both of them.

First, we need a node environment to compile our React projects, and we need the name of the folder which we need to build, in this case admin or product. We get that using the BUILD_CONTEXT argument.



FROM node:14.17.1 as build
ARG BUILD_CONTEXT

The next step is to copy over the source code into the image.



WORKDIR /base
COPY package.json .
COPY yarn.lock .
COPY ./apps/$BUILD_CONTEXT/package.json apps/$BUILD_CONTEXT/
RUN yarn install

We are defining /base as our working directory. All our code goes here.
In the next 3 lines, we are copying package.json, yarn.lock and the package.json file of the particular app into the image.
Then we run yarn install to install our dependencies.

Interesting thing to note here is, we could have copied our entire source code into the container in one go. The reason we don't do that is, every instruction in a Dockerfile is cached in the background. By copying just the package.json and yarn.lock files, we can take advantage of this caching system. These files rarely change in the course of the project, so if we install our dependencies once, and if they don't change the next time we build, Docker will use the existing cache and not run yarn install every-time we build. This will significantly reduce our build times.

The next step is to copy the app's code and build.



COPY ./apps/$BUILD_CONTEXT apps/$BUILD_CONTEXT
RUN yarn build:$BUILD_CONTEXT

Great, as of now our Dockerfile looks like this



FROM node:14.17.1 as build
ARG BUILD_CONTEXT

WORKDIR /fe
COPY package.json .
COPY yarn.lock .
COPY ./apps/$BUILD_CONTEXT/package.json apps/$BUILD_CONTEXT/
RUN yarn install
COPY ./apps/$BUILD_CONTEXT apps/$BUILD_CONTEXT
RUN yarn build:$BUILD_CONTEXT

Our compilation step is complete. Our React app has been compiled into static files and they are inside the image. But order to serve them, we need a web server. We could use node as our web server as we are already using it for building. But a node image is significantly bigger(close to a gigabyte) in size compared to a traditional web server like nginx.

We'll add nginx configuration as part our build step in our Dockerfile.



FROM nginx:stable-alpine
ARG BUILD_CONTEXT
COPY --from=build /fe/apps/$BUILD_CONTEXT/build /usr/share/nginx/html
COPY nginx.conf /etc/nginx/conf.d/default.conf
EXPOSE 80
CMD ["nginx", "-g", "daemon off;"]

The first two lines are self-explanatory.
The third line is where is gets interesting. If you see the first line of our Dockerfile, it says as build next to our node version. This is done so we can refer to this as context in later parts of our build steps.
We have our compiled React app in the node image. We need to take those files and put it in our nginx image. That's what this line does. It copies the /fe/apps/$BUILD_CONTEXT/build folder from build context into /usr/share/nginx/html.
The last line is to start our nginx web server.

The next step is to define an nginx.conf config file nginx can use to run our app, which looks like this. This is a barebones nginx web server configuration which can be used for any frontend application.



server {

  listen 80;

  location / {
    root   /usr/share/nginx/html;
    index  index.html index.htm;
    try_files $uri $uri/ /index.html;
  }

  error_page   500 502 503 504  /50x.html;

  location = /50x.html {
    root   /usr/share/nginx/html;
  }

}

Our entire Dockerfile now looks like this



#build
FROM node:14.17.1 as build
ARG BUILD_CONTEXT

WORKDIR /base
COPY package.json .
COPY yarn.lock .
COPY ./apps/$BUILD_CONTEXT/package.json apps/$BUILD_CONTEXT/
RUN yarn install
COPY ./apps/$BUILD_CONTEXT apps/$BUILD_CONTEXT
RUN yarn build:$BUILD_CONTEXT

#webserver
FROM nginx:stable-alpine
ARG BUILD_CONTEXT
COPY --from=build /base/apps/$BUILD_CONTEXT/build /usr/share/nginx/html
COPY nginx.conf /etc/nginx/conf.d/default.conf
EXPOSE 80
CMD ["nginx", "-g", "daemon off;"]

This setup is enough for us to build a Docker image of our app and run by running



docker run <image-name> -e BUILD_CONTEXT=admin/product

We want to go a bit further and add in an orchestration step using docker-compose

For this, we need to add a docker-compose.yml file in the root of our project.



version: '3'

services:
  admin:
    container_name: admin
    build:
      context: .
      dockerfile: Dockerfile
      args:
        - BUILD_CONTEXT=admin
    ports:
      - '8080:80'
  product:
    container_name: product
    build:
      context: .
      dockerfile: Dockerfile
      args:
        - BUILD_CONTEXT=product
    ports:
      - '8082:80'

We define two services here, admin and product for our two apps.
In our service section, we define three properties, container_name, build and ports.

container_name defines the name of the container
context in build refers to the directory this build needs to be executed on, dockerfile refers to the name and location of the Dockerfile and args refer to build time arguments. These are the arguments that will be used in the Dockerfile ARG section
ports lets us map ports on the host machine to the container port. Value 8082:80 indicates that any request on port 8082 on host machine will be routed to port 80 on the container.

Awesome! We are done with our docker-compose setup. Final thing left to do is run and see for ourselves.



docker-compose build

command is used to build out both our apps. This will compile our app using instructions from our Dockerfile and create an image.

To run these images,



docker-compose up

This command will take our images and create containers and run them.

Now we can go to http://localhost:8080 and http://localhost:8082 to see our apps in action.

Conclusion

What we have now is a very simple implementation of workspace and docker setup. We can use this as a starting point and start adding backend services and component libraries to this setup.

If you are interested in setting up a component library from scratch, check out my article on Setting up a component library with React, TypeScript and Rollup

We can add new projects into the apps folder and yarn would take care of the dependency resolutions for us.

The source code for this setup can be found here

Cheers!

Component library setup with React, TypeScript and Rollup

Siddharth Venkatesh — Mon, 16 Aug 2021 18:48:42 +0000

Introduction

Component libraries are becoming more and more popular by the day, especially at organisations with multiple products and teams. Organisations are dedicating teams just to maintain the component library. The end goal here might be a Design System, with well thought our principles and practices. But, a good design system takes months or even years of research and a dedicated team which a lot of organisation cannot afford. Google's Material design and Atlassian's Design system are some of the excellent ones that come to mind. A good place to start for majority of teams is a component library. A collection of commonly used components which can help attain consistency across applications. We can start out with simple components like button, inputs, modal and add more along the way.

Let's try to build a simple component library from scratch using React, Typescript, and Rollup to bundle it, and learn a thing or two along the way.

Initialise the project

Let's start by creating a directory and initialising an npm project called react-lib



mkdir react-lib
cd react-lib
npm init

You can fill out the questions or pass the -y flag to initialise with default values. We now have a package.json file in our project.

Since we're going to be using react and typescript, we can add those dependencies



npm i -D react typescript @types/react

Since we are going to be shipping this as a library, all our packages will be listed under devDependencies. Also, the app in which this library will be used will come with react, we don't have to bundle react along. So, we'll add react as a peerDependency. Our package.json looks like this now

Adding components

My preferred way of organising components is inside src/components folder, where each component would have its own folder. For example, if we have a Button component, there would be a folder called Button in src/components with all the button related files like Button.tsx, Button.css, Button.types.ts, and an index.ts file to export the component

There are also a couple of index files along the way to export stuff. One is the main entrypoint to the project, at src/index.ts, and one which exports all the components at src/components/index.ts. The folder structure with the button component would look like this.

Button component

Now, let's add the code for the Button component. I'm going with a very simple component as this is not really our concern right now.

Button.tsx

Button.css

Button.types.ts

Button/index.ts

Now that we have our Button component, we can export it from components and from src.

src/component/index.ts

src/index.ts

TypeScript configuration

We've added our components and now in order to build our library, we need to configure Typescript. We've already installed the typescript dependency, now we need to add the tsconfig.json. We can do this by



npx tsc --init

This creates a tsconfig.json file with most of the available options commented. I use most of the defaults with some minor changes.



{
  "compilerOptions": {
    "target": "es5",
    "module": "esnext",
    "jsx": "react",
    "sourceMap": true,
    "outDir": "dist",
    "strict": true,
    "moduleResolution": "node",
    "allowSyntheticDefaultImports": true,
    "esModuleInterop": true,
    "skipLibCheck": true,
    "forceConsistentCasingInFileNames": true,
  }
}

Let's add a build script in our package.json to test this out.



"scripts": {
    "build": "tsc"
 },

If we run npm run build, we should see a dist folder with all our ts files transpiled into js files. If you notice, there are no css files in dist and they are not bundled by out ts compiler. Let's do that using Rollup

Rollup configuration

We'll be using Rollup as the bundler of choice here. So, lets install it



npm i -D rollup

Plugins

Rollup has a plugin system by which we can specify all the tasks that need to be performed during the bundling process. We'll need the following plugins

@rollup/plugin-node-resolve - Resolve third party dependencies in node_modules
@rollup/plugin-commonjs - To convert commonjs modules into ES6
@rollup/plugin-typescript - To transpile our Typescript code in JS
rollup-plugin-peer-deps-external - To prevent bundling peerDependencies
rollup-plugin-postcss - To handle our css
rollup-plugin-terser - To minify our bundle

Lets install these plugins



npm i -D @rollup/plugin-node-resolve @rollup/plugin-commonjs @rollup/plugin-typescript rollup-plugin-peer-deps-external rollup-plugin-postcss rollup-plugin-terser

rollup.config.js

The next step is to add the rollup.config.js file. This is where all our rollup configs live.

The entrypoint to our library is the src/index.ts file and we'll be bundling our library into both commonjs and es modules formats. If the app using this library supports esmodules, it will use the esm build, otherwise cjs build will be used.

rollup.config.js



import resolve from '@rollup/plugin-node-resolve';
import commonjs from '@rollup/plugin-commonjs';
import typescript from '@rollup/plugin-typescript';
import { terser } from 'rollup-plugin-terser';
import external from 'rollup-plugin-peer-deps-external';
import postcss from 'rollup-plugin-postcss';

const packageJson = require('./package.json');

export default {
    input: 'src/index.ts',
    output: [
        {
            file: packageJson.main,
            format: 'cjs',
            sourcemap: true,
            name: 'react-lib'
        },
        {
            file: packageJson.module,
            format: 'esm',
            sourcemap: true
        }
    ],
    plugins: [
        external(),
        resolve(),
        commonjs(),
        typescript({ tsconfig: './tsconfig.json' }),
        postcss(),
        terser()
    ]
}

We have defined the input and output values for our cjs and esm builds.

Putting it all together

Notice that we have specified the file option in output from package.json. Let's go ahead and define these two values in package.json



"main": "dist/cjs/index.js",
"module": "dist/esm/index.js",

Now that we've configured Rollup, we can use it in our build script in package.json instead of the tsc command before.



"build": "rollup -c"

If we run npm run build now, we can see that there is a dist folder created with our library output.

The cjs folder contains the commonjs bundle and esm folder contains modern esmodules bundle.

We have our own library which can now be published to the npm registry or used with other applications locally as well.

Testing it out

We can test out our library locally by using npm pack or npm link.

Bundling types

If you notice in our dist folder after running npm run build, we can see that we are not bundling our types. The advantage of using TS here is that code editors can pick up the types and provide Intellisense and static type-checking, which is super useful. It also reduces the need to look at documentation often.

We need a add a few options in our tsconfig.json to generate types.



"declaration": true,
"declarationDir": "types",
"emitDeclarationOnly": true

Adding this would add a types folder in our cjs and esm folders in dist.

We can further improve this by providing a single file which would contain all the types used in our library. For this, we are going to be using a Rollup plugin called rollup-plugin-dts which takes all our .d.ts files and spits out a single types file.



npm i -D rollup-plugin-dts

We can add another entrypoint in our rollup.config.js to add our types config.



{
        input: 'dist/esm/types/index.d.ts',
        output: [{ file: 'dist/index.d.ts', format: "esm" }],
        external: [/\.css$/],
        plugins: [dts()],
},

What this does is take the index.d.ts file from our esm bundle, parse through all the types file and generate one types file index.d.ts inside our dist folder.

Now we can add a types entry in our package.json



"types": "dist/index.d.ts"

The entire rollup.config.js looks like this now



import resolve from '@rollup/plugin-node-resolve';
import commonjs from '@rollup/plugin-commonjs';
import typescript from '@rollup/plugin-typescript';
import { terser } from 'rollup-plugin-terser';
import external from 'rollup-plugin-peer-deps-external';
import postcss from 'rollup-plugin-postcss';
import dts from 'rollup-plugin-dts';

const packageJson = require('./package.json');

export default [
    {
        input: 'src/index.ts',
        output: [
            {
                file: packageJson.main,
                format: 'cjs',
                sourcemap: true,
                name: 'react-ts-lib'
            },
            {
                file: packageJson.module,
                format: 'esm',
                sourcemap: true
            }
        ],
        plugins: [
            external(),
            resolve(),
            commonjs(),
            typescript({ tsconfig: './tsconfig.json' }),
            postcss(),
            terser()
        ],
    },
    {
        input: 'dist/esm/types/index.d.ts',
        output: [{ file: 'dist/index.d.ts', format: "esm" }],
        external: [/\.css$/],
        plugins: [dts()],
    },
]

Now, if we use our library in other projects, code editors can pick up the types and provide Intellisense and type checking.

Conclusion

This is by no means a comprehensive or perfect way to setup a component library. This is just a basic setup to get started and learn about bundling in the process. The next step in this process would be to add tests and tooling like Storybook or Styleguidist.

The source code can be found here react-ts-lib

Thanks for reading!
Cheers!

Cleaner data fetching with react-query

Siddharth Venkatesh — Sat, 14 Aug 2021 20:04:41 +0000

Data-fetching is something I feel does not get the importance and limelight as say state management in the React world. People often combine client state and server state into their state management solution. By server state, I mean the data from your backend servers. For example, in a redux setup, client state and data from server are stored in stores and updates are handled through actions and reducers. Client state change usually causes a change in server state, so combining them makes sense in most cases. But I feel they are two separate entities and if handled properly, we can even get rid of client side state management solutions in some cases.

I started looking at solutions to separate client and server state management. Coming into the React world from an Angular background, I wanted something simple as storing your server data in a service and inject it into your components, and you're good to go. In React, you'd have to maintain a global state management solution if the data you're fetching is needed in multiple components.

react-query to the rescue

I then looked at libraries which performs data fetching and maintain server state. I stumbled up react-query and boom! It had everything I needed and more. It provided a way of maintaining a global context of server state, and also provided an excellent caching solution with minimal configuration. There's also swr which is equally awesome.

Example

Alright, enough chit-chat. Let's get to the code. Here's how I usually setup my React projects. I have a folder called pages which has all the top level routes. components folder houses all the UI components and a folder called api which has all the server side APIs.

Let's say we have a product entity. A product has all of CRUD operations attached to it. So, the following API calls need to be there in products

1. Fetch all products
2. Fetch a specific product
3. Add a new product
4. Edit a product
5. Delete a product

react-query provides us with a useQuery hook which we can use for all our queries. This should cover points 1 and 2 in the above list.
We'll create our own data-fetching hooks for product by wrapping useQuery.

Our product file in api/product.js looks something like this

Let's go over how we setup functions for data-fetching with react-query.

Fetch products

Let's start with fetching products. The barebones implementation with useQuery would look like

Not much going on here. We pass a unique id key as the first argument to useQuery and a fetch function to actually make the API call.

If we want to use this in a component, we can do so like

It gives us loading and error states which I think is neat

We have a working data-fetching setup, but the feature set doesn't end here. Any listing page would have additional features like search, filters, pagination etc. This is where react-query makes it really simple to add these. Let's set these things up.

Pagination

In our Products component, we can have page and limit values as state. page denotes the current page number and limit denotes the number of products to be displayed on the page.

The next step would be to hook this up with our useFetchProducts hook. Lets make our useFetchProducts hook take in page and limit as arguments.

Let's unpack what's going on here. The useFetchProducts hook now takes in page and limit as arguments. It also adds these two to the key and adds them to the fetch URL.

Great! That's it. We now have our pagination implemented. Now, whenever the value of page and limit changes in our Products component, react-query would automatically fire the API request and update the UI.

Search

Another important common feature is search. So, lets add a search on name field in products. As you might have guessed already, it is the exact same process as pagination. We'll have a name field in state, and this state value would be passed to our useFetchProducts hook.

Our useFetchProducts will look something like this.

Similarly, we can hook any number of filtering/search parameters to out useFetchProducts hook. This hook can be used across multiple components without any global state management system.

Caching

Caching is hands-down my favourite feature of react-query. With minimal code, we can setup a powerful caching system. In the case of our products example, let's say we want our products to be cached for 10 seconds. We can do this by adding the staleTime option.

This would use the data from the cache whenever this hook is called with the same page, limit and name.

An important thing to understand here is the key option. The key uniquely identifies a request. So, a request with page as 1 is not the same as a request with page as 2. The value from the cache will only be used all three key values are same.

Updating internal cache

react-query also give us access to its internal cache. We can update this cache whenever we want. What this means is, we can set internal cache values of individual products.

Imagine this, we have fetched a list of products and displayed them on screen. The user clicks on a product, we take them to the product screen. In the product screen, we would have to fetch the product's details and display it. But! We already have the product's details in react-query's internal cache. What if we can use that instead of making an API call?

Let's start with creating a useFetchProduct hook for fetching individual product.

Nothing crazy going on here. Pretty much the same thing we did before, just that it take id as an argument.
The important thing to note here is ['product', id] as the key. We are associating a product's id as its key.

Now to the interesting part, whenever we fetch the list of products, we set the internal cache with value of each individual product. react-query exposes a useQueryClient hook which gives us the internal cache.

Whenever our list of products is successfully fetched, the onSuccess function is called which has the API response it. We loop through each product and store it in the cache using the setQueryData method.

Now, whenever the user moves to an individual product's page from the products list page, the value from the cache will be used rather than making an API call.

Conclusion

So, I found react-query to be extremely simple and powerful data-fetching solution. After using react-query, I even removed global state management solutions from some of my projects. Go give them some love on their repo

Cheers!

Cleaner async-await for asynchronous JavaScript

Siddharth Venkatesh — Fri, 13 Aug 2021 13:44:43 +0000

So, there are thousands of articles floating around the internet about why callbacks are bad and you should be using Promises and async/await. As the popular saying goes, the answer to most of the opinions in world of programming is "It depends". There is no one right solution for any problem.

What I'm addressing here is a very simple problem. I need to run multiple async operations in a function, and I need the code to look clean and readable. I have a handler function of a POST request to create a new product. It is written in Express and does the following

const createProduct = (req, res, next) => {
    // Check if the user is valid
    // Check if the product name already exists
    // Check if the store exists
    // Save product to database
}

Promise based approach

A promise based approach would look something like this

const createProduct = (req, res, next) => {
    const { id: userId } = req.user;
    User.findOne({id: userId})
        .then((user) => {
            if (!user) {
                console.log('User does not exist');
                return res.status(400).json({
                    status: 'error',
                    message: 'User does not exist',
                });
            }
            const { name, storeId, price } = req.body;
            Product.findOne({name})
                .then((product) => {
                    if (product) {
                        console.log('Product with the same name already exists');
                        return res.status(400).json({
                            status: 'error',
                            message: 'Product with the same name already exists',
                        });
                    }
                    Store.findOne({id: storeId})
                        .then((store) => {
                            if (!store) {
                                console.log('Store does not exist');
                                return res.status(400).json({
                                    status: 'error',
                                    message: 'Store does not exist',
                                })
                            }
                            // Valid product. Can be saved to db
                            const newProduct = new Product({
                                name,
                                storeId,
                                price,
                            });
                            newProduct.save()
                                .then(() => {
                                    console.log('Product saved successfully');
                                    return res.status(200).json({
                                        status: 'success',
                                        message: 'Product saved successfully',
                                    });
                                })
                                .catch((err) => {
                                    console.log('err');
                                    next(err);
                                })
                        })
                        .catch((err) => {
                            console.log(err);
                            next(err);
                        })
                })
                .catch((err) => {
                    console.log(err);
                    next(err);
                })
        })
        .catch((err) => {
            console.log(err);
            next(err);
        })
}

Async-await based approach

And if you convert this to a async await based approach, you would end up with something very similar.

const createProduct = async (req, res, next) => {
    const { id: userId } = req.user;
    try {
        const user = await User.findOne({id: userId});
        if (!user) {
            console.log('User does not exist');
            return res.status(400).json({
                status: 'error',
                message: 'User does not exist',
            });
        }
        const { name, storeId, price } = req.body;
        try {
            const product = await Product.findOne({name});
            if (product) {
                console.log('Product with the same name already exists');
                return res.status(400).json({
                    status: 'error',
                    message: 'Product with the same name already exists',
                });
            }
            try {
                const store = await Store.findOne({id: storeId});
                if (!store) {
                    console.log('Store does not exist');
                    return res.status(400).json({
                        status: 'error',
                        message: 'Store does not exist',
                    })
                }
                try {
                    const newProduct = new Product({
                        name,
                        storeId,
                        price,
                    });
                    await newProduct.save();
                    console.log('Product saved successfully');
                    return res.status(200).json({
                        status: 'success',
                        message: 'Product saved successfully',
                    });
                } catch (err) {
                    console.log('Error when saving product', err);
                    next(err);
                }
            } catch (err) {
                console.log('Error when fetching store', err);
                next(err);
            }
        } catch (err) {
            console.log('Error when fetching product', err);
            next(err);
        }
    } catch (err) {
        console.log('Error when fetching user', err);
        next(err);
    }
}

There is nothing wrong with this approach and works pretty well for small functions. But when the number of async operations increase, the code goes into this pyramid structure which is hard to understand. Usually called the Pyramid of doom.

Linear async-await

To overcome this and to give our code a linear structure, we can write a utility function which fires the promise and returns the error and success states.

const firePromise = (promise) => {
    return promise
        .then((data) => {
            return [null, data];
        })
        .catch((err) => {
            return [err, null];
        })
}

We can pass any async operation which returns a promise to this function and it will give us error and success states in an array. Goes something like this.

const [error, user] = await firePromise(User.findOne({id: userId}));

Now we can refactor our createProduct handler to use our firePromise function.

const createProduct = async (req, res, next) => {
    let error, user, product, store;
    const { id: userId } = req.user;
    try {
        [error, user] = await firePromise(User.findOne({id: userId}));
        if(error) {
            console.log('Error when fetching user', error);
            next(error);
        }
        if(!user) {
            console.log('User does not exist');
            return res.status(400).json({
                status: 'error',
                message: 'User does not exist',
            });
        }
        const { name, storeId, price } = req.body;
        [error, product] = await firePromise(Product.findOne({name}));
        if(error) {
            console.log('Error when fetching product', error);
            next(error);
        }
        if (product) {
            console.log('Product with the same name already exists');
            return res.status(400).json({
                status: 'error',
                message: 'Product with the same name already exists',
            });
        }
        [error, store] = await firePromise(Store.findOne({id: storeId}));
        if(error) {
            console.log('Error when fetching store', error);
            next(error);
        }
        if (!store) {
            console.log('Store does not exist');
            return res.status(400).json({
                status: 'error',
                message: 'Store does not exist',
            })
        }
        const newProduct = new Product({
            name,
            storeId,
            price,
        });
        [error] = await firePromise(newProduct.save());
        if (error) {
            console.log('Error when saving product', err);
            next(error);
        }
        console.log('Product saved successfully');
        return res.status(200).json({
            status: 'success',
            message: 'Product saved successfully',
        });
    } catch (err) {
        console.log('Unexpected error');
        next(err);
    }
}

In my opinion, this is much more readable because of its linear structure. This function can be used with any JS framework to write readable and maintainable async code.

This was inspired by await-to-js library, and I use it in almost all my JS projects. Go give them a star.

Cheers!

Lazy loading react components with dynamic imports and intersection observer

Siddharth Venkatesh — Fri, 13 Aug 2021 06:43:38 +0000

Lazy Loading

Lazy loading is a way by which we can load content only when they are needed. This is achieved by code-splitting, where we split our app into multiple chunks. The idea here is to serve the user with only the content they can view, and serve the other contents as and when the user visits them.

Route based code-splitting

For example, lets say we have a website where we have /home, /profile and /about routes, and /home is where the user first lands on. If we can compile the three routes into three bundles, we can serve them as and when the user visits the respective pages. The code for home route would contain only the code in <Home /> component. And when the user visits /about, the content for this route will be downloaded and displayed. If we have a fairly large app with lots of routes, this would give us a significant performance gain on initial page load times.

Component based code-splitting

The above example describes what is a route-based code-splitting strategy. We can take this a step further with a component based code-splitting strategy. Lets say we have a heavy form component buried deep in the app which the user would rarely use. It doesn't make sense for us to add it to our main bundle and it's a perfect recipe for lazy-loading.

Dynamic Imports

We can achieve this in React using Dynamic imports. React provides us way by which we can leverage dynamic imports with React.lazy and Suspense From React docs.

Lets build our example. We have a dummy form component <HeavyForm />. It does nothing, but you get the idea.

And, if we want to dynamically import it, we would do something like this

Intersection Observer

If you run the code now, you can see the HeavyForm is downloaded as a separate js file. This means that HeavyForm was bundled as a separate chunk and it is not part of our main bundle.

Great! But still, it is downloaded as soon as the page loads. We want this to be downloaded only when it is in the viewport, i.e when the user actually sees it.

This is where Intersection Observer comes in. IntersectionObserver lets us know if the target element is in the viewport or not. We can safely assume that if the IntersectionObserver fires, the target element is in the viewport. We can leverage this and lazily load any component when it is in the viewport.

I'm going to be using react-intersection-observer library, which uses native IntersectionObserver underneath and gives us neat hooks for ease of use.

This is what the complete implementation would look like with IntersectionObserver. react-intersection-observer gives us useInView hook, which gives us a ref and inView flag. The ref should be attached to the target element and inView lets us know if the target element is in the viewport. The threshold option is a value between 0 and 1 indicating the percentage of element that should be visible before triggering.

Now, <HeavyForm /> would only be downloaded when it is in the viewport.

Conclusion

This technique can be extended to multiple routes and components for easy gains on initial page load times. Remember to strike a balance between the components you lazy-load and components that are added to the main bundle. There is a penalty of network round trip that needs to be made when requesting lazy-loaded content.
Cheers!

You can take a look at the entire source code here