DEV Community: Mohammad Faisal

Setup Mixpanel Analytics in a NextJS Application

Mohammad Faisal — Sun, 01 Dec 2024 10:49:54 +0000

Analytics is crucial for any profitable application, whether it’s a small application with 100 users or a large application with 10,000 users.

Understanding your users is one of the most critical things. And mixpanel is one of the best tools to do that.

Today, we will learn how to integrate and start mixpanel tracking.

Setup the project

I assume you already have a NextJS project setup. Also, create a new Mixpanel account from here (If you haven’t already).

Keep in mind that I am showing for NextJS but it’s applicable for any ReactJS app too.

Then, install the dependency

npm install mixpanel-browser

Get a token

First, add the following environment variable

NEXT_PUBLIC_MIXPANEL_TOKEN="YOUR_TOKEN_HERE"

Now, you can get the mixpanel token from your project’s dashboard.

Then go to Settings -> Project Settings

Then grab the Project Token and add it in the environment file.

Create Configuration File

Create a file named mixpanel.ts and add the following code

import mixpanel from 'mixpanel-browser';

// Define event names as constants to prevent typos
export const ANALYTICS_EVENTS = {
  PAGE_VIEW: 'Page View',
  BUTTON_CLICK: 'Button Click',
  FORM_SUBMIT: 'Form Submit',
  USER_SIGNED_IN: 'User Signed In',
  USER_SIGNED_UP: 'User Signed Up',
  USER_SIGNED_OUT: 'User Signed Out',
  // ... other events
};

// Initialize mixpanel
const MIXPANEL_TOKEN = process.env.NEXT_PUBLIC_MIXPANEL_TOKEN;

if (MIXPANEL_TOKEN) {
  mixpanel.init(MIXPANEL_TOKEN, {
    debug: process.env.NODE_ENV === 'development',
    track_pageview: true,
    persistence: 'localStorage',
    ignore_dnt: true,
  });
}

So, initialize the mixpanel as high as possible in your component tree.

Add Analytics Functions

Now, after you add the configuration, it’s time to add some reusable functions to track mixpanel events.

So add the following code in the same file:

export const MixpanelAnalytics = {
  track: <T extends Record<string, any>>(
    eventName: string,
    properties?: T & CommonEventProperties
  ) => {
    try {
      if (MIXPANEL_TOKEN) {
        mixpanel.track(eventName, {
          ...properties,
          timestamp: Date.now(),
          path: typeof window !== 'undefined' ? window.location.pathname : undefined,
        });
      }
    } catch (error) {
      console.error('Error tracking event:', error);
    }
  },

  pageView: (pageName: string, properties?: Record<string, any>) => {
    try {
      if (MIXPANEL_TOKEN) {
        MixpanelAnalytics.track(ANALYTICS_EVENTS.PAGE_VIEW, {
          page: pageName,
          ...properties,
        });
      }
    } catch (error) {
      console.error('Error tracking page view:', error);
    }
  }
};

If you analyze these 2 functions above

track

This function is used to track any kind of event.

For example, if you want to track a user, click a button to visit an external website. Maybe for affiliate calculation

You can do the following:

MixpanelAnalytics.track("VISIT_WEBSITE", {
  website_name: name,
  website_url: website,
  visit_count: (mixpanel.get_property('total_website_visits') ?? 0) + 1,
});

pageView

This is a pretty straightforward method to track every page view inside your application.

Now remember — when we initialized mixpanel, we already told it to track page views:

mixpanel.init(MIXPANEL_TOKEN, {
  track_pageview: true,  << HERE
  ...others
});

So this custom tracking is only for more detailed analysis.

Know your users

Now, tracking clicks is cool and all, but many times, it’s not enough.

Maybe you want to track specific users. Maybe you want to know who is doing what. Maybe you are creating a funnel to analyze user behavior.

For these scenarios,mixpanel provides 2 functions.

identify
reset

So, on a high level, after a user logs in, you can call

mixpanel.identify("whatever identified you want (usually email or userid)")

And on logout, you can reset it

mixpanel.reset()

Now you can also add additional context or details about your users using the people.set() method

For example,

mixpanel.people.set({
    plan: 'Enterprise',   // Update "plan" from "Premium" to "Enterprise"
    company: 'mixpanel'  // Create new "company" profile prop
});

There are some additional methods like append, union, increment etc., to handle more scenarios, but skip them as they are not the focus of this article. You can read more here

But what about anonymous users?

Now, in many applications (especially public sites) — it’s not mandatory to log in to see the contents.

But how do we track those people if they don’t log in?

To handle all these scenarios, let’s create two more utility functions.

export const MixpanelAnalytics = {
  // ... other methods ...

  identifyKnownUsers: (userId: string, userProperties?: Record<string, any>) => {
    try {
      mixpanel.identify(userId);
      if (userProperties) {
        mixpanel.people.set(userProperties);
      }
    } catch (error) {
      console.error('Error identifying user:', error);
    }
  },

  identifyUnknownUsers: () => {
    try {
      const anonymousId = 
        mixpanel.get_distinct_id() ||
        `anon-${Date.now()}-${Math.random().toString(36).substr(2, 9)}`;

      MixpanelAnalytics.identify(anonymousId);
      MixpanelAnalytics.setPeople({
        $first_seen: new Date().toISOString(),
        user_type: 'anonymous',
        platform: typeof window !== 'undefined' ? window.navigator.platform : 'unknown',
      });
    } catch (error) {
      console.error('Error initializing user:', error);
    }
  }
};

So you can track your known and unknown users with this.

An example usage can look like the following: In one of the root files — (layout.tsx file in app router, _app.tsx in pages router)

Add the following

useEffect(() => {
  const getUser = async () => {
    const user = functionToGetUser();
    if (user) {
      await MixpanelAnalytics.identifyKnownUser(user);
    } else {
      await MixpanelAnalytics.initializeUnknownUser();
    }
  };
  getUser();
}, []);

So this will initialize the user appropriately when they visit the site.

You can gather data and assign it to this particular user going forward.

Example usage

Now comes the fun part. Notice the following code and update it according to your needs.

const handleClick = () => {
  MixpanelAnalytics.setPeople({
    $last_visited: websiteName,
    last_visit_date: new Date().toISOString(),
    total_visits: (mixpanel.get_property('total_visits') ?? 0) + 1,
  });

  MixpanelAnalytics.track(ANALYTICS_EVENTS.VISIT_WEBSITE, {
    website_name: websiteName,
    website_url: website,
    visit_count: (mixpanel.get_property('total_visits') ?? 0) + 1,
  });
};

In the above function we are tracking the particular user’s profile with the tracking data and also making sure that we are counting their visits to the particular website as well.

Cool right?

Best practices

When working with analytics — it’s very important to keep the data consistent.

So, make sure to add proper types for analytics events.

For example

Define constants for the events.

Never use plain strings for event names.

export const ANALYTICS_EVENTS = {
  PAGE_VIEW: 'Page View',
  BUTTON_CLICK: 'Button Click',
  // ... other events
} as const;

Type safety

For events payload, make sure to use consistent structure by using types

interface CommonEventProperties {
  timestamp?: number;
  path?: string;
  referrer?: string;
}

User Properties

Always maintain consistent user properties across sessions.

MixpanelAnalytics.setPeople({
  $email: user.email,
  $name: user.user_metadata?.full_name,
  $created: user.created_at,
  last_sign_in: new Date().toISOString(),
  user_type: 'registered',
  auth_provider: user.app_metadata?.provider || 'email',
});

Otherwise, down the road, the data will be useless.

Conclusion

Remember to handle analytics initialization properly in your client-side components.

Also, ensure that sensitive user data is handled appropriately according to your privacy policy and data protection regulations.

Hope you learned something new today.

Have a great day!

NextJS Internationalization using Google Translator

Mohammad Faisal — Mon, 26 Aug 2024 18:00:00 +0000

How to internationalize a NextJS website?

Setup a CMS or use i18n
Connect each piece of text with a key in the translation file.
Translate each piece of text.

But all of these take a huge amount of time.

Today I will show you the fastest way to add internationalization in a NextJS website.

In 10 minutes you are going to have a fully translated website using the power of Google Translate.

** Here is the demo and here is the github project **

Let’s begin!

Step 1: Create a new project

Creating a new NextJS project is simple. Just run

npx create-next-app@latest

On your local machine create a basic NextJS project.

Also, we need to install a dependency named nookies . This will be used for cookie management.

yarn add nookies

You are done with dependencies.

Step 2: Configure the public folder

Create 2 files under the public/assets directory.

a. lang-config.js

Create the following file in public/assets/lang-config.js

window.__GOOGLE_TRANSLATION_CONFIG__ = {
    languages: [
        { title: 'English', name: 'en' },
        { title: 'Deutsch', name: 'de' },
        { title: 'Español', name: 'es' },
        { title: 'Français', name: 'fr' },
    ],
    defaultLanguage: 'en',
};

This file defines the available languages and the default language.

You can add whatever language is supported by Google Translate here.

b. translation.js

Create another file in public/assets/translation.js

function TranslateInit() {
    if (!window.__GOOGLE_TRANSLATION_CONFIG__) {
        return;
    }
    new google.translate.TranslateElement({
        pageLanguage: window.__GOOGLE_TRANSLATION_CONFIG__.defaultLanguage,
    });
}

This file checks for the configuration and initializes the Google translator on the site.

Step 3: Import scripts

Now it’s time to import these scripts into the project.

Head over to the

root layout.tsx file (if you are using app router)
or _document.tsx file (if you are using pages router)

And use NextJS’s Script tag to include these scripts.

return (
  <html lang="en">
      <Script src="/assets/lang-config.js" strategy="beforeInteractive" />
      <Script src="/assets/translation.js" strategy="beforeInteractive" />
      <Script src="//translate.google.com/translate_a/element.js?cb=TranslateInit" strategy="afterInteractive" />
      <body className={inter.className}>
          <div id="google_translate_element"></div>
          {children}
      </body>
  </html>
);

Let’s understand what’s going on here.

The first 2 Script tags are loading the config and initialization. We used strategy beforeInteractive to ensure that they are available from the get-go of the site.

The third script is the script that does the actual translation for us. This can be loaded after the hydration so we used strategy afterInteractive here.

Step 4: Create the language switcher

Now comes the most important component.

Create a new component named language-switcher.tsx and add the following code.

'use client';
    import { useEffect, useState } from 'react';
    import { parseCookies, setCookie } from 'nookies';

    // The following cookie name is important because it's Google-predefined for the translation engine purpose
    const COOKIE_NAME = 'googtrans';

    // We should know a predefined nickname of a language and provide its title (the name for displaying)
    interface LanguageDescriptor {
        name: string;
        title: string;
    }

    // Types for JS-based declarations in public/assets/scripts/lang-config.js
    declare global {
        namespace globalThis {
            var __GOOGLE_TRANSLATION_CONFIG__: {
                languages: LanguageDescriptor[];
                defaultLanguage: string;
            };
        }
    }

    const LanguageSwitcher = () => {
        const [currentLanguage, setCurrentLanguage] = useState<string>();
        const [languageConfig, setLanguageConfig] = useState<any>();

        // Initialize translation engine
        useEffect(() => {
            // 1. Read the cookie
            const cookies = parseCookies();
            const existingLanguageCookieValue = cookies[COOKIE_NAME];

            let languageValue;
            if (existingLanguageCookieValue) {
                // 2. If the cookie is defined, extract a language nickname from there.
                const sp = existingLanguageCookieValue.split('/');
                if (sp.length > 2) {
                    languageValue = sp[2];
                }
            }
            // 3. If __GOOGLE_TRANSLATION_CONFIG__ is defined and we still not decided about languageValue - use default one
            if (global.__GOOGLE_TRANSLATION_CONFIG__ && !languageValue) {
                languageValue = global.__GOOGLE_TRANSLATION_CONFIG__.defaultLanguage;
            }
            if (languageValue) {
                // 4. Set the current language if we have a related decision.
                setCurrentLanguage(languageValue);
            }
            // 5. Set the language config.
            if (global.__GOOGLE_TRANSLATION_CONFIG__) {
                setLanguageConfig(global.__GOOGLE_TRANSLATION_CONFIG__);
            }
        }, []);

        // Don't display anything if current language information is unavailable.
        if (!currentLanguage || !languageConfig) {
            return null;
        }

        // The following function switches the current language
        const switchLanguage = (lang: string) => () => {
            // We just need to set the related cookie and reload the page
            // "/auto/" prefix is Google's definition as far as a cookie name
            setCookie(null, COOKIE_NAME, '/auto/' + lang);
            window.location.reload();
        };

        return (
            <div className="text-center notranslate">
                {languageConfig.languages.map((ld: LanguageDescriptor, i: number) => (
                    <>
                        {currentLanguage === ld.name ||
                        (currentLanguage === 'auto' && languageConfig.defaultLanguage === ld) ? (
                            <span key={`l_s_${ld}`} className="mx-3 text-orange-300">
                                {ld.title}
                            </span>
                        ) : (
                            <a
                                key={`l_s_${ld}`}
                                onClick={switchLanguage(ld.name)}
                                className="mx-3 text-blue-300 cursor-pointer hover:underline"
                            >
                                {ld.title}
                            </a>
                        )}
                    </>
                ))}
            </div>
        );
    };

    export { LanguageSwitcher, COOKIE_NAME };

The code is pretty self-explanatory. But let me give you an overview.

This has to be a client component as we are using useEffect and useState .

First, we read the cookies

const COOKIE_NAME = 'googtrans';

const cookies = parseCookies();
const existingLanguageCookieValue = cookies[COOKIE_NAME];

The cookies tell us which language is selected. If nothing is found we take the default language.

if (global.__GOOGLE_TRANSLATION_CONFIG__ && !languageValue) {
    languageValue = global.__GOOGLE_TRANSLATION_CONFIG__.defaultLanguage;
}

Otherwise

if (languageValue) {
   setCurrentLanguage(languageValue);
}

Then we set the current global config.

if (global.__GOOGLE_TRANSLATION_CONFIG__) {
   setLanguageConfig(global.__GOOGLE_TRANSLATION_CONFIG__);
}

Finally, we have a function to switch the language. This is again as simple as setting the cookie name.

const switchLanguage = (lang: string) => () => {
    setCookie(null, COOKIE_NAME, '/auto/' + lang);
    window.location.reload();
};

That’s basically it. You have freedom about the language you want to use.

Step 5: Use it

Now the only thing to do is add this component somewhere. I am adding it to the page.tsx file.

import { LanguageSwitcher } from '@/components/language-switcher';

export default function Home() {
  return (
      <main className="flex">
          <LanguageSwitcher />
          <div>
              {`Add some text here to test`}
          </div>
      </main>
  );
}

Then if you run the application

yarn dev

You should see the following.

Beautiful! You can switch between the buttons to see how the translations work!

But there is a bar at the top. Which is not beautiful. We can easily remove it using a little CSS trick.

Add the following to your globals.css file

    .skiptranslate {
        display: none !important;
    }

And voila!

And now you have a working language switcher that can support over 200 languages.

All you have to do is add a new language to the lang-config.js file under the public/assets folder.

Final thoughts

This is by far the fastest way to translate a website.

But…

You have less control over the content as Google Translate will do all the work.

Also probably not good for SEO optimization.

But completely fine for informational websites or landing pages.

Hope you have a wonderful day ahead!

Github Repo: https://github.com/Mohammad-Faisal/nextjs-internationalization-with-google-translate
Live demo: https://nextjs-internationalization-with-google-translate.vercel.app/

How I improved score from 21 to 61 for a Production Nextjs site.

Mohammad Faisal — Fri, 23 Aug 2024 17:01:22 +0000

Recently, I had to improve the performance of a Production NextJS website.

You can visit the website here (the changes are not live yet)
Invygo Rent-A-Car

Today, I will explain how we did it and the thought process.

Let’s start with the score

Here is where we started.

For a fair comparison, I deployed it to a test account using Vercel, and I will use this one for all benchmarks.

My focus is not only on technical improvement — but also on improving the user experience.

Let’s go…

Optimization 1: Parallel request.

Our filter page makes 4 API calls to get the appropriate filters for that region.

But as you can assume, nothing is stopping us from making these requests parallel.

It was simply wrapping everything into a Promise.all() .

This optimization improved the experience, but I didn’t benchmark the improvement.

Optimization 2: Static generation of the pages.

Our pages were server-side rendered using getServerSideProps . However, as we can already predict the initial pages, the second thing I did was generate those pages statically.

The main benefit is that we already have the pages generated during the build time, so when a user clicks on a button to view the page, we don’t have to make the 4 API calls we discussed above on the first load.

It improves the score and enhances the user experience.

Optimization 3: Prefetch the static pages

I wanted to take this one step further to improve the user experience even more.

We can use the Next Link component to prefetch URLs.

However, the pages we are discussing are referenced from many places on the website, as we want users to visit these pages frequently.

That’s why I decided to prefetch these 5–6 URLs whenever any user lands on any page of our website.

The way to do that is to create a custom component named. PrefetchStaticPaths

const PrefetchStaticPaths: React.FC = () => {
  const router = useRouter();

  useEffect(() => {
    const prefetchPaths = async () => {
      const paths = [...] // list of paths that you want to prefetch

      for (const path of paths) {
        await router.prefetch(path);
      }
    };

    prefetchPaths();
  }, [router]);

  return null;
};

Then, I included this component in the _app.tsx file.

This means that the pages will be ready to view whenever a user visits the website. And when you try to visit, it will be almost instant.

I also improved the above code to make those requests parallel.

So, the modified version looks like this.

const prefetchPromises = paths.map((path) => {
  const startTime = performance.now();
  console.log('Start prefetching path:', path);

  return router
    .prefetch(path)
    .then(() => {
      const endTime = performance.now();
      const duration = endTime - startTime;
      console.log(`Finished prefetching path: ${path}. Time taken: ${duration.toFixed(2)}ms`);
    })
    .catch((error) => {
      console.error(`Error prefetching path: ${path}`, error);
    });
});

await Promise.all(prefetchPromises);

This surely improved the user experience but didn’t have much impact on the page speed score.

Optimization 4: Optimize large images.

Images are often among the heaviest parts of any page. On our filter page, several banners increased the page size.

There is a nice free tool called TinyPng. You can drop the images there and get similar-looking images of much smaller sizes.

Here is what it looked like in the conversion.

This tool alone reduced the image size by 65%, which is pretty significant. We are talking about 300kb savings for each image.

Want to Take it even further?

There are other tools you can use even further. Like Image compressor

This tool allows you to specify the number of colours, which can reduce the image sizes by an additional 50%.

Usually, the PNG images use 256 colors. But if you use 8, you can get another 50% optimized image.

But don’t overuse it because it can reduce the quality visually.

Optimization 5: Optimize Script Loading

Any serious production application will have many tracking tools. Our app is no different. We are using Google Tag Manager and some other tools.

No NextJS offers a nice way to load these scripts lazily.

We have to add the strategy to the script tag.

This reduces the time for the initial load.

Optimization 6: Code splitting and lazy loading.

Not everything needs to be loaded on the first build. You can identify which components are loading more stuff than required if you inspect the output when you run yarn build

You will see that we have a section called First Load JS shared by all.

These resources will be loaded no matter what.

Let’s take a deeper look.

If you check the name, you will see one is framework , and another one is main . We will check that later, but for now, focus on the one that starts with _app .

This file is shared across the whole project, so whatever you load here will be loaded everywhere. So, it's very important to make sure that everything is correct here.

One way to load things efficiently is by loading components dynamically. Including the scripts.

So, I converted the imports from

import BottomNavbar from '@/widgets/navigation/BottomNavBar';

into

const BottomNavbar = dynamic(() => import('@/widgets/navigation/BottomNavBar'), { ssr: false });

And did this for all components.

Now, look at the output.

So, using this simple trick, we slashed off 60kb. Not bad, huh?

The final result so far

After doing all of these, our current score is following

That’s not great, but at least it’s usable now, right?

Can we go even further?

Yes. How?

Well… let me tell you.

Potential 1: Analyze the bundle

There is a nice tool called **webpack bundle analyzer. **You can hook this tool with your NextJS project to see which part of the output is casing the heaviest load.

I ran it, and I saw the following.

As you can see, we have used an icon library called lucide-react . And this is taking up the bulk of the space.

We need custom icons to replace it, which will take some time to manage. So that’s for another day.

Potential 2: Analyze the packages

When we are using the libraries on a project, we often do not consider their size.

Another tool is **BundlePhobia. **It allows you to check the size of any package, decide if it’s too heavy, and maybe use an alternative.

So, I will do this in the coming days to see how far we can take it.

Final Remarks

Optimizing any website is a constant process and battle. In this article, I tried to show some of the ways we can use it- there are, of course, many more.

Let's talk about those some other day.

Hope you enjoyed it. Have a great day! :D

Using Digital Ocean Spaces as Strapi Storage

Mohammad Faisal — Thu, 16 May 2024 11:50:03 +0000

To read more articles like this, visit my blog

So recently, I migrated a website from WordPress to Strapi + NextJS.

The client didn’t want Strapi’s own cloud service and wanted to stick to Digital Ocean as he already knows that well.

So, I set up Strapi on Digital Ocean using their app service, which is, I would say, not " beginner-friendly” but should be easy enough for someone technical.

The problem

So after I uploaded a couple of content with images. They worked pretty well.

But as soon as I added a new field to a new collection or do anything that triggers a new build the images were gone.

The Issue

The problem was that strapi, by default, uses a local upload provider. A local provider means the images are stored on the machine.

Every time you build — the machine restarts, and the images are gone.

Solution

The solution is to use persistent storage. In fact, Strapi provides several options.

Upload | Strapi Documentation

Now, here is another problem. Strapi’s official documentation supports

But my client wanted to stick to Digital Ocean. So what to do?

Comes the Digital ocean spaces

Digital Ocean Spaces is actually an S3-compatible storage provider, which means it only provides a wrapper on top of the S3.

To use this, First, create a digital ocean space using the following guide
How to Create a Spaces Bucket | DigitalOcean Documentation

Then, get the access key and the secret access key using this guide.
How to Manage Administrative Access to Spaces | DigitalOcean Documentation

Install AWS provider

Now, go to the Strapi application and install s3 provider

npm i @strapi/provider-upload-aws-s3

Then open up your plugins.ts file and add the following code.

export default ({ env }) => ({
  // ... other configs,
  upload: {
    config: {
  provider: "aws-s3",
      providerOptions: {
        credentials: {
          accessKeyId: env("DO_SPACE_ACCESS_KEY"),
          secretAccessKey: env("DO_SPACE_SECRET_KEY"),
        },
        region: env("DO_SPACE_REGION"),
        endpoint: env("DO_SPACE_ENDPOINT"),
        params: {
          Bucket: env("DO_SPACE_BUCKET"),
        },
      },
      actionOptions: {
        upload: {},
        uploadStream: {},
        delete: {},
      },
    },
  },
});

Then, add the following values to the .env file.

DO_SPACE_ACCESS_KEY=___
DO_SPACE_SECRET_KEY=___
DO_SPACE_ENDPOINT=https://fra1.digitaloceanspaces.com
DO_SPACE_BUCKET=YOUR_BUCKET_NAME
DO_SPACE_REGION=REGION_NAME (in this case fra1)

Notice several things.

The space endpoint can’t have the bucket name at the beginning. This is a very common mistake that is not mentioned in the docs.

So if your Origin Endpoint looks like this

https://any-bucket-name.fra1.digitaloceanspaces.com

Your DO_SPACE_ENDPOINT value should be

https://fra1.digitaloceanspaces.com

And obviously, the region will be fra1

Modify Middleware

You are not done yet. Although this will work, when you visit the CMS, you will see that the previews are not loading.

To fix this, we have to allow Strapi access to the S3 bucket.

Let’s do that…

Now open the middlewares.ts file and add the following

export default [  
// ...others 
{
    name: "strapi::security",
    config: {
      contentSecurityPolicy: {
        useDefaults: true,
        directives: {
          "connect-src": ["'self'", "https:"],
          "img-src": [
            "'self'",
            "data:",
            "blob:",
            "strapi.io",
            "{BUCKET_NAME}.s3.{REGION}.amazonaws.com",
            "https://{BUCKET_NAME}.s3.{REGION}.amazonaws.com"
          ],
          "media-src": [
            "'self'",
            "data:",
            "blob:",
            "strapi.io",
            "{BUCKET_NAME}.s3.{REGION}.amazonaws.com",
            "https://{BUCKET_NAME}.s3.{REGION}.amazonaws.com"
          ],
          upgradeInsecureRequests: null,
        },
      },
    },
  },
];

Notice a few things

Replace the BUCKET_NAME and the REGION values in the middlewares.ts.

Then restart your Strapi server, and you should be able to upload images to the digital ocean spaces.

This configuration should fix the issues, and you will see the previews.

Upload an image and check the spaces to verify that it’s working. Also, don’t forget to add the .env variables to your digital ocean dashboard.

Thank you for reading this far! Have a great day!

Have something to say? Get in touch with me via LinkedIn or Personal Website

Ditch Your Boring Terminal and Make it More Useful

Mohammad Faisal — Sat, 11 May 2024 11:00:13 +0000

To read more articles like this, visit my blog

Contrary to popular opinion, programmers are human. We like nice things. And the terminal is one of the most important things that we use on a day-to-day basis.

So why live with a boring terminal when you can make it colourful and fun?

Today, we will do precisely that. Let’s begin!

Install Homebrew

If you are using a mac, you are most probably already familiar with homebrew. It helps with installing software on macOS.

Run the following command:

/bin/bash -c "$(curl -fsSL [https://raw.githubusercontent.com/Homebrew/install/HEAD/install.sh](https://raw.githubusercontent.com/Homebrew/install/HEAD/install.sh))"

You may need to install the CLI tools for XCode before that:

xcode-select —-install

If you get an error, run the following command to reset:

xcode-select -r

And that’s it!

2. Install Iterm2

Iterm2 is a terminal emulator for macOS. It’s kind of a replacement for your original terminal. It comes with a bunch of cool features and customizations that we will go over later

Run the following command:

brew cask install iterm2

3. Install ZSH

ZSH, also called the Z shell, is an extended version of the Bourne Shell (sh). it has some advanced features like

Automatic cd: Just type the name of the directory.
Plugin and theme support: ZSH has many plugin and theme frameworks to boost up your terminal even more.

To install it run the following command:

brew install zsh

And you should be set up.

4. Install Oh My ZSH

According to their website…

Oh My Zsh is a delightful, open source, community-driven framework for managing your Zsh configuration

In short, it will help you to manage the configuration and customizations for your Z Shell.

To install, run the following command:

sh -c "$(curl -fsSL https://raw.githubusercontent.com/robbyrussell/oh-my-zsh/master/tools/install.sh)"

Okay, now, you have the basic setup complete.

You should already have a nice-looking terminal than before.

5. Customize the Theme

Now restart your iTerm2 and type the following command:

open .zshrc

it will open the config file with a text editor. You can also use nano if you like.

Now there will be a variable named ZSH_THEME . And the value there is robbyrussell . If we set it to random, then every time we open our terminal, a new theme will be loaded

However, you can choose from a wide variety of themes here.

Let’s use a popular custom theme named powerlevel10k. First, install it.

git clone --depth=1 https://github.com/romkatv/powerlevel10k.git ${ZSH_CUSTOM:-$HOME/.oh-my-zsh/custom}/themes/powerlevel10k

Then set the theme name:

ZSH_THEME="powerlevel10k/powerlevel10k"

Then save the file and run the following command from the terminal:

exec zsh

Then open iterm2 again and configure the terminal by running the following command:

p10k configure

It will take you through the configuration process and at the end of it, you will have a nice-looking terminal!

6. Customize with Plugins

Plugins are what make the zsh so powerful and fun to use. There are a lot of plugins that you can find here. Let’s try one of them!

Do you like typing? I guess not. We can have some autosuggestion inside our terminal by using a plugin named zsh-autosuggestions.

Run the following command.

git clone https://github.com/zsh-users/zsh-autosuggestions $ZSH_CUSTOM/plugins/zsh-autosuggestions

Then open up your .zshrc file. And find the plugins section. You will see there is already a plugin there (git). Add our new plugin.

That’s it. Now when you want to type a command the terminal will give you suggestions!

Final Thoughts

There are a lot of things that you can do with oh my zsh . I have shown the absolute basics here. I hope you enjoyed it.

Have a great day!

Resources:

Have something to say? Get in touch with me via LinkedIn or Personal Website

Why NVM is Tremendously Helpful for Web Developers

Mohammad Faisal — Sat, 11 May 2024 10:31:49 +0000

To read more articles like this, visit my blog

If you work in the Node.js ecosystem, you’ve probably heard about the Node version manager (NVM).

Today, we will see why this exists and why you should care about learning it.

What is NVM?

According to the official docs,

nvm allows you to quickly install and use different versions of Node.js via the command line.

Apart from that, it also keeps you away from many errors.

The problem at hand

Have you ever encountered the EACCESS error when installing a global npm package?

npm install -g some-package

I bet you have! But what’s the solution? Some StackOverflow answers suggest you use sudo, which is a very bad idea and can lead to more problems.

The official npm docs suggest that we use node version manager (nvm) to resolve this issue.

Install NVM

If you are using Linux, run the following command:

curl -o- https://raw.githubusercontent.com/nvm-sh/nvm/v0.39.1/install.sh | bash

If it’s successful you can check the version:

command -v nvm

You may have to close your terminal and reopen it again.

Install the Latest Version of Node.js

If we want to install the latest version of the node we can run the following command:

nvm install node

At the time of writing this article, the latest version of the Node.js is v17.3.0.

Install Specific Version of Node.js

At the time of writing this article, the latest version of Node.js is 16.13.1 . If we want to install this specific version:

nvm install 16.13.1

Check the installed versions

If you want to see the list of versions installed on your machine run the following command:

nvm ls

This will show the versions on your machine.

Use a specific version of the node

Now let’s say you have multiple versions of the node installed on your machine. And you want to switch between them. Maybe you want to work on a legacy app that uses Node.js 14.

You can just run:

nvm use 14

You can check the version of the node by running the following command:

node -v

And now you are using Node 14.

Install the Latest Version of npm

You can install the latest version of npm for the version run the following command:

nvm install-latest-npm

And check the version:

npm --version

So essentially you can have multiple versions of npm on your machine now!

How cool is that?

Final Notes

These tools are here to make our lives easier. Let’s take full advantage of them.

Hope you learned something new today. Have a great day!

Have something to say? Get in touch with me via LinkedIn or Personal Website

How do you Refactor a 2,700-Line React Component?

Mohammad Faisal — Wed, 06 Mar 2024 09:55:16 +0000

To read more articles like this, visit my blog

We are always so excited about the new shiny things that come out every week in the programming world. New ways to structure components, new techniques to reduce those two lines of code, and so on.

But in the real world, things are not so shiny. We often have to deal with codebases that have been evolving for many years, and many developers have left their mark on the components' different parts.

Our story is about a component that has 2700 lines of code. Let’s try to explain how things went south and how we can do better.

Background

I worked in a company with a fleet management tool dashboard showing vehicles roaming around the city in real-time.

The dashboard component is the hero of our story today. It has many functionalities. Nothing fancy, though.

A map that shows the vehicle markers
A way to search through the vehicles
A popup that shows vehicle details when clicked
A list of vehicles at the bottom
Some filtering options.

Sure, there are several features that this single component is responsible for displaying. But is it that much to contain 2700 lines of code?

The Component

Obviously, I will be stupid to paste 2700 lines of code into this article. (Also, it will be illegal :P ) But let me show you the general structure of the component.

// import statements -> 85 lines 
import React from 'react';
import styled from 'styled-components'
...
...

--------------

//constant declaration -> 15 lines
const MAX_LOOP= 40;
const FAILURE_MESSAGE = "Something Brutal Happened!"
...
...

--------------

//styled component declaration -> 580 lines!!!!!
const SomeStyle = styled.div`
 display:flex;
 flex-direction: column;
`
...
...

---------------

//Helper components -> 200 lines
const MapPortal = () => {

  return <div>
    // .. some placeholder component
  </div>
}
...
...


---------------

// Props and State Type Declaration -> 70 lines

type Props = {
  vin: string;
  .....
}

type State = {
  mapVisibility: boolean;
  ....
}

---------------

// The actual component with all it's helper methods -> 1700 lines!!!!!!!!!!!!!!!!!!

export class TheActualComponent extends React.Component<Props , State>{

  constructor(props: Props){
    // ...
  }

  componentDidMount() {
    // ... And other lifecycle methods
  }

  randomHelperMethod(){
    // These functions extracts some logic into a separate function
  }

  onClickHandlers() {
    // Lots of them!
  }

  render(){
    return <div>
       // {Something worth of it?}
    </div> 
  }

}


// redux+type declarations -> 200 lines

export const mapStateToProps = {
  // .. lots of them!
}
export const mapDisptachToProps = {
  // .. lots of them
}

export Component

Okay Now Tell Me What’s Wrong Here

To be 100% honest? Almost everything. Let me explain those.

1. Constant Declaration

This is such an obvious one. Unfortunately, I have seen many examples of it throughout many companies and many components.

You should Keep the Constants in a Separate File

It doesn’t matter if they are being re-used or not. It’s still better to store them in a separate File. Because down the line, someone else would create a separate constant with the same value, eventually creating confusion.

I do it this way.

// for related constants
export DeclaredConstants {
  SOME_CONSTANT_1 = "SomethingSpecial1";
  SOME_CONSTANT_2 = "SomethingSpecial2"
}

// or for single constants

export const SOME_CONSTANT_3 = "SomethingSpecial3"

And then import the inside my component and use them.

import { DeclaredConstants } from "ConstantDeclaration.ts"

console.log(DeclaredConstants.SOME_CONSTANT_1)

This particular component isn’t a big deal, but best practices are best practices.

2. Styles and Helper Methods

I think it’s a very common mistake and (sometimes allowable) for smaller components to put the styles and helper methods in the same file.

If your component is only 30–50 lines of code, then it can make sense to keep the styles and helper methods in the same file.

But having 580 lines of style declaration doesn’t make sense in any scenario. Because not very often you would need to touch these styles.

What I do is follow the following folder structure to keep things organized.

-component-name 
  |-- __tests__
  |-- styles.ts
  |-- helpers.ts
  |-- index.ts

The responsibility of the files is clear from the name of the files themselves. It’s so simple to simply split our massive component to 1/3rd of its current size just by putting things where they should be!

An Example

export const SomeStyle = styled.div`
  display: flex;
`

If you are using Raw CSS or SCSS, then you probably don’t make this mistake, but the projects using styled-components or material-ui mostly follow this bad practice.

Once someone started it. It became the standard practice.

So don’t fall into this trap. Create a separate file for styles ahead of time that can save your component in the future.

3. Dumb Components

There are two types of components.

Dumb components -> only act as a container or view
Intelligent components -> Shows something based on some logic

Now there is no reason to put two components in the same file. It directly violates the Single Responsibility Principle.

Every class or component should do one thing and one thing only

Sometimes we get lazy (Myself included) and put simple container components into the actual component. But what do you think when the next developer comes?

Does he take out the smaller component into its own file?

Umm… Probably not. So after 4–5 years, you now have 200 lines of helper dumb components that can easily be extracted from separate files.

4. Class Component

Not sure if you noticed, but this Massive component is using ClassComponent. I’m sure you know the reason. It was written at a time when functional components were not that common.

But nowadays, using functional components make more sense because

They are easy to maintain
Less code
More performant? (Arguably)

But even I wouldn’t try to convert it into a functional component at this stage. We need to refactor many things before converting this into a functional component.

Let’s see how much we could gain.

Let me show you an estimation of how much we can improve without even understanding what this component is doing.

We can just export the constants to constants.ts , Styles into the styles.ts and helper methods into the helpers.ts file.

Styles                    =  600
Constants(Types+Others)   =  100
Helper Methods.           =  600 (roughly)
Dumb Components           =  200
---------------------------------
Total Gain                =  1400 lines!

This is a work of two hours maximum. All we need to do is.

Putting things into appropriate files and import them

We can reduce a component from 2700 to 1300 lines!

Some might say that it’s still a lot. But hey!!! One step at a time, right?

Can we do better?

Yes, of course. When we look into the internal logic, we can do even better.

Break the actual components and re-usable parts into even smaller components
Using Functional component
Taking advantage of hooks
Using functional redux

And so on….. But that is a story for another day.

Show me the Good Parts.

Obviously, This component has lots of problems but some good things.

Typescript

Although the Type declarations add up to around 200 lines in total, they are worth it. Especially without Typescript, it would be impossible to maintain this component.

Extracting logic

Some of the dumb logic is extracted out of the view logic itself. For example, showing a message based on the vehicle status.

showVehicleStatusMessage (status: VehicleStatus){
  switch(status): {
    case SOME_STATUS: 
      return "Some Message"
    case SOME_OTHER_STATUS: 
      return "Some Other Message"
  }
}

It’s better to have them in a separate function like this instead of writing the logic into the view. Which can look something like this…

return (
  {vehicleStatus ===  SOME_STATUS && <div> {"Some Message"} </div> 
)

So it’s not all bad; some developers tried to do things correctly. At the end of the day, software development is a team effort.

What did I learn?

The most important takeaway for me is to understand the importance of following the best practices.

Best practices are there for a reason

The impact of following best practices may not be evident on the first day, but if you don’t follow those, you will feel the pain someday!

Conclusion

I hope you learned something new today. Have a great day!

Have something to say? Get in touch with me via LinkedIn or Personal Website

You are Wrong. Frameworks Do Matter

Mohammad Faisal — Wed, 28 Feb 2024 16:18:27 +0000

To read more articles like this, visit my blog

I am sure you have been asked this question at least once by someone, especially if you are a mid-level or senior engineer.

Which language/framework should I learn first?

And the answer is always very straightforward.

Framework doesn’t matter…. Learn whatever you want.

And….. There is nothing wrong with this statement. Theoretically, frameworks don’t matter because everything changes so fast in the tech world.

Right?

Umm... I don’t think so. However, the statement is true, but in a real-life framework matters. They matter very much!

Let me explain why…

If you go to any job search platform like LinkedIn, Toptal, or any other for that matter, you will see that,

Companies are looking for people with specific skill sets.

The only exception is FAANG (or MAANG… whatever) companies. Because they can’t find people with the specific skillset as they usually use custom technologies.

But, in reality….

Only 4% of people work for Big tech Companies. (give or take)

What about the rest of us?

The story is different on the other side.

For all the other companies that are small startups or even mid-level, the tech stack really matters.

They can’t afford to hire people, teach them specific frameworks, and invest 2–3 months to get up to speed.

So what do they do?

They look for a specific skillset

If a company uses React, then it’s highly unlikely that they will search for front-end engineers having five years of Angular experience.

Most of the time, the framework's name is written in the Job description, and the resumes are sorted based on that.

Sure… A senior engineer with this much experience in Angular can quickly pick up React.

But let's say they find a React engineer, even slightly less qualified. Who do you think they will hire? I think you already know the answer.

Don’t get me wrong

I am not saying that the basics are not important. You should have a solid foundation for whatever you are learning.

But when the time comes to dive deep into a framework, please don’t say that framework doesn’t matter.

Framework does matter, and you should learn the most popular one if possible, especially if you are starting out.

Early in my career, I started with React. I imagine my career could be very different if I started with something else. And mostly for the worse.

Let me explain why.

The popular framework is easier to learn.

If you are starting out, then learning any new framework can be challenging.

If the framework is the most popular one, then probably most of the problems are already solved by someone.

You will get better resources and a clear path.

Companies are using this.

Startups and new companies usually pick up the most popular option regarding technology choice.

There are a lot of reasons.

It’s easier to hire people.
Better support.
Better maintainability

So unless you are creating your own start-up, it will be easier for you to get a job when you are already familiar with the tech stack.

You are not preparing for FAANG.

Yes, the huge tech companies don’t care much about which framework you know or which language you are proficient in. But that’s because

They use custom technologies that you can’t learn outside anyway.
They have a lot of projects, and almost every language is being used, so it really doesn’t matter which language you know.

So, learning some random framework can be problematic for someone just starting in the industry.

The rise of freelancing

I have been involved with freelancing for a long time now, and the most important thing that employers ask for is specific skills.

Most of the time, that skill means familiarity with the most popular frameworks like React

Go to any freelance marketplace. I can assure you that getting a job in React is easier than in Angular. The number of jobs is much, much higher.

So What to Do?

I recommend learning the most popular language/framework if you are starting. You should look into the rising technologies even if you are already a mid-level engineer.

If you want to start with the front end, start with HTML, CSS, and Javascript. And probably learn React after that.

If you are starting with the backend, then learning NodeJS can give you the edge as you will learn the basics of backend development and the language Javascript, which is used everywhere.

Final Words

I just wanted to share my view on what a newcomer should learn. Obviously, once you get experienced, these things don’t matter anymore.

But when it comes to guiding juniors, we should emphasize learning a specific framework very well, getting a job, and then going from there.

Just a generic answer like **“It Depends” **doesn't help much.

Thanks for reading this far. I hope you have a wonderful day!

You can reach out to me via LinkedIn or Personal Website

How to Sanitize Data in React

Mohammad Faisal — Mon, 26 Feb 2024 14:08:20 +0000

To read more articles like this, visit my blog

Every modern ReactJS application deals with data in some way. We are fetching data and letting people interact with it throughout the whole application.

But if we are not careful about what data is coming into our website it can cause our application to fail.

One of the major reasons an application crashes on runtime is bad data.

Today we will see one way to sanitize data in React applications.

The problem

You should never trust anyone. Maybe you are expecting a remote API to return a piece of data that looks like this.

userData = {
  basicInfo: {
    name: string;
    age: number;
  }
}

And you have a component that shows these two piece of data in the following manner.

export const SomeComponent = ( {userData} ) => {
return (
    <div>
      <div> {userData.basicInfo.name ?? ""}
    </div>
  )
}

Now for some reason, the developer of the backend thought, okay, let’s change the structure of the data a bit to make it flatter and change it to

userData = {
  name: string;
  age: number;
}

Now, as soon as they do that, your application will fail on runtime because it doesn’t know that the structure was changed.

You will return to the desk and fix the issue, but your users will be greeted with a crashed application the whole time.

Error: Cant't read name of undefined

One way to avoid a crash like this is

<div> {user?.basicInfo?.name} </div>

But you have to put these ? sign everywhere which is not so clean, right?

Take advantage of Typescript.

Now let’s define our User model like the following

interface User {
  name: string;
  age: number;
}

function initUser(options?: Partial<User>): User {
  const defaults = {
    name: '',
    age: 0;
  };
return {
    ...defaults,
    ...options,
  };
}

const p1: User = initPerson();
console.log(p1); // -> {name: '', age: 0}

const p2: User = initPerson({ name: 'Tom'});
console.log(p2); // ->  {name: 'Tom', age: 0}

So this initUser method can be used to create an instance of User, and the options parameter makes it easy to pass parameters if we want.

If we don’t pass anything, we will get back the object with default values.

So before you push the data into the component, you will pass it via the initUser method and stay chilled.

Or you can use a hook.

const useCleanUserData = (userData: User) => {
   return initUser(userData);
}


// inside component
const cleanUserData = useCleanUserData(userData);

This way, we don’t need to worry about whether the API data is clean or not.

You can use this same approach for redux as well. take advantage of the selectors so that the data is cleaned before they enter into the component.

Have something to say? Get in touch with me via LinkedIn or Personal Website

A Well-Architected FastAPI Boilerplate

Mohammad Faisal — Thu, 22 Feb 2024 10:01:52 +0000

To read more articles like this, visit my blog

As a project grows larger, the value of a well-architected and consistent project structure becomes more evident.

Today, we will create a well-architected FastAPI project that should be easily scalable.

We will also do it from scratch, so there will be no surprise for you!

After we are done, our project will have -

A CRUD API ( Of course! )
Alembic for migrations.
SQL Alchemy for Database Operations.
Different environment handling.
Docker-compose file for local development.
Dockerfile for production
Linters.

You can find the repository here if you are looking for that.

GitHub - Mohammad-Faisal/fastapi-well-architected-boilerplate: A Well arthitected FastAPI…

If you want to see how to build one from scratch — Let’s get started!

First step.

First, create the project directory and navigate to it.

mkdir fastapi-well-architected-boilerplate
cd fastapi-well-architected-boilerplate

Then, create the following directories and files:

mkdir -p src/{api,core,db}
touch src/__init__.py
touch src/{main}.py

The project structure should look like this:

.
├── src
│   ├── main.py
└── README.md

The src directory contains the main application code.

The main.py file is the entry point of the application.

Create a virtual environment and install dependencies.

Create a virtual environment and install the required dependencies.

python3 -m venv venv
source venv/bin/activate

Install the two dependencies fastapi and uvicorn using the following command:

pip3 install fastapi uvicorn

The fastapi package is a modern, fast (high-performance), web framework for building APIs with Python 3.7+ based on standard Python type hints.
The unicorn package is a lightning-fast ASGI server implementation using uvloop and HTTP tools.

And create a requirements.txt file to store the dependencies.

pip3 freeze > requirements.txt

Add the following code to the src/main.py file.

from fastapi import FastAPI

app = FastAPI()

@app.get("/")
async def root():
    return {"message": "Hello World"}

Run the application using the following command:

uvicorn src.main:app --reload

Now open your browser and navigate to http://localhost:8000/docs to see the API documentation.

Different environments and configuration settings

The application will have different environments, such as development, testing, and production. Each environment will have its configuration settings.

The configuration settings will be stored in a .env file in the project's root directory. The .env file will contain the following settings:

# .env
ENV=development
DATABASE_URL=sqlite:///./test.db

The ENV setting will be used to determine the current environment. The DATABASE_URL setting will be used to connect to the database. The DATABASE_URL setting will be different for each environment.

Let’s install the pydantic-settings package to load the configuration settings from the .env file.

pip3 install pydantic-settings

The src/config.py file contains the following code:

from pydantic_settings import BaseSettings

class Settings(BaseSettings):
    ENV: str = "development"
    DATABASE_URL: str = "sqlite:///./test.db"

    class Config:
        env_file = ".env"

settings = Settings()

The Settings class contains the configuration settings. The Config class is used to load the settings from the .env file.

Now, you can print the configuration settings using the following code:

from src.config import settings

print(settings.ENV)
print(settings.DATABASE_URL)

Format the code using Black

The black package is a Python code formatter. It will format the code according to the Python PEP 8 style guide.

Install the black package using the following command:

pip3 install black

And create a pyproject.toml file in the root directory of the project with the following content:

[tool.black]
line-length = 88
target-version = ['py37']

The pyproject.toml file contains the configuration settings for the black package. The line-length setting is used to specify the maximum line length. The default is 88, but you can change it to any value.

The target-version setting is used to specify the Python version.

Now, you can format the code using the following command:

black src

This command will format the code in the src directory.

Now black is great for formatting the code but it can be a bit too aggressive. If you want to see what changes it would make without actually making them, you can use the --diff option:

black --diff src

If you want to see what changes it would make without actually making them, you can use the ---check Option:

black --check src

Also, black focuses more on the code formatting. If you want to check the code for style and programming errors, you can use the flake8 package.

Database models and migrations

The application will use the SQLModel package to work with the database.

This is built on top of SQLAlchemy and Pydantic. It will allow us to define the database models using Python-type hints.

Install the SQLAlchemy package using the following command:

pip3 install sqlmodel

The src/database.py file contains the following code:

from sqlmodel import create_engine, Session
from src.config import settings

SQLALCHEMY_DATABASE_URL = settings.DATABASE_URL

engine = create_engine(SQLALCHEMY_DATABASE_URL)

def get_session():
    with Session(engine) as session:
        yield session

This code creates a database engine and a session. The get_session function is used to get the session.

Create our first model

Now, let’s create our first database model. We will create separate domains for each domain in the application.

Let’s create our user domain

touch src/api/user/{__init__.py,models.py}

This will create the following structure:

├── src
│   ├── api
│       ├── user
│           ├── __init__.py
│           ├── models.py
# ... the other stuff
└── README.md

Then, create the user model using the following code.

from sqlmodel import Field, SQLModel

class User(SQLModel, table=True):
    id: int = Field(primary_key=True, index=True)
    username: str
    email: str
    password: str

The User class is a database model. It inherits from the SQLModel class.

The id field is the primary key. The username, email, and password fields are the columns in the database table.

Now we have the models, but we need to create the database tables.

Database migrations with Alembic

The Alembic package is a database migration tool for SQLAlchemy. It will create the database and tables in the production environment.

Install the Alembic package using the following command:

pip3 install alembic

Then, you can run the following command to initialize alembic.

alembic init migrations

This will create a migrations directory in the root directory of the project.

The migrations directory contains the following structure:

├── migrations
│   ├── README
│   
│   ├── env.py
│   ├── script.py.mako
│   └── versions/
├── alembic.ini

Notice that it will also create a alembic.ini file in the root directory of the project.

[alembic]
# other configs
sqlalchemy.url = driver://user:pass@localhost/dbname

You need to specify the sqlalchemy.url setting in the alembic.ini file. This setting is used to connect to the database.

Then edit the env.py file and add the following line to the section [target_metadata]:

from src.database import SQLModel
target_metadata = SQLModel.metadata

Then open the script.py.mako file and add the following at the top

import sqlmodel

Now, we are ready to run our first migration.

alembic revision --autogenerate -m "Initial migration"

This command will create a new migration file in the migrations/versions directory.

Now, you can run the migration using the following command:

alembic upgrade head

This command will create the database and tables in the production environment.

But wait, we are not in the production environment yet. We are still in the development environment. So, we need to create a separate configuration file for the development environment.

Create a separate configuration file for the development environment.

Create a development.env file in the root directory of the project with the following content:

ENV=development
DATABASE_URL=sqlite:///./test.db
# development.env

The development.env file contains the configuration settings for the development environment.

Now you can run the application using the following command:

uvicorn src.main:app --reload --env-file .development.env

This command will run the application in the development environment.

For local development, we need a local database. And we can use docker to create a local database. It will greatly improve the local development experience.

Create a local database using Docker

Let’s see how we can use docker-compose to create a local database and run the local server from the same file.

This will allow us to run the application and the database using a single command.

First, install Docker and Docker Compose on your machine.

Then create the base Dockerfile in the root directory of the project with the following content:

FROM python:3.12.1-alpine3.18

WORKDIR /app

COPY requirements.txt .

RUN pip install --no-cache-dir -r requirements.txt

You can deploy your application anywhere with this docker file.

Then create a docker-compose.yml file in the root directory of the project with the following content:

version: '3.8'

services:
  db:
    image: postgres:13
    environment:
      POSTGRES_USER: user
      POSTGRES_PASSWORD: password
      POSTGRES_DB: test
    ports:
      - "5432:5432"
    volumes:
      - postgres_data:/var/lib/postgresql/data

  app:
    build: .
    command: uvicorn src.main:app --reload --env-file .development.env --host 0.0.0.0 --port 8000
    volumes:
      - .:/app
    ports:
      - "8000:8000"
    environment:
      - DATABASE_URL=postgresql://user:password@db:5432/test
    depends_on:
      - db

volumes:
    postgres_data:

The db service is used to create the database. It uses the postgres:13 image. The POSTGRES_USER, POSTGRES_PASSWORD, and POSTGRES_DB settings are used to create the database.

The app service is used to run the application. It uses the uvicorn command to run the application. The --env-file .development.env setting is used to load the configuration settings from the development.env file.

Database credentials

In this configuration, our local database URL will be postgresql://user:password@localhost:5432/test

But in the docker file, we used DATABASE_URL=postgresql://user:password@db:5432/test because in this context, the database is a service, and the host is db.

You can update the .development.env file to use the new database URL in case we want to access it from the local machine.

# .development.env
ENV=development
DATABASE_URL=postgresql://user:password@localhost:5432/test

Also, don’t forget to upgrade the alembic.ini file to use the new database url.

[alembic]
# other configs
sqlalchemy.url = postgresql://user:password@localhost:5432/test

Now, you can run the application and the database using the following command:

This command will create the database and run the application.

docker-compose up

Now, you can run your first migration on the local database using the following command:

alemibc revision --autogenerate -m "Initial migration"

This command will create a new migration file in the migrations/versions directory.

Now, you can run the migration using the following command:

alembic upgrade head

Now, if you visit your local database, you will see the user table.

Create the user API

Now, let’s create the user API. We will create a router for the user domain.

Create a router.py file in the src/api/user directory with the following content:

from fastapi import APIRouter

router = APIRouter()

@router.get("/")
async def read_users():
    return [{"username": "Rick"}, {"username": "Morty"}]

The router object is an instance of the APIRouter class. It is used to define the routes for the user domain.

Now, you can add the user router to the main application using the following code:

from fastapi import FastAPI

from src.api.user.router import router as user_router

app = FastAPI()

app.include_router(user_router, prefix="/users", tags=["users"])

But we don’t want to deal with dummy data. Instead, we want to use the database to store and retrieve the users.

Create the user service.

Create a service.py file in the src/api/user directory with the following content:

from fastapi import Depends
from src.database import get_session
from src.api.user.models import User
from sqlmodel import  select, Session


class UserService:
    def __init__(self, session: Session = Depends(get_session)) -> None:
        self.session = session

    def get_users(self):
        statement = select(User)
        users = self.session.exec(statement).all()
        return users

The get_users function is used to get the users from the database. Also, we are initializing the session in the constructor.

Now, you can use the get_users Function in the user router using the following code:

from fastapi import APIRouter, Depends

from src.api.user.service import UserService

router = APIRouter()

@router.get("/")
async def read_users(user_service : UserService = Depends()):
    users = user_service.get_users()
    return users

Notice that we are using the Depends function to inject the UserService object into the read_users function.

Now you can run the application and navigate to http://localhost:8000/users to see the users.

Create a user.

Now, let’s create another function to create a user. But before that, we need to create the request and response models.

Create a new file named schems.py in the src/api/user directory with the following content:

from pydantic import BaseModel

class UserCreateInput(BaseModel):
    name: str
    email: str
    password: str

Then update the service.py file to include the create_user function.

def create_user(self, user_create_input):
        user = User(**user_create_input.model_dump())
        self.session.add(user)
        self.session.commit()
        self.session.refresh(user)
        return user

Finally, add the route.

from fastapi import APIRouter, Depends

from src.api.user.service import UserService

router = APIRouter()

@router.get("/")

async def read_users(user_service : UserService = Depends()):
    users = user_service.get_users()
    return users

@router.post("/")
async def create_user(user_create_input: UserCreateInput, user_service :UserService = Depends()):
    user = user_service.create_user(user_create_input)
    return user

Now, if you go to the terminal and send the following post request

curl -X 'POST' \
  'http://localhost:8000/users/' \
  -H 'accept: application/json' \
  -H 'Content-Type: application/json' \
  -d '{
  "name": "Mohammad Faisal",
  "email": "mohammadfaisal1011@gmail.com",
  "password": "faisal"
}'

You will receive the following response with the success message!

Get the details of a user.

Now, let’s create another function to get a user's details.

For this, we don’t need any request schema, as we will be using the user id to get the details.

@router.get("/{user_id}")
async def read_user(user_id: int, user_service : UserService = Depends()):
    user = user_service.get_user(user_id)
    return user

Then, remember to add the function to the service.

def get_by_id(self, user_id: int):
    statement = select(User).where(User.id == user_id)
    user = self.session.exec(statement).one()
    return user

But we need to handle the case when the user is not found.

from sqlmodel import select, Session, SQLModel

class UserService:
    def get_by_id(self, user_id: int):
        statement = select(User).where(User.id == user_id)
        user = self.session.exec(statement).one_or_none()
        if user is None:
            raise Exception("User not found")
        return user

Now, this check will raise an exception if the user is not found.

But we need to handle this exception in the router.

from fastapi import APIRouter, Depends, HTTPException

from src.api.user.service import UserService

router = APIRouter()

@router.get("/")

async def read_users(user_service : UserService = Depends()):
    users = user_service.get_users()
    return users

@router.post("/")
async def create_user(user_create_input: UserCreateInput, user_service :UserService = Depends()):
    user = user_service.create_user(user_create_input)
    return user

@router.get("/{user_id}")
async def read_user(user_id: int, user_service : UserService = Depends()):
    try:
        user = user_service.get_by_id(user_id)
        return user
    except Exception as e:
        raise HTTPException(status_code=404, detail="User not found")

Now, you will receive a 404 error if the user is not found.

Update The User

Now, let’s create another function to update the user.

For this, we need to create a request schema.

class UserUpdateInput(BaseModel):
    name: str
    email: str
    password: str

Then update the service.py file to include the update_user function.

def update_user(self, user_id, user_update_input):
    statement = select(User).where(User.id == user_id)
    user = self.session.exec(statement).one()
    for key, value in user_update_input.dict().items():
        setattr(user, key, value)
    self.session.add(user)
    self.session.commit()
    self.session.refresh(user)
    return user

Finally, add the route.

@router.put("/{user_id}")
async def update_user(user_id: int, user_update_input: UserUpdateInput,user_service : UserService = Depends()):
    try:
        user = user_service.get_by_id(user_id)
        return user
    except Exception as e:
        raise HTTPException(status_code=404, detail="User not found")

Now you can see that we have added the check for the user. If the user is not found, we will raise a 404 error.

Now, this is a duplicate code. We can move this check to the service.

We can use the concept of dependencies to create a dependency that will check if the user exists.

Create a new file named dependencies.py in the src/api/user directory with the following content:

from fastapi import HTTPException, Depends

from src.api.user.service import UserService

def get_user(user_id: int, user_service: UserService = Depends()):
    user = user_service.get_by_id(user_id)
    if user is None:
        raise HTTPException(status_code=404, detail="User not found")
    return user

Then, update the router.py file to include the get_user dependency.

@router.get("/{user_id}")
async def read_user(
    user_id: int,
    user_service: UserService = Depends(),
    user: Mapping = Depends(get_user),
):
    user = user_service.get_by_id(user_id)
    return user


@router.put("/{user_id}")
async def update_user(
    user_id: int,
    user_update_input: UserUpdateInput,
    user_service: UserService = Depends(),
    user: Mapping = Depends(),
):
    user = user_service.update(user_id, user_update_input)
    return user

Now you can see that we are using the get_user dependency to check if the user exists. And you can do the same. Now, you don't have any duplications.

Delete the user.

Now, let’s create another function to delete the user.

Then, update the service.py file to include the delete_user function.

def delete_user(self, user_id):
    statement = select(User).where(User.id == user_id)
    user = self.session.exec(statement).one()
    self.session.delete(user)
    self.session.commit()
    return user

Finlly, add the route.

@router.delete("/{user_id}")
async def delete_user(user_id: int, user_service : UserService =Depends()):
    user = user_service.delete(user_id)
    return user

That’s about it for today! Now you have a boilerplate project that you can just clone and start adding endpoints.

Hope you learned something new today!

Have something to say? Get in touch with me via LinkedIn or Personal Website

NodeJS Security Best Practices

Mohammad Faisal — Mon, 19 Feb 2024 11:46:18 +0000

To read more articles like this, visit my blog

Today, we will see how we can improve the security of a NodeJS application. There are lots of aspects to security in the NodeJS application. We will get familiar with many of the concepts and see how to prevent unwanted attacks on our application.

Nothing is bulletproof, but being safe doesn't hurt.

The most popular one!

We will first use an excellent npm package named helmet.

It sets up various HTTP headers to prevent attacks like Cross-Site-Scripting(XSS), clickjacking, etc.

yarn add helmet

Then use it inside your index.ts

import helmet from "helmet";

app.use(helmet());

That's it! You don't need to do anything else!

You should also take a look at helmet-csp

Prevent DOS attack

DOS means Denial of Service. If an attacker tries to swamp your server with requests, our real users can feel the pain of slow response time.

To prevent this, we can use an excellent package named toobusy-js
This will monitor the event loop, and we can define a lag parameter that will monitor the lag of the event loop and indicate if our event loop is too busy to serve requests right now.

yarn add toobusy-js

Then add a new middleware to indicate that the server is too busy.

import too busy from 'toobusy-js';

app.use(function (req, res, next) {
  if (toobusy()) {
    res.send(503, 'Server too busy!');
  } else {
    next();
  }
});

Rate Limiting

Rate limiting helps your application from brute-force attacks. This allows to prevent the server from being throttled.

Unauthorized users can perform any number of requests with malicious intent, and you can control that with rate-limiting.
For example, you can allow a user to make five requests per 15 minutes for creating an account.
Or you can allow unsubscribed users to make requests at a certain rate limit. something like 100requests/day

There is a nice package named express-rate-limit. First, install it

yarn add express-rate-limit

Then create a rate-limiting configuration for it.

import rateLimit from "express-rate-limit";

export default rateLimiter = rateLimit({
  windowMs: 24 * 60 * 60 * 1000, // 24 hrs in milliseconds
  max: 100, // maximum number of request inside a window
  message: "You have exceeded the 100 requests in 24 hrs limit!", // the message when they exceed limit
  standardHeaders: true, // Return rate limit info in the `RateLimit-*` headers
  legacyHeaders: false, // Disable the `X-RateLimit-*` headers
});

const app = express();

app.use(rateLimiter);

This will allow you to add a rate limit for all of your routes. You can also just add rate-limiting for specific routes.

But if you are behind a proxy. This is the case for most cases when you use any cloud provider like Heroku aws etc., then the IP of the request is basically the IP of the proxy, which makes it look like that request is coming from a single source, and the server gets clogged up pretty quick.

To resolve this issue, you can find out the numberOfProxies between you and the server and set that count right after you create the express application.

const numberOfProxies = 1;
const app = express();

app.set("trust proxy", numberOfProxies);

To learn more about trust proxy refer to the documentation

Configure Cors

CORS will keep your application safe from malicious attacks from unknown sources
It's really easy to configure in nodejs.

npm i cors

then use it inside the index.ts file

import cors from "cors";

let corsOptions = {
  origin: "http://example.com",
  optionsSuccessStatus: 200, // some legacy browsers (IE11, various SmartTVs) choke on 204
};
app.use(cors());

Prevent XSS attacks

XSS attack means cross-site scripting attacks. It injects malicious scripts into your application.

An attacker can use XSS to send a malicious script to an unsuspecting user. The end user's browser has no way to know that the script should not be trusted and will execute the script. Because it thinks the script came from a trusted source, the malicious script can access any cookies, session tokens, or other sensitive information retained by the browser and used with that site.

You can protect your application by using xss-clean

yarn add xss-clean

Then use it inside the index.ts file

import xss from "xss-clean";

app.use(xss());

Prevent SQL Query injection attacks

If you use user input without checking them, a user can pass unexpected data and fundamentally change your SQL queries.

For example, if your code looks like this.

UPDATE users
    SET first_name="' + req.body.first_name +  '" WHERE id=1001;

If a user types his first name as

Bobby", last_name="Egg"; --

Then the query becomes

UPDATE users
    SET first_name="Bobby", last_name="Egg"; --" WHERE id=1001;

So all your users are named Bobby Egg which is not cool.

If you use Sequalize, TypeORM or for MongoDB, we have Mongoose these types of ORM tools, then you are safe by default because these help us against the SQL query injection attacks by default.

If you don't want to use ORM then there are some other packages as well!
For PostgreSQL we have node-postgres

Limit the size of the body of the request

Using body-parser you can set the limit on the size of the payload

npm i body-parser

By default, body-parser is configured to allow 100kb payloads size. You can set the limit like the following.

import bodyParser from "body-parser";
app.use(bodyParser.json({ limit: "50kb" }));
app.use(bodyParser.urlencoded({ extended: true }));

Use linter

A linter can force you to follow these best practices by default. You can use eslint-plugin-security for that.

yarn add -D eslint-plugin-security

And inside your .eslintrc file

"extends": ["plugin:@typescript-eslint/recommended", "plugin:security/recommended"],

Enforce HTTPS

It would be best if you always use HTTPS over HTTP when possible.

yarn add hsts

Then use it inside your index.ts

import hsts from "hsts";

app.use(
  hsts({
    maxAge: 15552000, // 180 days in seconds
  })
);
// Strict-Transport-Security: max-age: 15552000; includeSubDomains

Use CSRF Protection middleware

To learn more about CSRF. Go here
Consider using csurf

import csrf from "csurf";
var csrfProtection = csrf({ cookie: true });

app.get("/form", csrfProtection, function (req, res) {
  // generate and pass the csrfToken to the view
  res.render("send", { csrfToken: req.csrfToken() });
});

This is not required for an application that doesn't handle any form of data.

Validate Incoming Requests

You should consider validating the incoming requests to your application to check if they contain any malicious data. There are many ways to do this, but the most popular way is to use a schema validation library like Joi or class-validator.

You can refer to the following article for more information.

https://www.mohammadfaisal.dev/blog/request-validation-nodejs-express

Validating Output

You want to secure the incoming data instead you should try to validate any data that you give output.
For example, if you render to the frontend using NodeJS, then this becomes an issue.

Let's say our user generates the following data for showing in the browser later.

<script>alert('I am not sanitized!');</script>

This is harmless, but if you don't sanitize it before giving it back to the javascript frontend, then it can cause trouble.

To avoid this, we have several options.

html-escape // if you need to render html output
xml-escape // if you need to give xml output
shell-escape // for shell commands

These libraries can help you to sanitize the output as well.

Using Hash for Sensitive Information

Storing passwords in plain text was never a good idea. You should instead create a has of your password.

A Hash is a fixed-sized string that we can generate from any string, and it's not decryptable by design. You can only produce it again if the same input text is given again.

So while storing passwords, you will create a hash of your password and store it in the database. This way, even if our database is exposed, the passwords will not be exposed.

Using the Salt with Hash

There is a problem with the Hashing mechanism because two identical texts will always generate the same hash. As most people use very common passwords, this technique is vulnerable to exploitation.

To avoid this issue, we use a technique called salt. It's basically a random string that is appended to the given input. So even if you give two identical inputs, the outputs will not be the same.

Hide errors from end-users

It would be best if you did not expose errors to your users. In order to do that, you will need to handle production environment error handling seriously. Following is a good guide on where you can start.

https://www.mohammadfaisal.dev/blog/error-handling-nodejs-express

Load secrets securely

Be extra careful about loading secrets into your application. You must not include secrets as plain strings in the application. Using some kind of environment file is necessary to achieve this. You can refer to the following example for that.

https://www.mohammadfaisal.dev/blog/nodejs-environment-handling

Compression

Compression is a technique that can reduce the size of the static file and JSON response.
In nodejs, that we can do with a nice middleware package named compression

First, install it

yarn add compression

Then add it inside your index.ts

import compression from "compression";
app.use(compression());

And that's it! There are other options that you can use. Refer to the documentation for that.

Some more resources:

https://cheatsheetseries.owasp.org/cheatsheets/Nodejs_Security_Cheat_Sheet.html
https://medium.com/@nodepractices/were-under-attack-23-node-js-security-best-practices-e33c146cb87d

Github Repo

https://github.com/Mohammad-Faisal/nodejs-security-best-practices

Have something to say? Get in touch with me via LinkedIn or Personal Website

Best Software Documentation Tools

Mohammad Faisal — Mon, 05 Feb 2024 15:18:52 +0000

To read more articles like this, visit my blog

In software development, documentation is like the broccoli of the meal — not always the most exciting part, but it’s essential for a healthy project.

Like how broccoli can be cooked in many different ways to make it more palatable, many tools are available to make the documentation less of a chore and more of a delight.

This article’ll explore the benefits of using these tools and how they can turn your documentation from a bland, forgettable side dish into a mouth-watering main course.

So grab your forks, and let’s dig in!

1. Docusaurus

This is developed by Meta. You can create really nice-looking documentation websites super fast.

It can also be used to build static websites. You can take a look here for some inspiration.

To create a website, Just run the following command.

npx create-docusaurus@latest my-website classic

It supports markdown files. You can write both blogs and docs with this tool.

You can also customize the look and feel using the configuration options of Docusaurus

This is based on React. So if you are already familiar with ReactJS then, it will help a lot!

Documentation

Build optimized websites quickly, focus on your content | Docusaurus

2. Docsify

This is a nice-looking, feature-rich documentation generator tool. It’s built on top of VueJS. So having prior knowledge of VueJS will come in handy.

To get started, you will need to run

npm i docsify-cli -g

Then initialize the project

docsify init ./docs

You can edit the pages and see the changes

docsify serve docs

And the documentation will be served at http://localhost:3000

You can customize everything, From the loading bars to the side nav bars, cover pages… Everything!

Also, it’s really easy to deploy using GitHub, GitLab, Vercel, etc.
docsify

Demo

GitHub - docsifyjs/awesome-docsify: 💖 A curated list of awesome things related to docsify

3. Gitbook

This is another general-purpose tool that can be used to write books or documentation. It’s also super easy to distribute as well.

Benefits

GitBook allows multiple users to collaborate on the same document
GitBook uses Git, a popular version control system.
GitBook integrates with GitHub, Slack, and Trello.
GitBook allows users to publish their documentation in multiple formats, including PDF, EPUB, and HTML.

You can set custom domains; the documentation is also auto-indexable. It supports internationalization as well. Overall it’s a fantastic tool.

Documentation:

GitBook
GitBookapp.gitbook.com

Let’s have a look at some of the API documentation tools now!

4. Swagger

Swagger is a widely used tool for documenting and testing APIs.

It has an online editor. You can easily play around with it and generate easy-to-use documentation.

Benefits

It provides a standardized way to document APIs
Interactive documentation
Swagger can auto-generate API documentation based on the code.
Collaboration is a breeze here.

You should definitely make yourself familiar with this platform.

Documentation:

API Documentation & Design Tools for Teams | Swagger

5. Redoc

Redocly is another OpenAPI documentation generator tool. It’s like Swagger but with a better interface (In my opinion).

You can provide an open API file, and it will auto-generate the documentation and provide you with some boilerplate code as well!

Demo

ReDoc Interactive Demo

Benefits:

It has a VS code extension
Tons of customization.
Direct integration with OpenAPI

Disadvantages

It’s not completely free. It has paid versions as well.

Documentation

Redocly Workflows

6. Readme

This is another API documentation generator tool for enterprise applications.

You can sync your API using this tool, and it supports OpenAPI 3.0, OpenAPI 3.1, Swagger 2, and Postman API formats.

You can import your open API file, and it will produce this beautiful-looking documentation.

In fact, it will add an example code snipped automatically! Also, it provides you a nice dashboard with statistics.

But it has both free and paid plans, So choose accordingly!

Documentation

ReadMe

Conclusion

Documenting code may never be as fun as binge-watching your favorite TV show, but with the right tools, it can be almost as enjoyable as ordering a pizza.

So don’t let documentation be the Brussels sprouts of your software project — spice it up with some humor, creativity, and a good documentation tool.

Who knows, you may even discover that you have a hidden talent for technical writing. And if not, at least you’ll have some well-documented code and a satisfied appetite.

Happy coding, and bon appétit! I hope you learned something new today!

Have something to say? Get in touch with me via LinkedIn or Personal Website