DEV Community: Tal Rofe

Utilize TOR network to get free proxy for any use case

Tal Rofe — Sat, 28 Sep 2024 14:26:05 +0000

What is TOR?

Tor Browser uses the Tor network to protect your privacy and anonymity. Using the Tor network has two main properties:

Your internet service provider, and anyone watching your connection locally, will not be able to track your internet activity, including the names and addresses of the websites you visit.

The operators of the websites and services that you use, and anyone watching them, will see a connection coming from the Tor network instead of your real Internet (IP) address, and will not know who you are unless you explicitly identify yourself.

(Source: https://tb-manual.torproject.org/about/)

One purpose for using proxy(ies) would be scraping websites & APIs without being rate limited or blocked. If you keep scraping using your own router IP facing the internet, you might be blocked very soon.

To skip it, you can utilize TOR network, without paying nothing - completely free.

Prerequisites

Docker
NodeJS (or any other programming language...)

Setup

To start using TOR network, you need to use this Docker image: https://hub.docker.com/r/dperson/torproxy, which allows you use TOR network in a Docker container. To simply use it, you can use this docker-compose.yaml file to start one TOR container:

services:
    tor_proxy:
        container_name: tor_proxy
        image: dperson/torproxy
        ports:
            - 8118:8118
            - 9051:9051
        environment:
            - PASSWORD=password

Port 8118 is required for network and port 9051 is required for renew the proxy IP (if blocked for example - you can change its IP).

Then, you can run docker compose up -d to have 1 proxy.

Use TOR network for proxies use case

You can scale up your TOR network easily with the Docker Compose file. You can actually create as many as you want (well, bounded to this list: https://www.dan.me.uk/torlist/?exit), using this Docker Compose file (example for managing 5 proxies):

services:
    proxy1:
        container_name: proxy1
        image: dperson/torproxy
        ports:
            - 8118:8118
            - 9051:9051
        environment:
            - PASSWORD=password
    proxy2:
        container_name: proxy2
        image: dperson/torproxy
        ports:
            - 8119:8118
            - 9052:9051
        environment:
            - PASSWORD=password
    proxy3:
        container_name: proxy3
        image: dperson/torproxy
        ports:
            - 8120:8118
            - 9053:9051
        environment:
            - PASSWORD=password
    proxy4:
        container_name: proxy4
        image: dperson/torproxy
        ports:
            - 8121:8118
            - 9054:9051
        environment:
            - PASSWORD=password
    proxy5:
        container_name: proxy5
        image: dperson/torproxy
        ports:
            - 8122:8118
            - 9055:9051
        environment:
            - PASSWORD=password

It allows you to load balance your egress network across multiple proxies.

Use case - GitHub scraping

GitHub API and GitHub UI is highly rate limited, even for offline requests. However, if you use 5 proxies concurrently, you multiply the rate limit threshold by 5.

For example, if one would like to scrape a GitHub repository's stargazers, he would need to scrape 40K stargazers (GitHub limits 40K results - even if the repository has 50K stargazers). This might limit your code without proxies because you will be blocked very quick using only your router IP. What with usage of TOR network for proxies, you can achieve this data very quick and easily:

Configure constants for configuration

Configure yourself your script configuration for proxies number and more:

export const PROXIES_COUNT = 16;

export const PROXY_INITIAL_PORT = 8118;

export const PROXY_RENEW_CIRCUIT_CONNECTION_PORT = 9051;

Use PROXIES_COUNT to declare how much proxies you want to use. Other constants should be kept as is (unless you change ports - why?..)

Create your proxies HTTP agents:

First install package https-proxy-agent to create a proxy agent:
pnpm add https-proxy-agent

Then create the proxies:

import { HttpsProxyAgent } from 'https-proxy-agent';

import { PROXY_INITIAL_PORT, PROXIES_COUNT } from '../constants/proxy';

export const proxyAgents = new Array(PROXIES_COUNT)
    .fill(null)
    .map((_, index) => new HttpsProxyAgent(`http://127.0.0.1:${PROXY_INITIAL_PORT + index}`));

Create HTTP client to send requests:

First install package got to create HTTP client:
pnpm add got

Then create the client:

import got, { type Headers } from 'got';

import { proxyAgents } from './proxy-agents';

const SECOND = 1000;

const TIMEOUT = 10 * SECOND;

const RETRIES = 3;

export const getBaseHttp = (proxyIndex: number, headers?: Headers) => {
    const newHeaders = { ...headers };

    return got.extend({
        timeout: { request: TIMEOUT },
        retry: {
            limit: RETRIES,
            methods: ['HEAD', 'OPTIONS', 'TRACE', 'GET'],
            statusCodes: [408, 413, 500, 502, 503, 504, 521, 522, 524],
            // * Keep others (backoff limit, ...) as default
        },
        throwHttpErrors: false,
        agent: { https: proxyAgents[proxyIndex]! },
        headers: newHeaders,
    });
};

Check all the proxies are ready to use

First you run docker compose up -d with your Docker Compose file, but it might take a minute or more until all the proxies are ready to use. Thus, we need to health check all the containers, until they are ready.

First, install package docker-compose:
pnpm add docker-compose and then declare this function to await the containers to be healthy:

import { v2 as dockerCompose } from 'docker-compose';

export const awaitProxiesContainerHealth = async () => {
    console.log('Awaiting docker proxies container to be healthy');

    try {
        const psResult = await dockerCompose.ps({ cwd: process.cwd() });
        const areAllReady = psResult.exitCode === 0 && psResult.data.services.every((service) => service.state.includes('healthy'));

        if (areAllReady) {
            return;
        }
    } catch (error) {
        console.warn('Failed to check health of containers', { errorCode: 'AWAIT_DOCKER_PROXIES_CONTAINER_HEALTH', error });
    }

    await new Promise((resolve) => setTimeout(resolve, 5000));

    await awaitProxiesContainerHealth();

    return;
};

Then, you can await, in your main script, for healthness:

const proxiesContainerHealthTimeout = setTimeout(() => {
    LoggerService.error('Failed to check health of proxies containers', { errorCode: ErrorCode.AWAIT_DOCKER_PROXIES_CONTAINER_HEALTH });

    process.exit(1);
}, PROXIES_HEALTH_TIMEOUT);

await awaitProxiesContainerHealth();

clearTimeout(proxiesContainerHealthTimeout);

You should configure yourself PROXIES_HEALTH_TIMEOUT constant, possible 5 minutes (1000 * 60 * 5 milliseconds).

Counting repository total stargazers count:

You can use this code to get the count:

import { TorControl } from 'tor-ctrl';

import { getBaseHttp } from '../utils/http';

export const getRepositoryStargazersCount = async () => {
    console.log(
        'Querying stargazers count of repository',
        { githubOwner: process.env.GITHUB_OWNER, githubRepository: process.env.GITHUB_REPOSITORY },
        true,
    );

    const baseHttp = getBaseHttp(0);

    let response = await baseHttp.get(`https://api.github.com/repos/${process.env.GITHUB_OWNER}/${process.env.GITHUB_REPOSITORY}`);

    while (response.statusCode === 403) {
        console.warn('Failed to get stargazers count because of rate limit', {
            errorCode: 'GITHUB_API_RATE_LIMIT',
        });

        const torControl = new TorControl({
            host: 'localhost',
            port: 9051,
            password: 'password',
        });

        await torControl.connect();
        await torControl.signalNewNym();
        await torControl.disconnect();

        response = await baseHttp.get(`https://api.github.com/repos/${process.env.GITHUB_OWNER}/${process.env.GITHUB_REPOSITORY}`);
    }

    if (!response.ok) {
        console.warn('Failed to get stargazers count because of an error', {
            errorCode: 'GITHUB_API_STARGAZERS_COUNT',
            error: response.body,
        });

        throw response.errored;
    }

    console.info('Successfully fetched stargazers count', { stargazersCount: response.body.stargazers_count });

    return response.body.stargazers_count;
};

Note that while statement. We run the block code as long as we are rate limited, and if we are - we signal the TOR container with the NewNym signal to replace the IP for other one (so bypass the IP blocking).

Then, in your main script use this function to get the count:

let repositoryStargazersCount: number;

try {
    repositoryStargazersCount = await getRepositoryStargazersCount();
} catch (error) {
    LoggerService.error('Failed to fetch stargazers count', { errorCode: ErrorCode.REPOSITORY_STARGAZERS_COUNT, error });

    return;
}

if (repositoryStargazersCount === 0) {
    return;
}

Get all stargazers details:

Create these 2 functions to scrape the stargazers details:

import { TorControl } from 'tor-ctrl';

import { MAX_STARGAZERS_PAGES, STARGAZERS_IN_PAGE } from '../constants/stargazers-page';
import { proxyAgents } from '../data/proxy-agents';
import { RepositoryStargazersResponseSchema } from '../schemas/repository-stargazers';
import { getBaseHttp } from '../utils/http';
import { PROXY_RENEW_CIRCUIT_CONNECTION_PORT } from '../constants/proxy';

const getPageStargazers = async (proxyIndex: number, page: number) => {
    const searchParams = new URLSearchParams({ per_page: STARGAZERS_IN_PAGE.toString(), page: page.toString() });

    const baseHttp = getBaseHttp(proxyIndex, {
        Accept: 'application/vnd.github.star+json',
    }).extend({ searchParams });

    let pageResponse = await baseHttp.get(`https://api.github.com/repos/${process.env.GITHUB_OWNER}/${process.env.GITHUB_REPOSITORY}/stargazers`);

    while (pageResponse.statusCode === 403) {
        console.warn('Failed to get stargazers for page because of rate limit', {
            errorCode: 'GITHUB_API_RATE_LIMIT',
            page,
            proxyIndex,
        });

        const torControl = new TorControl({
            host: 'localhost',
            port: PROXY_RENEW_CIRCUIT_CONNECTION_PORT + proxyIndex,
            password: 'password',
        });

        await torControl.connect();
        await torControl.signalNewNym();
        await torControl.disconnect();

        pageResponse = await baseHttp.get(`https://api.github.com/repos/${process.env.GITHUB_OWNER}/${process.env.GITHUB_REPOSITORY}/stargazers`);
    }

    if (!pageResponse.ok) {
        console.warn('Failed to get stargazers for page because of an error', {
            error: pageResponse.body,
            errorCode: 'GITHUB_API_STARGAZERS',
            page,
            proxyIndex,
        });

        throw pageResponse.errored;
    }

    const validatedPageResponse = await RepositoryStargazersResponseSchema.safeParseAsync(JSON.parse(pageResponse.body));

    if (!validatedPageResponse.success) {
        console.warn('Invalid page response', {
            errorCode: 'GITHUB_API_STARGAZERS_INVALID',
            invalidError: validatedPageResponse.error.message,
            proxyIndex,
            page,
        });

        throw validatedPageResponse.error;
    }

    return validatedPageResponse.data;
};

export const getRepositoryStargazers = async (count: number) => {
    // * GitHub restrict 400 pages
    const numberOfPagesToScrape = Math.min(Math.ceil(count / STARGAZERS_IN_PAGE), MAX_STARGAZERS_PAGES);

    const repositoryStargazers: z.infer<typeof RepositoryStargazersResponseSchema>[] = [];

    for (let i = 1; i <= numberOfPagesToScrape; i++) {
        const promises: ReturnType<typeof getPageStargazers>[] = [];

        for (let j = 0; j < 10 * proxyAgents.length && i <= numberOfPagesToScrape; i++, j++) {
            promises.push(getPageStargazers(j % proxyAgents.length, i));
        }

        const results = await Promise.allSettled(promises);

        const validStargazers = results
            .filter((result): result is PromiseFulfilledResult<z.infer<typeof RepositoryStargazersResponseSchema>> => result.status === 'fulfilled')
            .map((result) => result.value);

        repositoryStargazers.push(...validStargazers);
    }

    return repositoryStargazers.flat();
};

(Not all imports are explicitly given, it's up to you...)

Then use it in your main script:

const repositoryStargazers = await getRepositoryStargazers(repositoryStargazersCount);

And now you successfully scrape all (up to 40K..) stargazers of a given repository. Have fun.

React: Create YouTube-like loader when routing without screen flickering

Tal Rofe — Thu, 27 Apr 2023 20:27:47 +0000

On "localhost", everything is instant. On production, your application might be slow. On simple SPA, all pages are already downloaded on client-end browser once user enters your website. This behavior may lead to slow loading of your website (cold boot), if it is very big one. In these scenarios, "lazy loading" technique is used, to prevent a user from downloading all pages at once, although some might be irrelevant for the user. But- this technique leads to screen flickering - when a user navigates, you need to wait the browser to load your lazy-loaded route, while you see a blank screen.

What if lazy-loading a route might take 2-3 seconds? You want to show a loading progress in the top of the browser window (YouTube-like), while showing the current loaded page. These scenarios occur when you want to preload some data before user reaches the page, for example. Instead of showing blank screen while we preload some data, we will show progress bar with current loaded page, and then navigate.

Solution

We still use the lazy loading technique, but with some adjustments. When a user navigates, before leaving the current loaded route, we will show the progress bar first. Then, we start downloading the route-to-be-loaded (instead of showing blank screen until loaded) and (optional) preload some data required by the to-be-navigated page from our server. Once the download completes (and data is fetched) successfully, we stop the progress bar and actually navigate.
We can "delay" the navigation by using the loader feature of React Router: https://reactrouter.com/en/main/route/loader. This allows us the do anything we want, and only when completing the action, actually navigate.

Begin with wrapping your app with RouterProvider component:

import React, { useMemo } from 'react';
import { RouterProvider, createBrowserRouter } from 'react-router-dom';

import RouterBuilder from './App.router';

interface IProps {
    readonly isAuthenticated: boolean | null;
}

const AppView: React.FC<IProps> = (props: React.PropsWithChildren<IProps>) => {
    const routes = useMemo(() => RouterBuilder(props.isAuthenticated), [props.isAuthenticated]);

    return <RouterProvider router={createBrowserRouter(routes)} />;
};

AppView.displayName = 'AppView';
AppView.defaultProps = {};

export default React.memo(AppView);

In this snippet, we want to build the routes dependent on the authentication status of the user, to use "route guards" mechanism. We are using RouterProvider to enable the loader feature of react-router-dom package.

In the RouteBuilder we return the available routes for the user, dependent on his authentication status. We have 3 statuses:

General routes - for unauthorized & authorized user.
Unauthorized user routes
Authorized user routes

import React from 'react';
import type { RouteObject } from 'react-router-dom';

import AppLayout from './App.layout';
import BackendService from './services/backend';
import { endProgress, startProgress } from './services/progress-bar';

const Auth = React.lazy(() => import('./pages/Auth'));
const ExternalAuthRedirect = React.lazy(() => import('./pages/ExternalAuthRedirect'));
const ComplianceCenter = React.lazy(() => import('./pages/ComplianceCenter'));

const RouterBuilder = (isAuthenticated: boolean | null) => {
    const unAuthorizedRoutes: RouteObject[] = [
        {
            path: '',
            element: <Auth />,
            loader: async () => {
                startProgress();

                await import('./pages/Auth');

                endProgress();

                return null;
            },
        },
        {
            path: 'auth',
            element: <Auth />,
            loader: async () => {
                startProgress();

                await import('./pages/Auth');

                endProgress();

                return null;
            },
        },
    ];

    const authorizedRoutes: RouteObject[] = [
        {
            path: '',
            element: <ComplianceCenter />,
            loader: async () => {
                startProgress();

                await Promise.all([
                    BackendService.preload('/user/compliances'),
                    import('./pages/ComplianceCenter'),
                ]);

                endProgress();

                return null;
            },
        },
    ];

    const generalRoutes: RouteObject[] = [
        {
            path: 'external-auth-redirect',
            element: <ExternalAuthRedirect />,
        },
    ];

    let routesChildren = generalRoutes;

    if (isAuthenticated) {
        routesChildren = [...authorizedRoutes, ...generalRoutes];
    }

    if (isAuthenticated === false) {
        routesChildren = [...unAuthorizedRoutes, ...generalRoutes];
    }

    const routes: RouteObject[] = [
        {
            element: <AppLayout />,
            children: routesChildren,
        },
    ];

    return routes;
};

export default RouterBuilder;

For the <ComplianceCenter /> we also preload some data from the server (refrence: https://swr.vercel.app/docs/prefetching).

The AppLayout should render your actual website, by using the Outlet component:

import React, { Suspense } from 'react';
import { Outlet } from 'react-router-dom';

import EDNotification from '@/ui/EDNotification';

interface IProps {}

const AppLayout: React.FC<IProps> = () => {
    return (
        <Suspense fallback={null}>
            <Outlet />

            <div id="backdrop-root" />
            <div id="overlay-root" />

            <EDNotification />
        </Suspense>
    );
};

AppLayout.displayName = 'AppLayout';
AppLayout.defaultProps = {};

export default React.memo(AppLayout);

We are using fallback with null value, as we don't need a fallback for routes navigation (in this technique we still show the current loaded page, and only then navigate). In that case, at cold boot, a blank screen will show. You could set here some loader, but remember it will be relevant only for cold boot, when website is first loaded.

Application demo: https://app.exlint.io
Code repository: https://github.com/Exlint/dashboard (apps/frontend).

Use "cz-vinyl" package to commit more readable messages with one command

Tal Rofe — Thu, 20 Apr 2023 12:30:58 +0000

Commit messages are very important to understand what work is done on the specific commit, and also for much more important tools.

You can use your commit messages to automate JIRA flows (or other project management framework..), or to automatically upgrade your package version according to the commit message.
For example, you could tag your JIRA ticket ID in your commit message to help others to understand what is the scope of your commit.

You can follow this guide: https://semver.org/ and infer from your commit messages, what should be your next package version. For example, if someone on your team commits some breaking change code, you'll probably want to upgrade your "Major" version (1.0.0 --> 2.0.0).

How

To start formatting your commit messages, you need to install 2 packages: npm i -D commitizen cz-vinyl.
"Commitizen" is a tool which allows to get prompted in the command-line-interface to fill some data regarding your commit message, and then get the formatted commit message. "cz-vinyl" is an adapter for "Commitizen" which will format your message according to some guidelines.

In your project, create .cz.json file with the following content:

{
    "path": "cz-vinyl"
}

You can already start using the adapter by running npx cz. But we are gonna do it in more elegant way.

We will allow the developers to use the adapter by running git cmt (which is more intuitive command to make a commit message). To do so, create .gitconfig file with the following content:

[alias]
    cmt = "!npx cz"

This will create an alias for git. Once a developer runs git cmt, it will be an alias for running npx cz. This won't work out of the box for security aspects, so a developer must run git config --local include.path ../.gitconfig. But this makes it cumbersome of course. So we can make it automatically work by adding a "package.json" hook script, which will be running once a developer installs your project: "prepare": "git config --local include.path ../.gitconfig".

You can also configure the adapter, according to your needs with a configuration file, czvinyl.config.cjs (or other name according to the documentation):

const czvinylConfig = {
    headerFormat: '{type}: {emoji} {subject}',
    skipTicketId: true,
};

module.exports = czvinylConfig;

As a result, you can get a commit message header like: chore: [ESS-2234] 🤖 remove Depcheck.

You can also integrate a GitHub workflow to automatically upgrade your package version according to your commit message:

name: Semantic Release

on:
    push:
        branches: [main]

jobs:
    release:
        name: Release version
        permissions:
            contents: write

        runs-on: ubuntu-latest
        steps:
            - name: Setup Node environment
              uses: Exlint/node-environment@v1.2.6
              with:
                  should-cache: false
                  node-version: 18.15.0
                  package-manager: pnpm
                  package-manager-version: 8.3.1

            - name: Setup Node
              uses: actions/setup-node@v3
              with:
                  node-version: 14

            - name: Semantic Release
              uses: cycjimmy/semantic-release-action@v3.4.2
              with:
                  extra_plugins: |
                      @semantic-release/changelog
                      @semantic-release/commit-analyzer
                      @semantic-release/git
                      @semantic-release/npm
              env:
                  GITHUB_TOKEN: ${{ secrets.GH_TOKEN }}

And a release.config.cjs file:

const config = {
    branches: ['main'],
    repositoryUrl: 'git+https://github.com/Exlint/cz-vinyl',
    plugins: [
        [
            '@semantic-release/commit-analyzer',
            {
                preset: 'angular',
                releaseRules: [
                    { type: 'feat', release: 'minor' },
                    { type: 'fix', release: 'patch' },
                    { type: 'perf', release: 'patch' },
                    { type: 'breaking', release: 'major' },
                ],
            },
        ],
        '@semantic-release/release-notes-generator',
        '@semantic-release/changelog',
        '@semantic-release/npm',
        '@semantic-release/git',
        '@semantic-release/github',
    ],
};

module.exports = config;

Use Inflint to follow files and folders convention

Tal Rofe — Fri, 22 Apr 2022 19:22:52 +0000

INTRO

In the last months I worked with some juniors on the same project. Usually in order to overcome some bad practices and bad code, I add some linters: ESLint, Stylint, commitlint and also prettier.

It usually helps a lot and I avoid redundant code reviews for dummy things.

However, there was 1 thing I couldn't avoid. Files and folders names.
Although I had some clear convention with my file structure, other people kept on creating files and folders that did not follow the repository convention.

First thing I thought of was applying some file structure linter.
But I couldn't find any proper linter for this.
I found one which could not be applied with lint-staged and has very low maintenance and features. So.. I created my own package!

INFLINT

WHO??

It is a file system linter. This tool will scan your file structure in the repository, and validates, according to provided configuration, that the files and folders match the provided convention.

WHERE??

You can check it out here:
https://www.npmjs.com/package/inflint

Or look at the source code here:
https://github.com/Vinyl-Depository/Inflint

demo

Not only you get a lot of configurations options with the main features: "Rules" and "Aliases" but this tool can also be applied together with lint-staged:

"**/*": "inflint"

Compiling an Electron Application to pre-defined OS

Tal Rofe — Sat, 20 Nov 2021 22:25:02 +0000

Intro

Electron.JS, in its main purpose - is for creating desktop applications using Javascript. Most popular way of doing so is using React & Node.js.

I will introduce a code architecture for building scalable Electron application - and on the fly - compile the code on chosen OS platform.

I am not going to focus on implementing it with an Electron application because there is no need for this tutorial, but you should know that the best application of this tutorial resides in Electron applications.

We are going to code the application to both Darwin and Windows platforms in one workspace - but as we'll see, in compilation we will be compiling only one (chosen by us) platform code. Because in most cases, you will need to code a code for 2 (at-least) platforms. But, we of-course don't want any Darwin code for example to exist in Windows application (just an application size side-effect).

Design patterns

When working in a team, design patterns become more and more important. Not only for well-structured code, but also for "easy-to-understand" code and scalability.
Thus, we will be using the following: Provider, Singleton, Factory.

Code

When building such Electron application, splitting the code is important.

This is the code architecture I reckon on implementing:

Very basic one, not that advanced.

Short brief:

You want to catch the IPC events coming from your Renderer process via the Routers. Then, send the event, by reading the "channel" name, to the proper controller. A controller is a function to handle messages from the Renderer process. Then, heavy workload should be coded in the Providers. A provider is the one to implement the underlying OS logic for example. This is the module I am going to focus on, because all the rest modules are irrelevant with this tutorial.

Code preparation

So we are creating a dummy project, non-Electron one by the way. Simply create a folder for the project. Run npm init -y.
We are going to use Typescript, Webpack in this project. So please install the following: npm i -D typescript webpack webpack-cli ts-node ts-loader @types/webpack @types/node.

Next, init a tsconfig.json file by running tsc --init. We want to change it to the following:

{
    "compilerOptions": {
      "outDir": "./dist/",
      "noImplicitAny": true,
      "module": "commonjs",
      "target": "es5",
      "jsx": "react",
      "allowJs": true,
      "moduleResolution": "node",
      "esModuleInterop": true,
      "allowSyntheticDefaultImports": true,
    }
}

Now we want to utilize Webpack in our project, because this is a compilation tool for Javascript. So create the following file webpack.config.ts:

import webpack from 'webpack';

const config: webpack.Configuration = {
    resolve: {
        extensions: [".ts", ".js"],
    },
    module: {
        rules: [
          { test: /\.ts$/, use: 'ts-loader' },
        ],
    },
    entry: './src/main.ts',
    output: {
        filename: 'bundle.js',
        clean: true,
    },
    plugins: [
        new webpack.NormalModuleReplacementPlugin(
            /darwin/,
            function (resource) {
            resource.request = resource.request.replace(
                /darwin/,
                'darwin',
            );
            }
        ),
      ],
    mode: 'production',
};

export default config;

The important thing to node is that we are using the NormalModuleReplacementPlugin plugin. It is a plugin which reads your import statements and replace it with whatever you want.
At the moment, we are simply replacing any import statement with darwin string with the same string. Later we'll change it.

Provider Code

Let's start. In the root folder, create a .src folder, and another one inside src called factories. The last will hold your factories classes. Each should be dedicated to well-defined big task. We create 1, so create a folder called example and create inside 4 files: example.ts (the factory), example-provider.ts (the provider), example.darwin.ts (code dedicated to the Darwin application), example.windows.ts ( code dedicated to the Windows application).

The factory purpose is to returns us a Provider, which either instance of the Windows one or the Darwin one. That's because in the most cases the Darwin code is definitely different from the Windows one. So the factory purpose is to retrieve us the correct one, depending on the platform the code is actually being running.
However, sometimes the underlying platforms may share some code. This is why we are going to define an abstract Provider.

Begin with the factory:

import ExampleProvider from './example-provider';
import UnderlyingProvider from './example.darwin';

export default class ExampleFactory {
    private static _instance: ExampleFactory;
    private _provider: ExampleProvider; 

    private constructor() {
        this._provider = new UnderlyingProvider();
    }

    static get instance() {
        if (this.instance) {
            return this._instance;
        }

        return this._instance = new ExampleFactory();
    }

    public get provider() {
        return this._provider;
    }
}

This is our singleton factory class. This is a class to be used with the same instance wide-application. You may want to allow\disable this feature, but in this tutorial I chose to implement is as a Singleton one.
As you can see, it holds the provider. Currently, I import the provider from Darwin one. But later, we'll see how to change to the Windows one.

Let's take a look in the abstract Provider:

export default abstract class ExampleProvider {
    protected abstract executeCodeImpl(): void;

    public executeCode() {
        console.log('Hello world - I am shared code');

        this.executeCodeImpl();
    }
}

The reason we have this classes is important:
1. To have a shared platforms code. In this example the console.log is shared code which will be executed in both platforms.
2. To FORCE the developers to have same "idea" regarding the code. Think- obviously we want the 2 implementations (Windows and Darwin) to do the same task, but in the platform specific ways.

Using abstract class is great way to accomplish these 2 missions.

Let's take a look at the Windows provider:

import ExampleProvider from './example-provider';

export default class ExampleWindows extends ExampleProvider {
    protected executeCodeImpl() {
        console.log('Hello from Windows..');
    }
}

And the Darwin one:

import ExampleProvider from './example-provider';

export default class ExampleDarwin extends ExampleProvider {
    protected executeCodeImpl() {
        console.log('Hello from Darwin..');
    }
}

That's all. Now, wherever you want to execute the platform specific code anywhere outside of the factories folder, like in some arbitrary file try.ts just code:

import ExampleFactory from './factories/example/example';

ExampleFactory.instance.provider.executeCode();

What about compiling to the correct platform?

That's easy. You want Darwin? go to the webpack.config.ts and make sure, via the NormalModuleReplacementPlugin plugin, you chose the darwin imports. Same for Windows. Just change to replace all darwin imports statements with Windows ones by changing the replaced string to winodws in the example up above I've provided.

npm run build and have fun. You have a bundle file compiled to whatever platform code you want, Without the second platform code.

References:
https://webpack.js.org/plugins/normal-module-replacement-plugin/

Can you answer this Javascript question?

Tal Rofe — Wed, 20 Oct 2021 17:56:27 +0000

Can you tell what would be the following output of these snippets?

const promise = new Promise((resolve, reject) => {                                                                       20/10/2021
  console.log("YAZIF");

  setTimeout(() => resolve(), 0);
});

new Promise((resolve, reject) => {
  console.log("Yazof");

  setTimeout(() => resolve(), 0);
}).then(() => console.log(2));

console.log("CHEDDAR");

promise.then(() => console.log(1));

Verify your answer. Now, can you also answer this?

const promise = new Promise((resolve, reject) => {                                                                       20/10/2021
  console.log("YAZIF");

  resolve();
});

new Promise((resolve, reject) => {
  console.log("Yazof");

  resolve();
}).then(() => console.log(2));

console.log("CHEDDAR");

promise.then(() => console.log(1));

Singleton design pattern use case with Node.js (TypeScript) + Express.js

Tal Rofe — Sat, 21 Aug 2021 18:30:41 +0000

Today I will be introducing and using the Singleton Design Pattern, using Node.js (with TypeScript) and Express library.

First, Why would I need Singelton?

Sometimes you need to make sure that you have one and only one instance of an object. This is where the singleton pattern can be useful. A singleton represents a single instance of an object. Only one can be created, no matter how many times the object is instantiated. If there’s already an instance, the singleton will create a new one.

Let's take a look at some use cases that would be nice to be

It is popular to use a database connection (like MongoDB) within a Node.js application. But where and how you should instantiate that connection?

There are several ways to do it. You could just create a file dedicated for this connection that handles the connection for the database.

It is popular to use logger library such as Winston. Where should you instantiate the logger and define it?

Again - you could create a dedicated file to handle this whole thing.

There are more use cases of course, depends on your application. But we can already see - we have 2 dedicated files to manage. What if it would grow? What if you want to have some logic with each? Then in my opinion the whole thing becomes complicated and makes your code dirty.

An optional solution

Use a central singleton class for these global stuff to manage it in one place, well-organized.

So we are going to create simple server that connects to MongoDB and logs some text to the console & external file. For this I'm gonna be using Typescript, because it makes the creation of the singleton class more easy, and besides that, why not?

For this, let's create a file with arbitrary name: server-global.ts. So we know we will be using MongoDB and logging text. So let's install via npm the 3 packages: mongoose, @types/mongoose, winstion: npm i mongoose winston, npm i -D @types/winston.

So let's first build a simple class ServerGlobal within the file we created:

import mongoose from 'mongoose';
import winston from 'winston';

class ServerGlobal {

}

export default ServerGlobal;

So what makes a class singleton? We should avoid creating more than 1 instance of the class somehow. Making the class constructor to private one would easily solve it - then you would not be able to instantiate the class at all outside the class.

The problem is.. how the singleton instance is being created?
So making the constructor private, does not mean you cannot instantiate the class within the class:

import mongoose from 'mongoose';
import winston from 'winston';

class ServerGlobal {
    private static _instance: ServerGlobal;

    private constructor() { }

    static getInstance() {
        if (this._instance) {
            return this._instance;
        }

        this._instance = new ServerGlobal();
        return this._instance;
    }
}

export default ServerGlobal;

So what happened here?

We manage the singleton instance within the class. Then we provide function, getInstance, to allow using the singleton outside the class. Both are static ones, because as I said - the class constructor is private. It means you cannot create instance of the class. So, we need to allow somehow get an instance. For this we have the static.

We can already use the singleton now. If we create a dummy file, we would get the singleton with the following code:

import ServerGlobal from './server-global';

const instance = ServerGlobal.getInstance()

Now let's manage the MongoDB connection and the winston logger setup. So we want to connect to MongoDB and setup the logger ONLY ONCE - because, why would we want to establish connection or setup the logger twice?
For this, we can utilize the class constructor. As we saw, the constructor would only run once because we only create 1 instance of the class.

So first thing - let's connect to MongoDB using the mongoose package.

import mongoose from 'mongoose';

import winston from 'winston';

class ServerGlobal {
    private static _instance: ServerGlobal;

    private constructor() {
        mongoose.connect(process.env.DB_ENDPOINT, {
            useNewUrlParser: true,
            useUnifiedTopology: true,
            useCreateIndex: true,
            useFindAndModify: false,
        });
    }

    static getInstance() {
        if (this._instance) {
            return this._instance;
        }

        this._instance = new ServerGlobal();
        return this._instance;
    }
}

export default ServerGlobal;

That's all. But we missing one thing. What if the connection is either successfully setup or failed? We want to log it.
For this we would use class property to hold the winston logger object, so we could use the logger in other places in the appliction:

import path from 'path';

import mongoose from 'mongoose';
import winston from 'winston';

class ServerGlobal {
    private readonly _logger: winston.Logger;

    private static _instance: ServerGlobal;

    private constructor() {
        this._logger = winston.createLogger({
            level: 'info',
            format: winston.format.combine(
                winston.format.timestamp(),
                winston.format.json(),
            ),
            transports: [
                new winston.transports.Console(),
                new winston.transports.File({
                    filename: path.join(__dirname, '../logs.log'), 
                    level: 'info',
                }),
            ],
        });

        mongoose.connect(process.env.DB_ENDPOINT, {
            useNewUrlParser: true,
            useUnifiedTopology: true,
            useCreateIndex: true,
            useFindAndModify: false,
        }).then(() => this._logger.info('MongoDB connection established successfully'))
        .catch((e: mongoose.Error) => this._logger.error(`MongoDB connection failed with error: ${e}`));
    }

    static getInstance() {
        if (this._instance) {
            return this._instance;
        }

        this._instance = new ServerGlobal();
        return this._instance;
    }

    public get logger() {
        return this._logger;
    }
}

export default ServerGlobal;

So now it's all setup. The only thing left is to create the singleton right when your server boots.
So assume you have some server.ts file in which you boot the server, and you also want to log the boot and port. Then, the file would look something like this:

import http from 'http';

import app from './app';
import ServerGlobal from './server-global';

const port = process.env.PORT;

app.set('port', port);

const server = http.createServer(app);

// Init global set up
ServerGlobal.getInstance();

ServerGlobal.getInstance().logger.info(`Server is running on port ${process.env.PORT}`);

As you can see, In the last 2 lines, we created the singleton, then logged the port on which the server listen (Note that the first line is actually redundant).

Finally, if you would like to log some actions also in your RestAPI controller, you could do it by simply importing the ServerGlobal and using its logger.