DEV Community: Saurabh Bomble

Basic load balancer with NGINX

Saurabh Bomble — Tue, 16 May 2023 06:20:00 +0000

NGINX is an advanced web browser that can be used as a reverse proxy, load balancer, mail proxy, and caching mechanism. In this short tutorial, we scale a golang REST API service with docker-compose and distribute all the load from the client to these multiple service instances.

The client makes an API call to http://localhost:8000/ endpoint, which hits the nginx service that internally proxies the request to one of the 3 instances of the REST servers running on the port 8000. We will be using the Round robin load-balancing technique that can help us to almost evenly distribute the load across every instance.

build golang servers image with Dockerfile

FROM golang:1.18-alpine AS builder

RUN mkdir /app
WORKDIR /app
COPY . .

RUN CGO_ENABLED=0 GOOS=linux go build -o app cmd/api/main.go

FROM alpine:latest AS production
WORKDIR /app
COPY --from=builder /app/app .

CMD [ "./app" ]

docker-compose file - run the golang servers and nginx load balancer

version: "3.8"

services:
  rest_api:
    build:
      context: ./server
      dockerfile: Dockerfile
    environment:
      - PORT=:8000
    networks:
      - server-network
    command: /start

  nginx:
    image: nginx:latest
    volumes:
      - ./conf.d/nginx.conf:/etc/nginx/nginx.conf
    depends_on:
      - rest_api
    ports:
      - 8000:8000
    networks:
      - server-network

networks:
  server-network:
    driver: bridge

nginx conf file - We'll define events and set the expected maximum client concurrent connections to 1024. We can define a group of REST servers(image=rest_api) inside the upstream block. We can also mention the load balancing method inside the upstream block. NGINX server will listen on the port 8000 of the host. NGINX will proxy all the requests hitting at / route to app_servers group.

events {
    worker_connections 1024;
}

http {
    upstream app_servers {
        # default load balancing method is Round Robin
        # other ex. least_conn, ip_hash, etc
        server rest_api:8000;
    }

    server {
        listen 8000;

        location / {
            proxy_pass http://app_servers;
            proxy_set_header Host $host;
            proxy_set_header X-Real-IP $remote_addr;
            proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
        }
    }
}

Now we can start multiple instances of golang REST API servers with the following command -

$ docker-compose up --build --scale rest_api=3

NGINX will serve the same request through different REST server instances -

Twelve Factor app

Saurabh Bomble — Tue, 25 Apr 2023 07:51:02 +0000

The twelve factor methodology is a language and platform agnostic set of design principles that can help to build portable, scalable, maintainable, compatible, and robust apps rapidly. It was started by the Heroku engineering team.

Reading - https://12factor.net/

As per the documentation -

The twelve-factor app is a methodology for building software-as-a-service apps that:

Use declarative formats for setup automation, to minimize time and cost for new developers joining the project.
Have a clean contract with the underlying operating system, offering maximum portability between execution environments.
Are suitable for deployment on modern cloud platforms, obviating the need for servers and systems administration.
Minimize divergence between development and production, enabling continuous deployment for maximum agility.
And can scale up without significant changes to tooling, architecture, or development practices.

The Twelve Factors

1. Codebase

one codebase for an app tracked by version control where team members push the code.
microservices can have different codebases since they are essentially different apps.
within each codebase, we can have multiple deployments - dev/staging/test/prod envs.

2. Explicitly declare and isolate dependencies

twelve-factor app declares all the dependencies completely and exactly, via a dependency declaration manifest ex. requirements.txt, package.json files.
we can install app-related dependencies by creating virtual environments for the app to avoid interference from outside system dependencies. ex. node_modules, python venvs, etc.
docker containers help to run our apps in a self-contained environment that is isolated from the host system.

3. Config

An app’s config is everything likely to vary between deploys (staging, production, developer environments, etc). This includes:
- Resource handles to the database, Memcached, and other backing services.
- Credentials to external services such as Amazon S3 or Twitter.
- Per-deploy values such as the canonical hostname for the deploy.
twelve-factor app stores configs of the app in env files that are different for different deployment environments.

4. Backing services

backing services are shared resources that are attached externally to twelve-factor apps.
examples include:
- databases - MySQL, Postgres, Mongo, etc.
- messaging/queueing systems - RabbitMQ, MQTT, Beanstalked, etc.
- SMTP - postfix
- caching systems - Redis, Memcached, etc.

5. Build, release, run

Strictly separate build, release, and run stages.
build stage - converts a code repo into an executable bundle known as a build(an executable file). The build stage fetches dependencies and compiles binaries, and assets.
release stage - build from the previous stage along with configs makes a release with a unique release version/timestamp/id. The release is ready to be executed immediately in the execution environment. Releases are append-only ledgers and a release cannot be mutated once it is created. Every small change in the codebase should create a new release. twelve-factor app allows rollbacks to previous releases.
run stage (aka runtime) - runs the app in an execution environment against the release.

6. Processes

Twelve-factor processes are stateless and share-nothing.
If the same app is running on multiple processes and any information that is needed to be shared across all processes should be stored in a stateful backing service, typically a database/cache.
The data stored on a process won’t be available across other apps running across other processes.

7. Port binding

All the services needed to be exported should be connected with ports.
The web app exports HTTP as a service by binding to a port, and listening to requests coming in on that port.

8. Concurrency

In the twelve-factor app, processes are first-class citizens.
Apps should scale out horizontally not vertically by running multiple instances of the app on different machines concurrently.
The app should be built keeping concurrency in mind.

9. Disposability

The twelve-factor app’s processes are disposable, meaning they can be started or stopped at a moment’s notice.
This facilitates fast elastic scaling, rapid deployment of code or config changes, and robustness of production deploys.
Processes should strive to minimize startup time.
codebase should not have any complex start scripts.
Processes shut down gracefully when they receive a SIGTERM signal from the process manager.
docker stop sends a SIGTERM signal and after a grace period if the container is not stopped docker sends a SIGKILL signal to forcefully terminate the process running inside the container.
During the grace period after SIGTERM signal, apps/processes can stop accepting new requests and complete current requests.
The app should be able to handle SIGTERM signals to avoid any discrepancies.

10. Dev/Prod parity

Keep development, staging, and production as similar as possible.
The twelve factor app is designed for continuous deployment by keeping the gap between development and production small.
Possible gaps :
- Make the time gap small: a developer may write code and have it deployed hours or even just minutes later.
- Make the personnel gap small: developers who wrote code are closely involved in deploying it and watching its behaviour in production.
- Make the tools gap small: keep development and production as similar as possible.
- The twelve-factor developer resists the urge to use different backing services between development and production.

11. Logs

Treat logs as event streams.
Logs are the stream of aggregated, time-ordered events collected from the output streams of all running processes and backing services.
We can’t store logs on the containers as they get destroyed and recreated very often, so we can lose them.
Writing logs to a centralized backing service is also discouraged as it leads to tight coupling.

12. Admin processes

one-time admin operations involve running a migration, seeding a database, or resetting some database records.
All the one-time admin operations should be run on different processes but on an identical setup.

Aggregation in MongoDB

Saurabh Bomble — Fri, 15 Apr 2022 10:45:27 +0000

Hello, devs 👋

In this blog, we will learn the basics of the MongoDB aggregation framework to filter, sort, group, and transform our MongoDB results. MongoDB helps us to do all these operations through aggregation pipelines which are a series of operations that process data documents sequentially.

For practice, we can use - Mongo playground

Input docs

[
  {
    "key": 1,
    username: "saurabh",
    age: 18,
    languages: [
      "c",
      "c++"
    ]
  },
  {
    "key": 2,
    username: "leonord",
    age: 22,
    languages: [
      "c",
      "c++",
      "java"
    ]
  },
  {
    "key": 3,
    username: "sheldon",
    age: 14,
    languages: [
      "c",
      "c++",
      "java",
      "python"
    ]
  },
  {
    "key": 4,
    username: "howard",
    age: 32,
    languages: [
      "c",
      "c++",
      "java",
      "python",
      "dart"
    ]
  },
  {
    "key": 5,
    username: "raj",
    age: 5,
    languages: [
      "c",
      "c++",
      "java",
      "python",
      "dart",
      "ts"
    ]
  }
]

1. $group aggregation = used for grouping and summarizing documents. We must specify an _id field with a valid expression.

Query

db.collection.aggregate([
  {
    $group: {
      _id: "table_stats",
      // Get count of all docs in the collection
      count: {
        $sum: 1
      },
      // Get age stats by grouping age field
      avgAge: {
        $avg: "$age"
      },
      maxAge: {
        $max: "$age"
      },
      minAge: {
        $min: "$age"
      },
      sumAge: {
        $sum: "$age"
      },
      // Get all usernames by grouping username field
      allUsernames: {
        $push: "$username"
      },
      // Get username of first doc
      firstUsername: {
        $first: "$username"
      },
      // Get username of last doc
      lastUsername: {
        $last: "$username"
      }
    }
  }
])

Result

[
  {
    "_id": "table_stats",
    "allUsernames": [
      "saurabh",
      "leonord",
      "sheldon",
      "howard",
      "raj"
    ],
    "avgAge": 18.2,
    "count": 5,
    "firstUsername": "saurabh",
    "lastUsername": "raj",
    "maxAge": 32,
    "minAge": 5,
    "sumAge": 91
  }
]

2. $match aggregation = This is used to reduce the number of docs in the result by filtering.

Query

// Match all docs where `age` is greater than 20 or equal to 20

db.collection.aggregate([
  {
    "$match": {
      age: {
        $gte: 20
      }
    }
  }
])

Result

[
  {
    "_id": ObjectId("5a934e000102030405000001"),
    "age": 22,
    "key": 2,
    "languages": [
      "c",
      "c++",
      "java"
    ],
    "username": "leonord"
  },
  {
    "_id": ObjectId("5a934e000102030405000003"),
    "age": 32,
    "key": 4,
    "languages": [
      "c",
      "c++",
      "java",
      "python",
      "dart"
    ],
    "username": "howard"
  }
]

Query

// Match all docs that have languages either `python` or `dart` or both

db.collection.aggregate([
  {
    "$match": {
      languages: {
        $in: [
          "python",
          "dart"
        ]
      }
    }
  }
])

Result

[
  {
    "_id": ObjectId("5a934e000102030405000002"),
    "age": 14,
    "key": 3,
    "languages": [
      "c",
      "c++",
      "java",
      "python"
    ],
    "username": "sheldon"
  },
  {
    "_id": ObjectId("5a934e000102030405000003"),
    "age": 32,
    "key": 4,
    "languages": [
      "c",
      "c++",
      "java",
      "python",
      "dart"
    ],
    "username": "howard"
  },
  {
    "_id": ObjectId("5a934e000102030405000004"),
    "age": 5,
    "key": 5,
    "languages": [
      "c",
      "c++",
      "java",
      "python",
      "dart",
      "ts"
    ],
    "username": "raj"
  }
]

Query

// Match all docs with username `saurabh`

db.collection.aggregate([
  {
    "$match": {
      username: {
        $eq: "saurabh"
      }
    }
  }
])

Result

[
  {
    "_id": ObjectId("5a934e000102030405000000"),
    "age": 18,
    "key": 1,
    "languages": [
      "c",
      "c++"
    ],
    "username": "saurabh"
  }
]

Some of the match operators

$gte = Matches if values are greater or equal to the given value.
$lte = Matches if values are less or equal to the given value.
$lt = Matches if values are less than the given value.
$gt = Matches if values are greater than the given value.
$eq = Matches values that are equal to the given value.
$ne = Matches values that are not equal to the given value.
$in = Matches any of the values in an array.
$nin = Matches none of the values specified in an array.

3. $skip and $limit aggregation = $skip takes a positive integer that specifies the maximum number of documents to skip. $limit limits the number of documents to look at, by the given number starting from the current positions.

Without skip and limit

Query

// Get all docs with username lexicographically less than or equal to "saurabh"

db.collection.aggregate([
  {
    "$match": {
      username: {
        $lte: "saurabh"
      }
    }
  },
  // ignore this aggregation, for now, we'll look into it later
  {
    $project: {
      "languages": 0,
      "key": 0,
      "id": 0
    }
  }
])

Result

[
  {
    "_id": ObjectId("5a934e000102030405000000"),
    "age": 18,
    "username": "saurabh"
  },
  {
    "_id": ObjectId("5a934e000102030405000001"),
    "age": 22,
    "username": "leonord"
  },
  {
    "_id": ObjectId("5a934e000102030405000003"),
    "age": 32,
    "username": "howard"
  },
  {
    "_id": ObjectId("5a934e000102030405000004"),
    "age": 5,
    "username": "raj"
  }
]

Without skip = 1 and limit = 2

Query

// Skip the first doc ($skip) and return next 2 docs ($limit)

db.collection.aggregate([
  {
    "$match": {
      username: {
        $lte: "saurabh"
      }
    }
  },
  {
    $skip: 1
  },
  {
    $limit: 2
  },
  // ignore this aggregation for now
  {
    $project: {
      "languages": 0,
      "key": 0,
      "id": 0
    }
  }
])

Result

[
  {
    "_id": ObjectId("5a934e000102030405000001"),
    "age": 22,
    "username": "leonord"
  },
  {
    "_id": ObjectId("5a934e000102030405000003"),
    "age": 32,
    "username": "howard"
  }
]

4. $sort aggregation = Sorts all input documents and returns them to the pipeline in sorted order.

Query

// Get all the docs sorted in ascending order on the `age` field

db.collection.aggregate([
  {
    $sort: {
      age: 1
    }
  },
  // ignore this aggregation for now
  {
    $project: {
      "languages": 0,
      "key": 0,
      "id": 0
    }
  }
])

Result

[
  {
    "_id": ObjectId("5a934e000102030405000003"),
    "age": 32,
    "username": "howard"
  },
  {
    "_id": ObjectId("5a934e000102030405000001"),
    "age": 22,
    "username": "leonord"
  },
  {
    "_id": ObjectId("5a934e000102030405000000"),
    "age": 18,
    "username": "saurabh"
  },
  {
    "_id": ObjectId("5a934e000102030405000002"),
    "age": 14,
    "username": "sheldon"
  },
  {
    "_id": ObjectId("5a934e000102030405000004"),
    "age": 5,
    "username": "raj"
  }
]

*5. $unwind aggregation * = This is used to unwind documents that are using arrays.

Query

db.collection.aggregate([
  {
    $unwind: "$languages"
  },
])

Result

[
  {
    "_id": ObjectId("5a934e000102030405000000"),
    "age": 18,
    "key": 1,
    "languages": "c",
    "username": "saurabh"
  },
  {
    "_id": ObjectId("5a934e000102030405000000"),
    "age": 18,
    "key": 1,
    "languages": "c++",
    "username": "saurabh"
  },
  {
    "_id": ObjectId("5a934e000102030405000001"),
    "age": 22,
    "key": 2,
    "languages": "c",
    "username": "leonord"
  },
  {
    "_id": ObjectId("5a934e000102030405000001"),
    "age": 22,
    "key": 2,
    "languages": "c++",
    "username": "leonord"
  },
  {
    "_id": ObjectId("5a934e000102030405000001"),
    "age": 22,
    "key": 2,
    "languages": "java",
    "username": "leonord"
  },
  {
    "_id": ObjectId("5a934e000102030405000002"),
    "age": 14,
    "key": 3,
    "languages": "c",
    "username": "sheldon"
  },
  {
    "_id": ObjectId("5a934e000102030405000002"),
    "age": 14,
    "key": 3,
    "languages": "c++",
    "username": "sheldon"
  },
  {
    "_id": ObjectId("5a934e000102030405000002"),
    "age": 14,
    "key": 3,
    "languages": "java",
    "username": "sheldon"
  },
  {
    "_id": ObjectId("5a934e000102030405000002"),
    "age": 14,
    "key": 3,
    "languages": "python",
    "username": "sheldon"
  },
  {
    "_id": ObjectId("5a934e000102030405000003"),
    "age": 32,
    "key": 4,
    "languages": "c",
    "username": "howard"
  },
  {
    "_id": ObjectId("5a934e000102030405000003"),
    "age": 32,
    "key": 4,
    "languages": "c++",
    "username": "howard"
  },
  {
    "_id": ObjectId("5a934e000102030405000003"),
    "age": 32,
    "key": 4,
    "languages": "java",
    "username": "howard"
  },
  {
    "_id": ObjectId("5a934e000102030405000003"),
    "age": 32,
    "key": 4,
    "languages": "python",
    "username": "howard"
  },
  {
    "_id": ObjectId("5a934e000102030405000003"),
    "age": 32,
    "key": 4,
    "languages": "dart",
    "username": "howard"
  },
  {
    "_id": ObjectId("5a934e000102030405000004"),
    "age": 5,
    "key": 5,
    "languages": "c",
    "username": "raj"
  },
  {
    "_id": ObjectId("5a934e000102030405000004"),
    "age": 5,
    "key": 5,
    "languages": "c++",
    "username": "raj"
  },
  {
    "_id": ObjectId("5a934e000102030405000004"),
    "age": 5,
    "key": 5,
    "languages": "java",
    "username": "raj"
  },
  {
    "_id": ObjectId("5a934e000102030405000004"),
    "age": 5,
    "key": 5,
    "languages": "python",
    "username": "raj"
  },
  {
    "_id": ObjectId("5a934e000102030405000004"),
    "age": 5,
    "key": 5,
    "languages": "dart",
    "username": "raj"
  },
  {
    "_id": ObjectId("5a934e000102030405000004"),
    "age": 5,
    "key": 5,
    "languages": "ts",
    "username": "raj"
  }
]

Query

db.collection.aggregate([
  {
    $unwind: "$languages"
  },
  {
    $match: {
      username: "saurabh"
    }
  }
])

Result

[
  {
    "_id": ObjectId("5a934e000102030405000000"),
    "age": 18,
    "key": 1,
    "languages": "c",
    "username": "saurabh"
  },
  {
    "_id": ObjectId("5a934e000102030405000000"),
    "age": 18,
    "key": 1,
    "languages": "c++",
    "username": "saurabh"
  }
]

6. $project aggregation = Get some specific fields from a collection by giving the keys values as 0 (exclude) or 1 (include)

Basic Query

db.collection.aggregate([
  {
    $project: {
      username: 1,
      languages: 1
    }
  },
  {
    $unwind: "$languages"
  },
  {
    $match: {
      username: "saurabh"
    }
  }
])

Result

[
  {
    "_id": ObjectId("5a934e000102030405000000"),
    "languages": "c",
    "username": "saurabh"
  },
  {
    "_id": ObjectId("5a934e000102030405000000"),
    "languages": "c++",
    "username": "saurabh"
  }
]

Query with update column names

db.collection.aggregate([
  {
    $project: {
      "system_username": "$username",
      "system_languages": "$languages"
    }
  },
  {
    $unwind: "$system_languages"
  },
  {
    $match: {
      system_username: "saurabh"
    }
  }
])

Result

[
  {
    "_id": ObjectId("5a934e000102030405000000"),
    "system_languages": "c",
    "system_username": "saurabh"
  },
  {
    "_id": ObjectId("5a934e000102030405000000"),
    "system_languages": "c++",
    "system_username": "saurabh"
  }
]

Follow for more cool articles

Thanks 😎

Dockerize your MERN + Flask app

Saurabh Bomble — Thu, 03 Feb 2022 09:40:42 +0000

Hello developers,

In this short😂 article, we will try to understand why and how to use docker in your next project.

Why should I dockerize my project?

Suppose, we have a new developer joining our team, rather than wasting other developers' time in setting up the project, the new member can just run docker-compose up and get all the services up and running🚀

There have many instances where code runs on a local machine but breaks on someone else's or in the production. It happens cuz of different versions of libraries, configs, and services like databases or cache. Docker installs all the services and runs them in an isolated environment as per the given instructions.
Docker also helps to build workflows and automation for testing and linting with CI/CD pipelines which makes it easier to deploy to production.

App Description

Suppose your app has

Two servers running on -
- Express.js API - http://localhost:8000
- Flask API - http://localhost:5000
Two services running for Express server -
- Mongo Db - mongodb://localhost:27017/db_name
- Redis - redis://localhost:6379
React/Next.js frontend running - http://localhost:3000

Glossary -

Containers vs Images - We can consider Images as classes of OOPs and Containers as instances of those images. Images are mostly huge files that are built based on the Dockerfile and containers are isolated environments that run instances of those images. Images are stored locally but can be pushed to the Docker registry https://registry.hub.docker.com/ to share with other members. With our example, we will build images for every server and service.
Docker compose - It's a daunting task to build and run images for every service for a bigger project. So, we use docker compose to consolidate all these docker images and make them build and run together.
Volumes - Volumes are storages used for persistence. If we don't use volumes in services like mongo DB and Redis, then all the data will be lost as soon as the container is stopped or removed and can't be accessed after running the container again.
We also use volumes to map/mirror local code/files with the code/files inside the environment so that any changes made in local code gets mirrored and the server can be rerun with tools like nodemon
Dockerfile - Dockerfile has the set of instructions to build the image. It's kinda like GitHub actions.
.dockerignore - This file is like .gitignore file and has a list of excluded modules and files that you don't want in your isolated environment. ex. node_modules.
FROM - FROM instruction initializes a new build stage and sets the base Image (python for flask project and node for a node-based projects). A valid Dockerfile must start with a FROM instruction. It will pull the image from the public repo (Dockerhub) if not available locally. Always try to find a lighter version of the image (ex. alpine for node.js) to lower the overall size of the image of the app.
EXPOSE - EXPOSE is used to map the port of the host to the port of the container so that we can use the same port on the localhost as written in the code.

Folder structure of the project

Dockerfiles for Services

Flask API -
- Running flask will need python.
- set working directory inside environment (directory ai-models will be created by docker).
- copy requirements.txt file from the host into the container.
- RUN the command to install dependencies mentioned in the requirements.txt.
- Now, COPY all the remaining files inside the container.
- Set required env variables inside the container.
- Enter the final command for running the server with CMD.

Dockerfile -

FROM python:3.7-slim-buster

WORKDIR /usr/src/ai-models

COPY requirements.txt .

RUN pip3 install -r requirements.txt

COPY . .


# To use flask run instead of python main.py
ENV FLASK_APP=main.py

CMD ["flask", "run", "--host", "0.0.0.0"]

.dockerignore - I have used virtual environment (skip this if you haven't)

/venv

Build and Spin up the container alone - If you want to a single server, you can build the image of that server and spin up a container for that image.

a. Move into the API directory

 cd flask-api

b. Build an image - Next step is to build the image with a tag(i.e name of the image) and Dockerfile location ( '.' => current directory)

docker build -t app-flask-api  .

c. Run the container - Map the ports(-p) and spin up the container in detached mode (-d) to get API working

docker run -dp 5000:5000 api-flask-api

Express API -
- Running Express will need nodejs as a base image
- Use labels to describe the image (optional)
- set working directory inside environment
- copy package.json and package-lock.json files from the host into the container
- RUN a command to install dependencies mentioned in the package.json. If you use npm ci, it's important to have package-lock.json file inside the environment.
- Now, COPY all the remaining files inside the container.
- Set required env variables inside the container (if any or want to run it alone)
- Enter the final command for running the server with CMD

Dockerfile -

FROM node:alpine

LABEL version="1.0.0"
LABEL description="Community API server's image"

WORKDIR /usr/src/api

COPY package*.json .

# RUN yarn install --immutable
RUN npm ci

COPY . .

# CMD [ "yarn", "dev" ]
CMD [ "npm", "run", "dev" ]

.dockerignore - To avoid errors do not copy node_modules into your container.

node_modules

React/Next.js frontend -
- React's image can be build by following the same steps as Express API.

Dockerfile -

FROM node:alpine

LABEL version="1.0.0"
LABEL description="Next.js frontend image"

WORKDIR /usr/src/web

COPY package*.json . 
COPY yarn.lock .

# RUN npm ci
RUN yarn install --immutable

COPY . .

# CMD [ "npm", "run", "dev" ]
CMD [ "yarn", "dev" ]

.dockerignore - To avoid errors do not copy node_modules into your container.

node_modules

Docker compose - We will set instructions in docker-compose.yml file needed to spin up all the services and API containers with a single command.

We will use version 3.8 of docker-compose file formatting
Every image that is needed to spin up the container is a service a. Redis - cache_service (can be named anything) b. Mongo Database - db_service c. Flask API for AI models - api_models d. Express API - api_community e. Frontend web server - web
For Redis and Mongo DB services, we will use prebuilt images from the public repository (Dockerhub). For other services, we will build the images based on the Dockerfiles that we have written.
We will use named volumes for the persistence of the data (in Redis and Mongo DB services) and for mapping the files between host and container (in APIs and frontend). We need to create the volumes before using them inside any service.
restart = always makes sure that services will be restarted after every crash
Mention all of the env variables under environment.
By default Compose sets up a single network for your app that is shared between different services but we can specify our own custom network(here, shared_network) that could be different for different services. When we run docker-compose up, all the containers will join specified networks.
Hosts of the Redis and Mongo DB will not be localhost anymore but the corresponding service.
- Redis - redis://cache_service:6379
- Mongo db - mongodb://db_service:27017/db_name
Map all the required ports, so that they can be accessible from the host
Mention that express API depends_on cache_service and db_service docker-compose.yml -

version: "3.8"
services:
  cache_service:
    container_name: cache_service
    image: redis:6.2-alpine
    restart: always
    volumes:
      - cache_service:/data/
    ports:
      - 6379:6379
    networks:
      - shared_network

  db_service:
    container_name: db_service
    image: mongo
    restart: always
    volumes:
      - db_service:/data/db 
    ports:
      - 27017:27017
    networks:
      - shared_network

  api_models:
    container_name: api_models
    build: 
      context: ./flask-api
      dockerfile: Dockerfile
    volumes:
      - ./flask-api:/usr/src/ai-models
    ports:
      - 5000:5000
    restart: always
    networks:
      - shared_network

  api_community:
    container_name: api_community
    depends_on:
      - cache_service
      - db_service
    build: 
      context: ./express-api # Path to the directory of Express server
      dockerfile: Dockerfile # name of the Dockerfile 
    restart: always
    volumes:
      # Map local code to the code inside container and exclude node_modules
      - ./express-api:/usr/src/api 
      - /usr/src/api/node_modules 
    ports:
      - 8000:8000
    environment: 
      - PORT=8000
      - DB_URI=mongodb://db_service:27017/db_name 
      - REDIS_URL=redis://cache_service:6379
      - ACCESS_TOKEN_SECRET=12jkbsjkfbasjfakb12j4b12jbk4
      - REFRESH_TOKEN_SECRET=lajsbfqjb2l1b2l4b1lasasg121
    networks:
      - shared_network

  web:
    container_name: web
    depends_on:
      - api_community
    build: 
      context: ./web-app
      dockerfile: Dockerfile
    restart: always
    volumes:
      - ./web-app:/usr/src/web
      - /usr/src/web/node_modules 

    ports:
      - 3000:3000
    networks:
      - shared_network


volumes:
  db_service: 
    driver: local
  cache_service: 
    driver: local

# [optional] If the network is not created, docker will create by itself
networks:
  shared_network:

And we are done, I think

To run all the containers, go to the root directory where docker-compose.yml resides and -

docker-compose up

To stop the containers

docker-compose down

If you have made it till here, WOW

Follow for more cool articles

Thanks 😎

Basics of Solidity

Saurabh Bomble — Sat, 27 Nov 2021 21:23:33 +0000

Hello Developers 👋

In this article, we are going to learn the basics of solidity. Solidity is a new, high-level programming language created specifically to implement smart contracts on the Ethereum blockchain.

Features of Solidity
Access Modifiers
memory vs storage
Basic Types
Variables
Data Structures
Conditionals and Loops
Functions
Error handling
Inheritance

Features of Solidity

Object-oriented programming language just like C++ or Java.
Statically typed language like Typescript - helps to catch bugs in the early stage of the development process.
Highly influenced by other programming languages such as Javascript, Python, and C++.
Supports Inheritance of contracts and runs on Ethereum Virtual Machine(EVM).

Access Modifiers

Public: The value of the variables and functions can be accessed within the contract, child contract, and from external functions.

contract MyContract {
    uint256 public myPublicVar;
    function getVar() public view returns(uint) { 
        return myPublicVar;
    }
}

External: The value of the variables and functions can be accessed only from external functions.

contract MyContract {
    uint256 external externalVar; 
    function getVar() external view returns(uint) { 
        return externalVar;
    }
}

Internal: The value of the variables and functions can be accessed within the contract and child contract.

contract MyContract  {
    uint256 internal internalVar; 
    function getVar() internal view returns(uint) { 
        return internalVar;
    }
}

Private: The value of the variables and functions can be accessed within the same contract only.

contract MyContract { 
    uint256 private privateVar; 
    function getVar() private view returns(uint) { 
        return privateVar;
    }
}

memory vs storage

We can consider memory as a RAM and storage as a Hard disk. Any value that is stored in a memory variable will be temporary and will be wiped after execution stops while the value in the storage variable will be stored permanently on the blockchain and after the new execution, the variable will still have the previously stored value. The structure of storage can never be changed after the deployment of the contract but values inside it can be.

memory consumes comparatively less gas.
memory can only be used in methods, not at the contract level (state variable).
Solidity tries to declare variables in storage if the memory keyword is not used.
memory creates a new copy of the variable while storage points to the same location and can alter the values of the original variable.

pragma solidity >=0.8.7;

// Creating a contract
contract MemoryContract
{ 
    // state variable (always in storage)
    uint[] public numbers;

    function addNumbers() public  {
        numbers.push(1);
        numbers.push(2);  
        // numbers = [1, 2]

        //1. Creating a new memory instance 
        uint[] memory myMemoryNumbers = numbers;
        myMemoryNumbers[0] = 10;
        // myMemoryNumber = [10, 2]
        // numbers = [1, 2]  


        //2. Creating a new storage instance
        uint[] storage myStorageNumbers = numbers;
        myStorageNumbers[0] = 10;
        // myStorageNumbers = [10, 2]
        // numbers = [10, 2] 
    } 
}

Basic Types

// specify the version of solidity
pragma solidity >=0.8.7;   

// Creating a contract
contract BasicTypes {   

    // Boolean value
    bool public myBoolean = false;

    // Integer variable
    int32 public myInt = -60313;
    // unsigned Integer
    uint public myUInt = 60313; // alias for uint256

    // String variable
    string public myStr = "Saurabh";
    // Byte string (uses less gas)
    bytes32 public myByteString = "Bomble";

    // Byte variable
    bytes1 public b = "a";

    // Defining an enumerator
    enum myEnum { ACCEPTED, PENDING, REJECTED }  
}

Variables

Syntax

<type> <access modifier> <variable name>;

State variables - These variables are the variables that are stored permanently in the contract storage on the blockchain.
Local variables - These variables are not stored on blockchain but are created within a scope (e.g function) to help write business logic.
Global variables - These variables exist in the global namespace and can be invoked from anywhere in the contract. Global variables hold information about different transactions and blockchain properties. eg. block, msg, and tx.

pragma solidity >=0.8.7;

contract VariableType {
    // State variables => stored on blockchain  
    uint myUint = 1;  
    //160-bit ethereum address 
    address public myAddress = 0xE9Bc4d002161aA50E15728ba19aDC3aA34a91D98; 

    constructor() {
        // msg is global variable
        myAddress = msg.sender  // address of user who deployed the contract
    }

    function getValue() public pure returns(uint) {
        myAddress = msg.sender  // address of current caller
        uint value = 1; // local variable
        return value;
    }
}

Data Structures

Array - We can store a collection of elements of the same type in an array. We can access, push, and pop an element from an array and can calculate the length of the array. Arrays in solidity are 0-indexed.
Struct - Struct allows us to create user-defined data types.
Mapping - Mapping in solidity is like unordered_map in C++ or dictionary in python. It is used to store key-value pairs. Here, keys can be of any built-in data type and values can be of any type including user-defined types.

pragma solidity >=0.8.7;

contract DataStructure {
    // Array
    uint[] public myIntArray = [1, 2, 3];
    string[] public myStringArray = ["apple", "banana"];
    uint[][] public array2D = [ [1,2,3], [4,5,6] ];

    function addValue(string memory _value) public {
        myStringArray.push(_value);
    }

    function valueCount() public view returns(uint) {
        return myStringArray.length;
    } 

    // Mapping (key-value store)
    mapping(uint => string) public names;

    constructor() {
        names[1] = "Saurabh";
        names[2] = "John";
        names[3] = "Doe"; 
    }

    struct Book {
        string title;
        string author;
    }


    mapping(uint => Book) books; // key = id of book(unit), value = Book
    // Nested mapping
    mapping(address => mapping(uint => Book)) public myBooks; 

    function addBook(uint _id, string memory _title, string memory _author) public {
        books[_id] = Book(_title, _author);
    }

    function getBook(uint _id) public view returns(string memory, string memory){
        string memory _title = books[_id].title;
        string memory _author = books[_id].author; 
        return (_title, _author);
    }

    function addMyBook(uint _id, string memory _title, string memory _author) public {
        myBooks[msg.sender][_id] = Book(_title, _author);
    } 
}

Conditionals and Loops

Conditionals - Conditionals include if-else, ternary operator
For Loop - similar to c++ loop. Takes three arguments separated by a semi-colon. viz. initialization, condition, and iteration.
While loop - Execute a statement or block of statements repeatedly as far as the condition is true and once the condition becomes false the loop terminates.
Do while - Similar to while loop except the condition check happens at the end of the loop

pragma solidity >=0.8.7;

contract ConditionalsLoops {
    // Loop
    uint[] public numbers = [1,2,3,4,5];
    address public owner;

    constructor() {
        owner = msg.sender;
    }

    function countEven() public view returns(uint count) {
        count = 0;
        for(uint i = 0; i < numbers.length; i++) {
            count += isEven(numbers[i]) ? 1 : 0;
        } 
    }

    // Conditionals (just like js)
    function isEven(uint _value) public pure returns(bool) {
        return _value % 2 == 0; 
    }

    function isOwner() public view returns(bool) {
        if(msg.sender == owner) return true;
        return false;
    }
}

Functions

View functions - This function can only read the state variables.
Pure functions - This function can neither read nor modify the state variables.
returns(type) - denotes return type of the function.

pragma solidity >=0.8.7;

contract FunctionContract {  
    uint myUint = 1;

    function getValue() public pure returns(uint) {
        uint value = 1; 
        return value;
    }

    function getMyUint() public view returns(uint) { 
        return myUint;
    }
}

Error handling

revert(string memory msg) - aborts the execution and revert any changes done to the state. returns the msg.

if(currentStatus == Status.VACANT) {
     revert("Currently occupied");
}

require(bool condition, string memory message) − If the condition is not met, this method call reverts to its original state and throws a custom message.

require(currentStatus == Status.VACANT, "Currently occupied");

Inheritance

Inheritance is one of the most important features of the object-oriented programming language. It makes code scalable, reusable, and more organized. Parent contract is known as base contract and the contract that inherits methods and properties from the base contract is called a derived contract.

A derived contract can access all public and internal features of the base contract.
Function overriding is allowed provided the function signature remains the same.
super keyword in the child contract helps to call the constructor of the parent.
Contracts can inherit other contracts by using the is keyword.
Function to be overridden by a child contract must be declared as virtual.

pragma solidity >=0.8.7; 

contract BaseContract {  
    uint public a;   
    uint public b;    

    // Wrong defination to be corrected in derived 
    function getDiff() public view virtual returns(uint) {
        return(a*b);
    }
} 

// Defining child contract 
contract DerivedContract is BaseContract{   
    constructor(uint _a, uint _b) {
        a = _a;
        b = _b;
        super;
    }

    // Correct defination with same signature
    function getDiff() public view virtual override returns(uint) {
        return(a-b);
    }
} 

// Defining calling contract
contract CallerContract {
    DerivedContract derivedContract;
    constructor(uint _val1, uint _val2) {
        derivedContract = new DerivedContract(_val1, _val2); 
    }

    function getDiffAandB() public view returns(uint) { 
        return derivedContract.getDiff();
    }
}

Conclusion

Thanks for reading🙏. That's it for this chapter. We have learned most of the basic concepts in solidity to get started to write our smart contracts.

Follow me on dev.to for

Upcoming articles on Blockchain technology.
Web development (React.js+Nest.js)
App development (Flutter, React Native)

Best portfolio web designs

Saurabh Bomble — Thu, 30 Sep 2021 10:19:20 +0000

Hello Developers 👋🏼

I am Saurabh, a beginner in the design world.
In this article, we'll see some of the best web designs that I use to get my inspirations and to get my research started. I hope this will help you too.

Portfolio websites are the best way to boast about your front-end skills. A portfolio website is nothing less than a resume for a developer. All of these listed websites are portfolio sites of wonderful designers and developers.

Some tips for your next design

Do your research regarding the project.
Keep it simple and stupid.
Keep it clean.
Don't use multiple fonts or colours.
Create typography hierarchy and font styles first.
Consistency always makes your design look professional - use consistent icons, illustrations, and images.
Check contrast and space - need to have good contrast between element and background. Appropriate breathing space inside cards and buttons is the key.
Use grids and rulers to keep components aligned perfectly.
Keep yourself updated with new trends.

Best practices - Apple's Guide

Let's get started 🚀

Preview - https://saurabhbomble.me

Design - Figma file

Code- Github

Thanks 😊 for reading. Hope you got some inspiration to get started with your portfolio site or to finish it. That's it for this chapter, will see you all in the next one.

DEV Community: Saurabh Bomble

Basic load balancer with NGINX

Twelve Factor app

The Twelve Factors

1. Codebase

2. Explicitly declare and isolate dependencies

3. Config

4. Backing services

5. Build, release, run

6. Processes

7. Port binding

8. Concurrency

9. Disposability

10. Dev/Prod parity

11. Logs

12. Admin processes

Aggregation in MongoDB

Input docs

Query

Result

Query

Result

Query

Result

Query

Result

Without skip and limit

Query

Result

Without skip = 1 and limit = 2

Query

Result

Query

Result

Query

Result

Query

Result

Basic Query

Result

Query with update column names

Result

Dockerize your MERN + Flask app

Why should I dockerize my project?

App Description

Glossary -

Folder structure of the project

Dockerfiles for Services

Basics of Solidity

Table of contents

Features of Solidity

Access Modifiers

memory vs storage

Example

Basic Types

Example

Variables

Syntax

Types of variables

Example

Data Structures

Example

Conditionals and Loops

Example

Functions

Example

Error handling

Example

Example

Inheritance

Conclusion

Best portfolio web designs

Hello Developers 👋🏼

1. Leonid Kostetskyi

2. Alex Frison de Isla

3. Ben Mingo

4. Abhishek Jha

5. Kuon Yagi

6. Brittany Chiang

7. Stephen calvillo