DEV Community: Pankaj Gupta

Consistency >>> Talent

Pankaj Gupta — Sun, 07 Apr 2024 18:57:22 +0000

I've been working in the field for 7 years now, and I wish I had someone in my life, a mentor if you will, who told me that consistency is much more appreciated and valued than talent.

I have to set something straight before I go on and tell you a story about it. When I say 'Talent,' I mean the exceptional people who are born to do this, not someone who doesn't want to accept they are not the best.

I had a hemi shoulder replacement on May 19th, 2023. The days that followed were the most painful days of my life. On the first day, the physios were trying to bring some movement, and due to the post-anesthetic effect, I couldn't feel anything. As soon as the anesthetic effect started fading, the shoulder felt like a block of ice. I had only 15 degrees of movement.

Every day, I regretted getting the replacement. It used to be the worst hour of the day for me when the physio visited the house. It felt like trying to bend a block of metal and hurt like hell. After the first week, without my realization, my movement had increased by 10 degrees, and now I could lift my hands higher. It felt good, but no matter how good I felt that day, the next day it would hurt like hell again. Every other week, out of nowhere, I could see some improvements. Weeks turned into months, and I found myself training with weights. Weights helped during the lying down position exercises and made it worse during the ones where I had to stand up. I was able to lift my hands about 140 degrees standing up at the end of 2 months.

The last few degrees were the most difficult to achieve; the pain was minimal by then, but to achieve each degree of movement, I had to work many times harder. On the last day of physiotherapy, I had a normally functioning shoulder. It took about 3 months.

I had joined back at work after 2 weeks after surgery, and Baani (my daughter) was 7 months old at the time. I never thought about the pain and suffering I went through again. During one of the visits a couple of months later, my doctor told me I was the first patient of the physio who visited my house, and out of the 4 similar patients in that time frame, I was the only one with almost complete movements. My physio had earned a promotion with the help of my case.

It hit me. If I had asked my physio about the other patients and learned that they were not getting past this stage of movements, I would have also given up along the way. If I had information that they were not consistent, I would have as well taken some days off here and there. I would have somehow found a way to not work as hard as I did.

I realized my recovery experiences can be very good life lessons for everyone, including myself.

Be consistent, set small goals, and achieve your targets regularly.
Don't compare yourself with others; it might have a negative impact on your performance.

Now, imagine you lead a tech team. You have two engineers in your team: one is an average engineer who is consistent and keeps everything up-to-date, attends every meeting, and updates you regularly on delays and other things. The other is brilliant, but his work fluctuates based on multiple factors. Some days he'll come up with brilliant solutions, but due to a lack of planning and consistency, he'll end up delaying the project.

A lot of people, including myself, fall into the second category. According to me, I was coming up with solutions, and nothing with me was wrong, but the reality is different.

If I were leading a team with two people like this, I would always rely on the first person, as I would always have more control over the situation.

Conclusion

Everyone needs someone to look up to and someone who can mentor them. If you need such help in your career, I am available. Please leave a comment and let me know.

Including Polymorphic Association In Rails To Avoid N+1 Queries

Pankaj Gupta — Sun, 04 Feb 2024 02:32:33 +0000

Introduction

Polymorphic association is a more advanced type of association where a model can belong to one or more models. For example:

class Document < ApplicationRecord
  belongs_to :object, polymorphic: true
end

class Note < ApplicationRecord
  has_many :documents, as: :object
end

class Story < ApplicationRecord
  has_many :documents, as: :object
end

Problem Statement

When querying the notes model, it is easier to include the associated documents because your model knows that there is only one model related to the documents. For example:

#note_ids is a list of note ids
notes = Note.includes(:documents).where(id: note_ids.pluck(:object_id))

documents = notes.flat_map { |note| note.documents }

This will execute two queries, one on the notes and one on the documents model. It seems straightforward, but if you want to achieve the same thing from the other way around, let's see what happens:

docs = Document.includes(:notes).where(id: documents.map(&:id))

docs.map(&:id)

It will throw an ActiveRecord::AssociationNotFoundError: Association named 'notes' was not found on Document; perhaps you misspelled it? error, which seems fair because we haven't declared anything on the Document model corresponding to the Note model.

Solution

We need to do that first of all, and let's see how it goes:

belongs_to :note

After including this and running this again, we get a surprising result:

docs = Document.includes(:note).where(id: documents.map(&:id))

# request on notes
SELECT "notes".* FROM "notes" WHERE "notes"."id" = $1  [["id", nil]]

This throws 2 DB requests, but it is not able to pick up the id for the notes somehow. If we provide the foreign key column, things might work in our favor, let's see:

belongs_to :note, foreign_key: "object_id"

Now, if we check the queries, we get something like this:

docs = Document.includes(:note).where(id: documents.map(&:id).first(2))

# query on the Document model
Document Load (216.8ms)  SELECT "documents".* FROM "documents" WHERE "documents"."id" IN ($1, $2)  [["id", 26954], ["id", 26955]]

# query on the Note
Note Load (215.3ms)  SELECT "notes".* FROM "notes" WHERE "notes"."id" IN ($1, $2)  [["id", 68057], ["id", 68058]]

We were able to reduce the N + 1 queries, but there is still an issue. It is querying for all the types, not just Note.

This would be a significant problem because this is not reliable at all. Imagine the consequences. To solve this issue, we have to use the following:

has_one :self_ref, class_name: 'Document', foreign_key: :id

has_one :note, through: :self_ref, source: :object, source_type: 'Note'

Now, every time we include note, it will only query on the notes model and not any other model.

Of course, the downside is you'll have to include all the models you want to query this way.

Conclusion

There can be other ways to solve this problem. This is just one of the solution. Please feel free to explore and connect for other solutions.

How To Setup MongoDB Like a Pro

Pankaj Gupta — Wed, 24 Jan 2024 07:46:28 +0000

Introduction

I started my career 6 years ago as a Frontend Engineer. We had mongoDB as our main database. Although I was a FE, I have always been fascinated by how the fancier things like replication, caching actually works. I was too afraid to try it, hoping I would break the running things on my system and would have to spend a lot of time fixing that. The fear stopped me from trying a lot of things I wanted to try. Today we will be trying one of those things and break the shackles.

We will be going through setting up mongoDB on your local machine and running multiple instance of it to setup replicaSet and how to add new machines and remove machine and all that, this skill can be directly transffered to a cloud based setup as well.

Installing MongoDB(community edition - 6.0)

Installing MongoDB DB can be varied based on your local machine. You can follow the steps below if you are using Ubuntu 22.04 otherwise you can follow this guide here.

MongoDB only supports the 64-bit versions of these platforms. To determine which Ubuntu release your host is running, run the following command on the host's terminal:

cat /etc/lsb-release

Import the public key used by the package management system

# install gnupg and curl if they are not already available
sudo apt-get install gnupg curl

# import the MongoDB public GPG key
curl -fsSL https://pgp.mongodb.com/server-6.0.asc | \
   sudo gpg -o /usr/share/keyrings/mongodb-server-6.0.gpg \
   --dearmor

Create a list file for MongoDB

echo "deb [ arch=amd64,arm64 signed-by=/usr/share/keyrings/mongodb-server-6.0.gpg ] https://repo.mongodb.org/apt/ubuntu jammy/mongodb-org/6.0 multiverse" | sudo tee /etc/apt/sources.list.d/mongodb-org-6.0.list

Reload local package database

sudo apt-get update

Install the MongoDB Package

sudo apt-get install -y mongodb-org

If there was no error, you would have successfully installed mongod and mongosh CLI Tools.

I ran into some issues, I will mention the StackOverflow links here which helped me with the issues:

Unable to locate package mongodb-org
- You can try to install the package, mongodb instead of monggodb-org. It is the unofficially package.
- I was installing the version 7 which is the latest, but I chose to used the previous version and the issue was solved.

Getting comfortable with `mongod` and `mongosh`

If you are running a Mac, you'll most likely be using brew to install all the background services and running them. You can use the similar commands in Ubuntu as well using systemd to run the services in daemon mode or in background.

Starting the service

sudo systemctl start mongod

This should start the mongod on port 27017, If you run into any issues, you'll have to debug the issue.

Stopping the service

sudo systemctl stop mongod

Restarting the service

sudo systemctl restart mongod

Connecting to a `mongod` instance with `mongosh`

mongosh

You'll be taken into the shell and you'll be connected to test db by default and you can experiment all you want with it.

Running `mongod` in foreground

You can run mongod on the foreground as well. No matter where you run it from background or foreground, it knows where to store the data, it knows where to write the logs, it knows what port to start on. How does it know? The secret is /etc/mongod.conf file.

Now if you cat /etc/mongod.conf file, you'll get something like this:

# mongod.conf

# for documentation of all options, see:
#   http://docs.mongodb.org/manual/reference/configuration-options/

# Where and how to store data.
storage:
  dbPath: /var/lib/mongodb
#  engine:
#  wiredTiger:

# where to write logging data.
systemLog:
  destination: file
  logAppend: true
  path: /var/log/mongodb/mongod.log

# network interfaces
net:
  port: 27017
  bindIp: 127.0.0.1


# how the process runs
processManagement:
  timeZoneInfo: /usr/share/zoneinfo

#security:

#operationProfiling:

#replication:

#sharding:

## Enterprise-Only Options:

#auditLog:

#snmp:

For the beginners, there are 2 important things mentioned in this file:

storage.dbPath
net.port

storage.dbPath

It refers to a directory, where mongo will be storing all the data.

net.port

It refers to a port, on which the mongod will start.

It is because of this conf file, you can't start multiple mongod instance. You can try starting two mongod instances on two terminal windows and it'll give a port error, mentioning the port is already occupied.

The good thing is now we know what are the keys and we have an option to overwrite these keys with our values. Lets explore that option.

Starting `mongosh` with different arguments

When you start mongod, it picks the default port from /etc/mongod.conf file, which is 27017. In order to start the mongod on a different port, either you can update the mongod.conf file to use a different port or you can provide command line argument like this:

mongod --port 27019

Running this command you'll be able to start the mongod on port 27019.

Keep in mind to connect to this instance using mongosh, you'll need to pass the port as well using --port

It also picks up a default location of the db. You can pass in the dbPath to override that as well. Before we do that lets make a directory call mongo-dbs and create db1, db2 and db3 directory inside of it and start 3 different instance of mongo on different ports and dbs separate from each other.

mkdir mongo-dbs
cd mongo-dbs
mkdir db1
mkdir db2
mkdir db3

Now in the first terminal, type in:

mongod --dbpath ./db1 --port 27020

In the second terminal type in:

mongod --dbpath ./db2 --port 27021

In the third terminal, type in:

mongod --dbpath ./db3 --port 27022

Now you have three different instances of mongod running, you can connect to each of them using mongosh --port .

Replication

If you have made it this far in the blog, congratulations. Lets see how the replication works in mongodb and setup a master - slave - slave replication.

If you see the mongod.conf file, you'll see:

#replication:

That means by default the mongod doesn't have any replication setup.

Inorder to setup mongod as replicas, we need to start it in a replication mode and give the replica a name. You can do that by:

mongod --dbpath ./db1 --port 27020 --replSet my_replica

And you can start the other instances as well in the same manner keeping the name my_replica consistent.

Once all the instances are running choose which mongod you want as primary and log on to that instance with mongosh --port .

Now type in the command to initiate the replication:

rs.initiate()

This should give an acknowledgement that the replication has been started. In order check the status of the replicaSet:

rs.status()

It throw out something like this:

{
  set: 'myreplica',
  date: ISODate('2024-01-24T07:07:53.872Z'),
  myState: 1,
  term: Long('6'),
  syncSourceHost: '',
  syncSourceId: -1,
  heartbeatIntervalMillis: Long('2000'),
  majorityVoteCount: 2,
  writeMajorityCount: 2,
  votingMembersCount: 3,
  writableVotingMembersCount: 3,
  optimes: {...},
  lastStableRecoveryTimestamp: Timestamp({ t: 1706080036, i: 1 }),
  electionCandidateMetrics: {...},
  electionParticipantMetrics: {...},
  members: [
    {
      _id: 0,
      name: 'localhost:27020',
      health: 1,
      state: 1,
      stateStr: 'PRIMARY',
      uptime: 2800,
      optime: { ts: Timestamp({ t: 1706080066, i: 1 }), t: Long('6') },
      optimeDate: ISODate('2024-01-24T07:07:46.000Z'),
      lastAppliedWallTime: ISODate('2024-01-24T07:07:46.580Z'),
      lastDurableWallTime: ISODate('2024-01-24T07:07:46.580Z'),
      syncSourceHost: '',
      syncSourceId: -1,
      infoMessage: '',
      electionTime: Timestamp({ t: 1706078026, i: 1 }),
      electionDate: ISODate('2024-01-24T06:33:46.000Z'),
      configVersion: 5,
      configTerm: 6,
      self: true,
      lastHeartbeatMessage: ''
    }
  ],
  ok: 1,
  '$clusterTime': {
    clusterTime: Timestamp({ t: 1706080066, i: 1 }),
    signature: {
      hash: Binary.createFromBase64('AAAAAAAAAAAAAAAAAAAAAAAAAAA=', 0),
      keyId: Long('0')
    }
  },
  operationTime: Timestamp({ t: 1706080066, i: 1 })
}

If everything has worked for you till now, congratulations. Now we will be adding new members to this replicaSet. To add a new member:

rs.add('localhost:27021')

This will also give an acknowledgement that a member has been added. Now if you check the status of the replicaSet:

rs.status()

You have two members marked:

stateStr: 'PRIMARY' and the other as stateStr: 'SECONDARY'

Denoting the primary and the secondary instances. Secondary will also have one key

syncSourceHost: 'localhost:27020'

Denoting which instance it will sync the data from. In the similar fashion, you can add the last instance as a member of this replicaSet and check the status.

Now if you add any data in the primary member it'll automatically be synced to the secondary members.

Production Setup

We have learned a lot of stuff, and it can be very daunting to try this for the first time. Lets expand this knowledge and move from local to production.

We replicate data, to increase our availability, so if somehting happens to one instance other instances are running. Replicating data on same machine would mean if something wrong happens to the machine like power outage etc, the whole database would stop responding to requests.

So in production we have multiple machines, and we have update the mongod.conf file on each of the machine to setup replication.

Things to explore

I want you to explore:

What happens when you shut off a instance?
Suppose we have data in the instances already, and you add a blank member, what will happen.
What happens when a primary member is not working?

Object Oriented Express API

Pankaj Gupta — Fri, 19 Jan 2024 01:51:24 +0000

I've been toying around with the idea of having a scalable express api service, which will handle some of the basic crud operations for a model without writing the actual code for the controller. I've three years of experience and I was not sure if you could something like that but then something happened.

I started to work on Ruby on Rails last year and my perspective has been changed a lot since then, working on different tech stacks, with different people really opens you up. Someone had written a generic handler for all the models and all the basic operations of CRUD were handled without writing a piece of code for the new controller. It was then I realised the power of Object Oriented Programming.

There are a lots of language specific perks of writing code in Ruby, but Object Orientation is a fundamental concept which can be very useful, so I decided to use it in Node and Express and make my first worthwhile contribution to the Open Source Community. I choose Node and Express because I have worked on it before and wanted to see the difference in the quality of code, In other words, I wanted to see how my skills have improved over the years.

I started the project on a Saturday afternoon, I was almost done on Sunday. During the project, I found out some important things:

In Express 5, which is currently a beta version, you don't have to throw the error explicitly, It throws the error itself and call the next() with the error, so no more if this then that in the handlers.
You can use a whole javascript class as a route handler. (AMAZING!!)

It had to dig a bit to find these two solutions. Now with the help of these two solutions my handlers were looking something like this.

var express = require('express');
var router = express.Router();

const Book = require('../models/Book');
const books = require('../caches/book.cache');
const Base = require('./base.controller');

class BookController extends Base {
  constructor(model) {
    super()
    this.model = model;

    // All the routes declared here!
    router.get('/', books.cache, this.index);
    router.get('/:id',books.cache, this.get);
    router.post('/', this.create);
  }
}

new BookController(Book);

module.exports = router;

You can use this in your routes as this:

var booksRouter = require("../controllers/book.controller");

module.exports = function(app) {
  app.use("/books", booksRouter);
};

Isn't this neat and clean? As you can see I've imported the Model, and a cache class I've implemented for this. If you look closely you'll realize how is this working. Thats the best part. I've implemented a base controller which can be extented to every controller and they can use the methods defined in the base controller to implement the basic crud.

The base controller looks something like this:

class Base {
  constructor() {}

  index = async (req, res, next) => {
    const resources = await this.model.find({})
    return res.status(200).json(this.apiSend(resources))
  }

  get = async (req, res, next) => {
    const resource = await this.model.findById(req.params.id)
      .orFail(new NotFoundError());
    return res.status(200).json(this.apiSend(resource))
  }

  create = async (req, res) => {
    const { ...body } = req.body
    const resource = await this.model.create(body)
    return res.status(200).json(this.apiSend(resource))
  }

  apiSend = (json) => {
    return {
      data: json
    }
  }
}

module.exports = Base;

As You can see there is nothing specific to any model or controller. If you have a create method method which needs a bit of manipulations of post changes, You can implement that in your new controller using a service module specific to the controller or the model.

A important point to notice is there is no try catch blocks in any of the handlers, it's the magic of express 5.

If you guys are interested you can check it out here

This project still needs a lot of improvements, but I am happy that it started this way! I'll keep posting further updates.

Querying MongoDB Like an SQL DB Using Aggregation Pipeline

Pankaj Gupta — Mon, 15 Jan 2024 02:15:59 +0000

What Are Aggregations?

Aggregation operations process data records and return computed results. Aggregation operations group values from multiple documents together and can perform a variety of operations on the grouped data to return a single result.

In the db.collection.aggregate method and db.aggregate method, pipeline stages appear in an array. Documents pass through the stages in sequence. We will go through some of the stages to achieve a relational DB like results.

$match (WHERE)

Filters the documents to pass only the documents that match the specified condition(s) to the next pipeline stage.

It has the following prototype:

{ $match: { <query> } }

It is the equivalent of WHERE in SQL queries. Let us take an example to make things clear. This example uses a collection named articles with the following documents:

{ "_id" : ObjectId("512bc95fe835e68f199c8686"), "author" : "dave", "score" : 80, "views" : 100 }
{ "_id" : ObjectId("512bc962e835e68f199c8687"), "author" : "dave", "score" : 85, "views" : 521 }
{ "_id" : ObjectId("55f5a192d4bede9ac365b257"), "author" : "ahn", "score" : 60, "views" : 1000 }
{ "_id" : ObjectId("55f5a192d4bede9ac365b258"), "author" : "li", "score" : 55, "views" : 5000 }
{ "_id" : ObjectId("55f5a1d3d4bede9ac365b259"), "author" : "annT", "score" : 60, "views" : 50 }
{ "_id" : ObjectId("55f5a1d3d4bede9ac365b25a"), "author" : "li", "score" : 94, "views" : 999 }
{ "_id" : ObjectId("55f5a1d3d4bede9ac365b25b"), "author" : "ty", "score" : 95, "views" : 1000 }

Equality match.

db.articles.aggregate(
    [ { $match : { author : "dave" } } ]
);
// Result
{ "_id" : ObjectId("512bc95fe835e68f199c8686"), "author" : "dave", "score" : 80, "views" : 100 }
{ "_id" : ObjectId("512bc962e835e68f199c8687"), "author" : "dave", "score" : 85, "views" : 521 }

We can have multiple constraints inside $match, like $or, $and, etc. according to our requirements, although it has some limitations as well. You can read about it in the Mongo docs.

$skip (OFFSET)

Skips over the specified number of documents that pass into the stage and passes the remaining documents to the next stage in the pipeline. It has the following prototype:

{ $skip: <positive integer> }

In the above example, we were able to match the records related to dave. If we want to skip the few results from the beginning we would write the query as:

db.articles.aggregate([ 
    { $match : { author : "dave" } },
    { $skip: 1 } 
]);
// We are skipping 1 result and we should get just this
{ "_id" : ObjectId("5dc1d22f24a8e913bfcf4f60"), "author" : "dave", "score" : 85, "views" : 521 }

We generally use $skip with $limit to paginate the data, let’s insert few more records into our collection and see how $skip and $limit work together in the next section.

$limit (LIMIT)

Limits the number of documents passed to the next stage in the pipeline. It has the following prototype:

{ $limit: <positive integer> }

We have added new records for dave, let’s see how the collections look with a simple $match:

db.articles.aggregate(
    [ { $match : { author : "dave" } } ]
);
{ "_id" : ObjectId("5dc1d22124a8e913bfcf4f5f"), "author" : "dave", "score" : 80, "views" : 100 }
{ "_id" : ObjectId("5dc1d22f24a8e913bfcf4f60"), "author" : "dave", "score" : 85, "views" : 521 }
{ "_id" : ObjectId("5dc1d53924a8e913bfcf4f65"), "author" : "dave", "score" : 185, "views" : 1521 }
{ "_id" : ObjectId("5dc1d54f24a8e913bfcf4f66"), "author" : "dave", "score" : 15, "views" : 21 }

We have four matching records in the example collection. Suppose you are asked to paginate the results to show two at a time, how would you go about it? Let’s see.

db.articles.aggregate([ 
    { $match : { author : "dave" } },
    { $skip: 0},
    { $limit: 2}
]);
// We are not skipping any records and but limiting the records to 2
{ "_id" : ObjectId("5dc1d22124a8e913bfcf4f5f"), "author" : "dave", "score" : 80, "views" : 100 }
{ "_id" : ObjectId("5dc1d22f24a8e913bfcf4f60"), "author" : "dave", "score" : 85, "views" : 521 }
// We got the first two results, to get the next two results just update the $skip
db.articles.aggregate([ 
    { $match : { author : "dave" } },
    { $skip: 2},
    { $limit: 2}
]);
// This should give two records after skipping the first two.
{ "_id" : ObjectId("5dc1d53924a8e913bfcf4f65"), "author" : "dave", "score" : 185, "views" : 1521 }
{ "_id" : ObjectId("5dc1d54f24a8e913bfcf4f66"), "author" : "dave", "score" : 15, "views" : 21 }

We are doing good so far, but suppose your manager comes up to you and asks you to sort the result by views. What are you going to do? We have $sort for that.

$sort (ORDER BY)

Sorts all input documents and returns them to the https://www.mongodb.com/docs/manual/core/aggregation-pipeline/ in sorted order. It has the following prototype:

{ $sort: { <field1>: <sort order>, <field2>: <sort order> ... } }

Let us use the $sort stage in our pipeline.

1 to specify ascending order
-1 to specify descending order
db.articles.aggregate([ 
    { $match : { author : "dave" } },
    { $sort: { views: 1}}
]);
// Result
{ "_id" : ObjectId("5dc1d54f24a8e913bfcf4f66"), "author" : "dave", "score" : 15, "views" : 21 }
{ "_id" : ObjectId("5dc1d22124a8e913bfcf4f5f"), "author" : "dave", "score" : 80, "views" : 100 }
{ "_id" : ObjectId("5dc1d22f24a8e913bfcf4f60"), "author" : "dave", "score" : 85, "views" : 521 }
{ "_id" : ObjectId("5dc1d53924a8e913bfcf4f65"), "author" : "dave", "score" : 185, "views" : 1521 }

Voilà ! The results are sorted now.

Place the $match as early in the aggregation pipeline as possible. Because $match limits the total number of documents in the aggregation pipeline, earlier $match operations minimize the amount of processing down the pipe.

$group

Groups input documents by the specified _id expression and, for each distinct grouping, outputs a document.

The _id field of each output document contains the unique group by value. The output documents can also contain computed fields that hold the values of an accumulator expression. It has the following prototype:

{
  $group:
    {
      _id: <expression>, // Group By Expression
      <field1>: { <accumulator1> : <expression1> },
      ...
    }
}

Suppose we want to group the articles by author, in other words, the number of articles by each author, we can make use of the group stage in the pipeline. So, let’s see it live:

db.articles.aggregate([ 
    { $group : { _id: "$author", count: { $sum: 1 }}},
    { $sort: { count: 1 }}
]);
// We have grouped the articles by author ann getting the count and sorting it by count
{ "_id" : "annT", "count" : 1 }
{ "_id" : "ahn", "count" : 1 }
{ "_id" : "li", "count" : 2 }
{ "_id" : "dave", "count" : 4 }
// We can have more constraints like if we want only the results whose count is greater than 1, then we can add a $match stage in the pipeline after $group
db.articles.aggregate([ 
    { $group : { _id: "$author", count: { $sum: 1 }}},
    { $sort: { count: 1 }},
    { $match: { count : { $gt: 1 }}}
]);
{ "_id" : "li", "count" : 2 }
{ "_id" : "dave", "count" : 4 }

Let us take this grouping up a notch. Suppose we want to group by values stored in an array structure. We have something called $unwind. Let’s see how it works.

$unwind

Deconstructs an array field from the input documents to output a document for each element. Each output document is the input document with the value of the array field replaced by the element.

You can pass the array field path to $unwind. When using this syntax, $unwind does not output a document if the field value is null, missing, or an empty array. It has the following prototype:

{ $unwind: <field path> }

Let us take a new collection inventory and a new record to it with the following command:

db.inventory.insertOne({ "_id" : 1, "item" : "ABC1", sizes: [ "S", "M", "L"] })

That’s the beauty of MongoDB, you can create a new collection and each document is identical to the input document, except for the value of the sizes field which now holds a value from the original sizes array. Add a record to it without any setup.

Let us $unwind this by the sizes.

db.inventory.aggregate( [ { $unwind : "$sizes" } ] )
// Result
{ "_id" : 1, "item" : "ABC1", "sizes" : "S" }
{ "_id" : 1, "item" : "ABC1", "sizes" : "M" }
{ "_id" : 1, "item" : "ABC1", "sizes" : "L" }

Each document is identical to the input document except for the value of the sizes field which now holds a value from the original sizes array.

Let us take a new collection inventory2 and do a group by the size, use this command to insert more records:

db.inventory2.insertMany([
  { "_id" : 1, "item" : "ABC", price: NumberDecimal("80"), "sizes": [ "S", "M", "L"] },
  { "_id" : 2, "item" : "EFG", price: NumberDecimal("120"), "sizes" : [ ] },
  { "_id" : 3, "item" : "IJK", price: NumberDecimal("160"), "sizes": "M" },
  { "_id" : 4, "item" : "LMN" , price: NumberDecimal("10") },
  { "_id" : 5, "item" : "XYZ", price: NumberDecimal("5.75"), "sizes" : null }
])

If we unwind this, we would get something like this:

db.inventory2.aggregate( [ { $unwind: "$sizes" } ] )
// Results
{ "_id" : 1, "item" : "ABC", "price" : NumberDecimal("80"), "sizes" : "S" }
{ "_id" : 1, "item" : "ABC", "price" : NumberDecimal("80"), "sizes" : "M" }
{ "_id" : 1, "item" : "ABC", "price" : NumberDecimal("80"), "sizes" : "L" }
{ "_id" : 3, "item" : "IJK", "price" : NumberDecimal("160"), "sizes" : "M" }
// Notice it ignores the null and undefined values
db.articles.aggregate([ 
    { $unwind: { path: "$sizes" } },
    { $group: { _id: "$sizes", count: { $sum: 1 }}}
]);
// Results
{ "_id" : "M", "count" : 2 }
{ "_id" : "L", "count" : 1 }
{ "_id" : "S", "count" : 1 }

We can apply different stages to this like $match, $sort, $skip, $limit, etc. to get the desired results.

Now, let’s move on to SQL JOINS, to achieve joins in MongoDB we have $lookup.

$lookup

New in version 3.2.

Performs a left outer join to an unsharded collection in the same database to filter in documents from the “joined” collection for processing.

To each input document, the $lookup stage adds a new array field whose elements are the matching documents from the “joined” collection. The $lookup stage passes these reshaped documents to the next stage.

There can be different join conditions but we will be looking into the most basic one, which is an equality match.

Equality match

To perform uncorrelated subqueries between two collections as well as allow other join conditions besides a single equality match. The $lookup stage has the following syntax:

{
   $lookup:
     {
       from: <collection to join>,
       let: { <var_1>: <expression>, …, <var_n>: <expression> },
       pipeline: [ <pipeline to execute on the collection to join> ],
       as: <output array field>
     }
}
from: Specifies the collection in the same database to perform the join with.
let: Optional. Specifies variables to use in the pipeline field stages. Use the variable expressions to access the fields from the documents input to the $lookup stage.
pipeline: Specifies the pipeline to run on the joined collection. The pipeline determines the resulting documents from the joined collection. To return all documents, specify an empty pipeline [].
as: Specifies the name of the new array field to add to the input documents. The new array field contains the matching documents from the from collection. If the specified name already exists in the input document, the existing field is overwritten.

Let us look at some examples to better understand the terminologies:

Perform a single equality join with $lookup

Create a collection orders with the following documents:

db.orders.insert([
   { "_id" : 1, "item" : "almonds", "price" : 12, "quantity" : 2 },
   { "_id" : 2, "item" : "pecans", "price" : 20, "quantity" : 1 },
   { "_id" : 3  }
])

Create another collection inventory with the following documents:

db.inventory.insert([
   { "_id" : 1, "sku" : "almonds", description: "product 1", "instock" : 120 },
   { "_id" : 2, "sku" : "bread", description: "product 2", "instock" : 80 },
   { "_id" : 3, "sku" : "cashews", description: "product 3", "instock" : 60 },
   { "_id" : 4, "sku" : "pecans", description: "product 4", "instock" : 70 },
   { "_id" : 5, "sku": null, description: "Incomplete" },
   { "_id" : 6 }
])

The following aggregation operation on the orders collection joins the documents from orders with the documents from the inventory collection using the fields item from the orders collection and the sku field from the inventory collection:

db.orders.aggregate([
   {
     $lookup:
       {
         from: "inventory",
         localField: "item",
         foreignField: "sku",
         as: "inventory_docs"
       }
  }
]);

The operation returns the following documents:

{
   "_id" : 1,
   "item" : "almonds",
   "price" : 12,
   "quantity" : 2,
   "inventory_docs" : [
      { "_id" : 1, "sku" : "almonds", "description" : "product 1", "instock" : 120 }
   ]
}
{
   "_id" : 2,
   "item" : "pecans",
   "price" : 20,
   "quantity" : 1,
   "inventory_docs" : [
      { "_id" : 4, "sku" : "pecans", "description" : "product 4", "instock" : 70 }
   ]
}
{
   "_id" : 3,
   "inventory_docs" : [
      { "_id" : 5, "sku" : null, "description" : "Incomplete" },
      { "_id" : 6 }
   ]
}

Conclusion

This was just a basic overview of using SQL-like queries in MongoDB.

There is a lot more that can be done using many other stages that are available. The best way to have a good grasp of it is by practicing different scenarios and using them in your projects. I hope this will help.

Resources

MongoDB Documentation

Understanding the Efficiency of vim

Pankaj Gupta — Fri, 05 Jan 2024 07:13:13 +0000

Highlights of vim

Ubiquitous — Shipped with every NIX Platform.
Highly customizable — Can be tailored to your needs.
Lightning-fast.
Insanely efficient.

Basics

Before we start learning how vim works, I just want to mention some commands so that people who are new to vim can use it.

# Open a file in vim
vim textfile.text
# Come out of vim
[Esc]:q # quit without saving
[Esc]:wq # save and quit
[Esc]:q! # discard changes and quit
# Navigation
h => left
j => down
k => up
l => right
# Copy
[Esc]v[h|j|k|l]y # h,j,k,l to highlight a block
# Cut
[Esc]v[h|j|k|l]d # h,j,k,l to highlight a block
# Paste
[Esc]p

Why vim

What distinguishes vim from other editors? It’s a modal editor that makes it so efficient. Unlike any other editors, that only have one mode which is insertion mode, you are constantly typing and streaming text into the buffer.

Now, for writing this is great, however, as programmers, what we care about most isn’t constant flow of text but editing and navigating. I think the best analogy to explain this would be:

You can imagine programmers as painters and painters don’t spend their time continuously doing one brush stroke, they prepare the canvas, mix up the colors, and then they do individual discrete strokes.

Vim achieves this with different modes. Here are the four modes to get started with vim.

1. Normal mode

Here, we interact with text and tell it what to do. When we launch vim, this is the default mode.

Most people are used to just typing, but in this mode, we don’t insert text into the buffer, we do other types of activities like navigation. We can get to this mode by pressing Esc. This is the default mode in vim.

2. Insert mode

You can get into this mode using i: insert, a: append, c: change. This is the mode most people are used to. So, remember, if you open vim and want to type something, switch to the insert mode using the mentioned keystrokes.

3. Visual mode

This is just like normal mode, except it works with highlighted blocks of texts. Another problem people face is copying and cutting text, you can do that in this mode. You can get here from normal mode using V, v, .

4. Command-line mode

This is where you find texts, exit vim, navigate files, and whatnot.

Operators, Text Objects, and Motions

To use vim, you have to think in terms of Operators, text objects, and motions, then you will unleash the real power of vim.

1. Operators

These are the verbs of the vim language. They specify actions to perform on your text objects or motions and most of them are easy to remember.

2. Text objects

These are the actual visual text on your screen.

Again, very simple to remember: aw: a word. Now, here we can combine these with operators to see some powerful actions. Here are a few of my favorites:

3. Motions

Motions are all about efficient navigation through the file that you’ve opened. Here are some of the most useful motions:

Let me explain how this works, suppose you press +, this would take you to the first non-blank character of the next line, but if you press 10+, it’ll take you to the first non-blank character of the 10th line from the current position.

Let me give you some more examples to make things clear.

Putting It All Together

[count][operator][text objects/ motions]

This is the template for using all the vim commands. Your imagination is your limit. Here is a list of commands you can look at.

Here is the list of commands that I use the most:

Conclusion

This guide is just the beginning into the world of vim, but this guide here provides a basic understanding of how vim works, which is enough to get you excited about vim.

Resources

YouTube

3 Terminal Commands to Increase Your Productivity

Pankaj Gupta — Sun, 31 Dec 2023 01:44:37 +0000

Here are a few important shortcuts that help me be more productive throughout the day at work:

Creating aliases for commands.
Using pbcopy.
Using reverse search in the terminal.
Bonus tricks and tips.

Alias for Commands

Alias can be one of the most powerful tools in our hands, it provides us with the power to write our own shortcuts. Let’s see with an example what I mean.



alias dev="cd ~/Project/development"

Whenever I type dev and hit enter it’ll run this command. This becomes very useful when navigating to different folders. We can run almost all the commands in the alias. Here is the list of some of my most-used commands:



alias ..="cd .."
alias gs="git status"
alias gp="git pull"
alias gb="git branch"
alias ga="git add ."

So, using these aliases, I save a lot of time during the day, to up a directory I type, instead of cd… I think the rest of them are self-explanatory. We can also use arguments with $1, $2, etc. for more extendability, like in this example:



alias gc="git commit -m $1"

Now, all I have to type is gc “Commit message” and it’ll commit my changes with the provided message. All you have to do is find your most-used commands and try to make them shorter using an alias to make yourself more productive.

Now that we know what the aliases can do, let us see how we can set them. There are two ways in which we can achieve this, the first is temporary and can be set by running the command:



alias dev="cd ~/Project/development"

This will work until the session is closed. The other way is to set these aliases permanently. For that, we need to set it up in our shell, I use Zsh, so I’ll be updating my ~/.zshrc file.

If you are using Bash, use the ~/.bashrc file. Add the commands to the file and your file should look like this:

After making changes to this file, you need to run the command:



source ~/.zshrc

Then, all your aliases will be available for your use.

pbcopy

This command is available on Mac and if you want to use it on Linux distributions, you can follow this guide.

pbcopy is copy on steroids. You can use this command to copy the contents of a file to the clipboard. Let me give you an example. Suppose you have to copy your SSH identity to the clipboard, you can do it with this command:



pbcopy < ~/.ssh/id_rsa.pub

You can save your other passwords in different files and make use of this while logging in.

Suppose you are using and accessing a remote server, and you have to provide the password, instead of opening the file, you can just pbcopy it to your clipboard and without all the hassle of opening and closing files, you’ll have the passwords.

It becomes more useful when it is piped with other commands like grep. It’ll copy the grepped results to the clipboard. Let’s see with an example:



grep "<keyword>" | pbcopy

I use this command when I am debugging the log files, and I provide a keyword to be searched for, like a timestamp, and all the lines are copied onto my clipboard.

I can paste it in a file to see the required logs instead of the whole file. It can be even more useful if you use the tee command as a pipe to grep, it’ll write the results to a file.

It has the following syntax:



grep "<keyword>" | tee myfile.txt

Using Reverse Search

Reverse search is one of the coolest things available on the Unix system.

Suppose you forget the full command and you only remember some parts, what you can do is you can go to the reverse search and type the words that you remember. Let’s see it with an example.

I have to restart my server running in the staging environment, I only remember the staging keyword and forgot the rest of the command. So, I type ctrl + rto go into reverse-search mode and type:



(reverse-i-search)`stag': cd /home/ubuntu/server; pm2 stop app.js && export NODE_ENV="staging" && pm2 start app.js && pm2 logs
```

It’ll remember the commands previously entered and finds the right match that you are looking for.

---

## Bonus Tricks and Tips

### cal
It prints the current month on the terminal. It has many different options available, which can be checked using man cal.


![Calendar View](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/ronihzsjagme3dp5fifp.png)

### Encrypting a file using vim

You can encrypt a file using vim, just type :X. It’ll ask you to set a password which will look like this:


![Encrypting a file](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/4uunprvj2jm2do8azxe1.png)

When you access this file again, it’ll ask you for the password.


![Opening an encrypted file](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/ke8oweb76p36djzxf0kz.png)

---

## Conclusion

Please do mention your most used commands and add to the list.

DEV Community: Pankaj Gupta

Consistency >>> Talent

Conclusion

Including Polymorphic Association In Rails To Avoid N+1 Queries

Introduction

Problem Statement

Solution

Conclusion

How To Setup MongoDB Like a Pro

Introduction

Installing MongoDB(community edition - 6.0)

Getting comfortable with mongod and mongosh

Starting the service

Stopping the service

Restarting the service

Connecting to a mongod instance with mongosh

Running mongod in foreground

storage.dbPath

net.port

Starting mongosh with different arguments

Replication

Production Setup

Things to explore

Object Oriented Express API

Querying MongoDB Like an SQL DB Using Aggregation Pipeline

What Are Aggregations?

$match (WHERE)

$skip (OFFSET)

$limit (LIMIT)

$sort (ORDER BY)

$group

$unwind

$lookup

Equality match

Perform a single equality join with $lookup

Conclusion

Resources

Understanding the Efficiency of vim

Highlights of vim

Basics

Why vim

1. Normal mode

2. Insert mode

3. Visual mode

4. Command-line mode

Operators, Text Objects, and Motions

1. Operators

2. Text objects

3. Motions

Putting It All Together

Conclusion

Resources

3 Terminal Commands to Increase Your Productivity

Alias for Commands

pbcopy

Using Reverse Search

Getting comfortable with `mongod` and `mongosh`

Connecting to a `mongod` instance with `mongosh`

Running `mongod` in foreground

Starting `mongosh` with different arguments