DEV Community: RAM PANDEY

Breaking Down AI Buzzwords: A Developer’s Guide to Understanding the Basics

RAM PANDEY — Wed, 09 Apr 2025 06:29:48 +0000

Hey there, developers!

AI is absolutely everywhere these days, right? And let's be honest, so is the jargon: tokenization, embeddings, self-attention... it's enough to make your head spin! If you've ever found yourself nodding along in a meeting while secretly wondering what these terms really mean, trust me, you're not alone. I've been there too.

That's why I wrote this post. My goal is to break down some of the most common AI buzzwords you'll encounter, especially in Large Language Models (LLMs), using plain language. Forget the dense academic papers and overwhelming math for now – we're focusing on simple explanations to help you grasp how these concepts work and why they actually matter in the models we use.

Ready to make sense of the AI hype together? Let's dive in!

Tokenization: Chopping Up Language

First things first: before any AI model can even begin to understand text, it needs to break it down into smaller, manageable pieces. This crucial first step is called tokenization.

What's it really doing?

Think of it like slicing a loaf of bread before you can make sandwiches. You take a whole sentence (the loaf) and chop it into smaller bits – words or even parts of words (the slices). These pieces are called "tokens."

For example, the sentence "Hello, world!" might get tokenized into something like:
[Hello] [ , ] [world] [ ! ]

Why bother? Because computers don't speak English, French, or Hindi – they speak the language of numbers. Tokenization is the bridge. Each token gets assigned a numerical ID, turning language into a format the machine can actually perform calculations on.

A Simple Analogy: I found it helpful to think of reading a complex recipe. You don't tackle it all at once; you break it down ingredient by ingredient, step by step. Tokenization does this for language, making it digestible for the AI.

Key Things to Remember about Tokenization:

Vocab Size Matters: Every model has a pre-defined "vocabulary" – a list of all the unique tokens it recognizes. This might include letters (A-Z, a-z), numbers (0-9), punctuation, common words, or subwords. If a model's vocab size is, say, 50,000, it knows 50,000 unique pieces it can represent numerically.
Different Models, Different Strategies: Some models tokenize purely by words ([Hello], [world]). Others use "subword" tokenization, which is really clever. It can break down unknown or long words into smaller known parts, like tokenizing "Unbelievable" into [Un] [believ] [able]. This helps handle new words without needing an infinitely large vocabulary.

Why should you care?

Tokenization is the absolute foundation. Without it, text remains meaningless squiggles to an AI. It's the essential first step in turning human language into something a machine can process and learn from.

Vector Embeddings: Giving Words Mathematical Meaning

So, we've chopped our text into tokens and given each a number. But those numbers are just IDs – they don't capture the meaning or context of the token. That's where the magic of vector embeddings comes in.

What are they, exactly?

Imagine giving each word (or token) a rich, multi-faceted mathematical identity. Instead of just one ID number, each token gets mapped to a vector – basically, a list of numbers (often hundreds of them!). This vector isn't random; it's carefully calculated to represent the token's meaning and its relationships to other tokens.

Why vectors? Because while machines don't understand "king" or "queen," they excel at comparing lists of numbers. Embeddings translate semantic meaning into a format computers can work with.

Think of it like this: You're plotting concepts on a complex, multi-dimensional map. Each concept (token) gets coordinates (its vector). Crucially, concepts with similar meanings ("king," "queen," "monarch") end up close together on this map. This proximity represents semantic similarity.

How do they capture meaning?

Tokens to Vectors: Every token from the vocabulary gets its own unique vector. For instance (simplified!):
- "King" → [1.2, 3.4, -0.8, ...]
- "Queen" → [1.1, 3.5, -0.7, ...]
Semantic Relationships: These vectors are learned in a way that captures relationships. A famous example is that the vector math vector("King") - vector("Man") + vector("Woman") often results in a vector very close to vector("Queen"). This shows the model has learned gender and royalty relationships!
Finding Similarities: Because similar concepts cluster together, the model can find related ideas just by looking for vectors that are "close" in this mathematical space. "Cat" will be nearer to "dog" than to "car."

Why are embeddings so important?

Vector embeddings are critical for AI to:

Grasp relationships between words (synonyms, analogies, concepts).
Power recommendation systems (finding "similar" items).
Understand context and nuance far beyond simple keyword matching.

Without embeddings, AI would struggle to understand the rich tapestry of meaning woven into human language. They are the secret sauce for capturing semantics.

Positional Encoding: Keeping Track of Word Order

Okay, we have meaningful vectors for our tokens. But there's a big problem, especially for newer models like Transformers: they look at all the tokens in a sentence simultaneously. This is great for speed, but it means they might lose track of the original word order.

Consider "The cat chased the dog" vs. "The dog chased the cat." Same tokens, same embeddings, but completely different meanings! How do we solve this? Enter positional encoding.

What is it?

Positional encoding is a clever trick to inject information about a token's position in the sequence directly into its embedding. It modifies the token's vector slightly, giving the model a signal about where that token appeared (first, second, third, etc.).

Analogy Time: Imagine numbering the books on your shelf. Even if you pull them all off, the numbers tell you how to put them back in the correct order. Positional encoding gives each token's vector a unique "position number" or signal.

How does it work (conceptually)?

Adding Positional Info: A unique positional value (itself a vector, often generated using sine and cosine functions of different frequencies) is added to each token's embedding vector.
- Token Embedding + Positional Vector = Combined Embedding
Unique Signal for Each Position: The mathematical functions used ensure that each position (1st, 2nd, 3rd...) gets a distinct positional vector pattern that the model can learn to recognize.
Preserving Order: Now, even though the model processes tokens in parallel, the combined embeddings contain the necessary clues to understand the original sequence and differentiate "cat chased dog" from "dog chased cat."

Why is this necessary?

Older models like RNNs processed words one by one, inherently keeping track of order. Transformers, by processing tokens in parallel for speed and capturing long-range dependencies better, lose this built-in sequential awareness. Positional encoding adds it back explicitly. It's like giving the Transformer a GPS for the sentence structure. Without it, meaning derived from word order would be lost.

Self-Attention: Understanding Context Within a Sentence

How do you understand the meaning of "bank" in "The bank is near the river" versus "I need to go to the bank to deposit this check"? You use context – the surrounding words. Self-attention is the mechanism that allows AI models, particularly Transformers, to do the same thing.

What is Self-Attention?

It's a process that lets the model weigh the importance of all other tokens in the input sequence when processing a specific token. Instead of looking at words in isolation, the model "attends" to the relationships between them to better understand the context-dependent meaning.

A Human Parallel: Think of a group discussion. When someone says a potentially ambiguous word like "address," you pay attention to the rest of their sentence ("Do you need my home address?" vs. "How should I address this issue?") to get the real meaning. Self-attention lets the model do this internal "listening" across the sentence.

How does it work (the gist)?

Token Interactions: For every single token, the model calculates an "attention score" between it and every other token (including itself) in the sequence. This score represents how relevant token B is for understanding token A in this specific context.
Calculating Weighted Importance: High scores mean strong relevance, low scores mean weak relevance. For "The bank is near the river," the token "river" would likely get a high attention score when the model processes "bank."
Refining Embeddings: The model then recalculates each token's embedding by creating a weighted sum of all tokens' embeddings, guided by these attention scores. This means the final representation of "bank" is heavily influenced by the representation of "river," helping it resolve ambiguity.

Why is this a game-changer?

Language is slippery! Words change meaning based on context. Self-attention gives models the power to:

Disambiguate words: Understand "bank" (river) vs. "bank" (financial).
Understand relationships: Figure out what pronouns like "it" or "they" refer to earlier in the text.
Grasp the holistic meaning of a sentence rather than just processing word-by-word.

It's fundamental to how modern AI truly understands language nuances.

Multi-Head Attention: Looking at Context from Multiple Angles

Self-attention is powerful, letting tokens consider context. But what if one perspective isn't enough? What if different types of relationships are important? That's the idea behind multi-head attention.

What is it?

Instead of calculating attention just once, multi-head attention performs the self-attention process multiple times in parallel, using different, learned "perspectives" or "heads." Each head can potentially focus on different aspects of the relationships between tokens.

Analogy Upgrade: Remember the group meeting analogy for self-attention? Multi-head attention is like having multiple subgroups in that meeting, each analyzing the project from a specific angle (budget, timeline, resources, technical feasibility). Afterwards, they combine their findings for a much richer, comprehensive understanding.

How does it enhance self-attention?

Splitting and Projecting: The original token embeddings are split and transformed (projected) differently for each "head."
Parallel Attention: Each head performs the self-attention calculation independently on its transformed version of the embeddings. This allows different heads to learn to focus on different things:
- One head might capture subject-verb relationships.
- Another might focus on how adjectives modify nouns.
- A third might track long-distance dependencies.
Combining the Insights: The outputs from all the attention heads are combined (concatenated and projected again) to produce the final output for that layer. This final representation benefits from the diverse perspectives captured by the individual heads.

Why bother with multiple heads?

Language is complex! A single attention mechanism might latch onto the most obvious relationship but miss others. Multi-head attention allows the model to simultaneously consider various types of syntactic and semantic relationships, leading to a deeper and more robust understanding of the input text. It's a key ingredient in the success of powerful models like Transformers for tasks demanding nuanced understanding (translation, summarization, etc.).

Transformers: The Architecture Powering Modern AI

You've likely heard of ChatGPT, BERT, Google Translate, DALL·E... These incredible AI systems often have one thing in common under the hood: the Transformer architecture. Introduced in the seminal paper "Attention Is All You Need," Transformers revolutionized how we process sequential data, especially language.

What is a Transformer?

It's a specific type of neural network architecture designed primarily for sequence-to-sequence tasks (like translation or summarization). Its defining characteristics are:

Parallel Processing: Unlike older models (RNNs/LSTMs) that process sequences word-by-word, Transformers can process all tokens in a sequence simultaneously.
Attention Mechanisms: They heavily rely on self-attention and multi-head attention (which we just discussed!) to understand context and relationships between tokens, regardless of their distance in the sequence.
Positional Encoding: They use positional encodings to keep track of word order, compensating for the parallel processing.
The Big Picture: Think of a Transformer as a highly efficient information processing pipeline, built specifically to leverage attention mechanisms for understanding complex dependencies in data like text.

Core Components Often Include:

Input Processing: Tokenization, embedding generation, and adding positional encoding.
Encoder: A stack of layers (often including multi-head attention and feed-forward networks) that processes the input sequence and builds a rich, context-aware representation.
Decoder: Another stack of layers that takes the encoder's output and generates the target sequence (e.g., the translated sentence), also using attention mechanisms. (More on this next!)

Why are Transformers so dominant?

Efficiency & Speed: Parallel processing makes them much faster to train on modern hardware (GPUs/TPUs) compared to sequential RNNs.
Superior Context Handling: Self-attention excels at capturing long-range dependencies (connecting words far apart in a text) which was a struggle for older models.
Scalability: They scale remarkably well with more data and larger model sizes, leading to increasingly powerful AI capabilities.

Transformers are the backbone of much of the current AI revolution in natural language processing and beyond.

Encoder and Decoder: The Two Halves of Many Transformers

Many Transformer models, especially those used for tasks like machine translation or summarization, consist of two main parts working in tandem: the encoder and the decoder.

What do they do?

The Encoder: Its job is to understand the input sequence. It reads the entire input sentence (e.g., English text), uses layers of self-attention to build a rich internal representation that captures the meaning and context, and outputs this representation. Think of it as creating a detailed summary or "meaning vector" of the input.
The Decoder: Its job is to generate the output sequence (e.g., the French translation). It takes the encoder's output (the meaning vector) and, typically one token at a time, generates the target sentence. Crucially, the decoder also uses attention mechanisms:
- Self-attention on the words it has already generated, to maintain coherence.
- Cross-attention looking back at the encoder's output, to ensure the generated output stays true to the original input's meaning.

A Communication Analogy: Imagine translating a message. The encoder is like carefully reading and fully understanding the original message in English. The decoder is then taking that deep understanding and carefully crafting the message word-by-word in French, constantly referring back to the original meaning.

Why the split?

This separation of concerns is powerful:

The encoder can focus entirely on deeply understanding the source material.
The decoder can focus entirely on generating fluent and accurate target output, guided by the encoder's understanding.

Note: Not all Transformers have both. Models focused purely on understanding (like BERT) might primarily use the encoder part, while models focused purely on generation (like GPT) might primarily use the decoder part, often pre-trained without a specific encoder input for generation tasks.

Softmax: Making the Final Choice

When an AI model needs to predict the next word in a sentence or classify something, it usually ends up with a bunch of raw scores (called "logits") for each possible option. But how does it turn those scores into a single, confident choice? That's often the job of the softmax function.

What does Softmax do?

Softmax is a mathematical function that takes a list of raw scores and converts them into a list of probabilities that all add up to 1 (or 100%). It essentially highlights the most likely option while still giving some probability to less likely ones.

For example, predicting the next word after "How are you...":

Raw scores (logits): {"doing": 3.1, "feeling": 2.5, "today": 1.0}
Softmax output (probabilities): {"doing": 0.65, "feeling": 0.30, "today": 0.05} (approx.)

The model would then typically choose "doing" as the next word because it has the highest probability (65%).

Voting Analogy: Think of it like vote counts in an election. The raw scores are the initial tallies. Softmax converts these into percentages of the total vote, making it clear who the winner is, but also showing the support for other candidates.

The Key Steps (Simplified):

Exponentiate: It makes all scores positive and emphasizes the higher scores (e^score).
Normalize: It divides each exponentiated score by the sum of all exponentiated scores, ensuring the results sum to 1.

Why is Softmax used?

Interpretability: Turns arbitrary scores into meaningful probabilities.
Decision Making: Provides a clear basis for choosing the most likely output.
Training: It's mathematically convenient for training models using methods like cross-entropy loss.

Softmax is the standard way models often convert internal calculations into final predictions or choices.

Temperature: Dialing Up the AI's Creativity

When using generative AI, you might notice you can sometimes adjust a setting called temperature. This parameter directly influences how the model uses the probabilities generated by softmax (or a similar process) when picking the next word. It controls the randomness or "creativity" of the output.

What is Temperature?

Temperature is a scaling factor applied to the logits before they go into the softmax function.

Low Temperature (e.g., < 1.0, like 0.2): Dividing logits by a small number makes the differences between scores larger. This makes the softmax probability distribution sharper, meaning the model becomes more confident, more focused, and more likely to pick the absolute highest-probability word. The output becomes predictable, deterministic, and sometimes repetitive.
High Temperature (e.g., > 1.0, like 1.2): Dividing logits by a larger number makes the differences between scores smaller. This flattens the softmax distribution, meaning probabilities are spread more evenly across different words. The model becomes less confident, more adventurous, more random, and more likely to pick less common words. The output becomes more surprising, diverse, and creative – but also potentially less coherent or relevant.
Temperature = 1.0: This is the default, using the original softmax probabilities without modification.
Creativity Thermostat Analogy: Low temp is like sticking strictly to the recipe – safe and predictable. High temp is like improvising in the kitchen – more experimental, potentially brilliant, potentially disastrous!

Why control Temperature?

It allows you to tailor the AI's output style:

Use low temperature for tasks needing accuracy and consistency (e.g., factual summarization, code generation based on strict rules).
Use high temperature for tasks benefiting from creativity and diversity (e.g., brainstorming ideas, writing poetry, creating varied chatbot responses).

Temperature gives you a knob to turn, balancing between predictable coherence and creative exploration.

Vocab Size: The AI's Word Dictionary

We mentioned vocabularies back when discussing tokenization. The vocab size is simply the total number of unique tokens defined in that vocabulary. This number has significant implications for the model's performance and efficiency.

What determines Vocab Size?

It's determined during the tokenization setup phase, based on the training data and the chosen tokenization strategy (word, subword, character).

A model using character-level tokenization might have a tiny vocab size (e.g., ~100 for English letters, numbers, punctuation).
A model using word-level tokenization needs a huge vocab size (tens or hundreds of thousands) to cover most words.
Subword tokenization (like BPE or WordPiece used in many modern LLMs) offers a balance, typically having vocab sizes in the tens of thousands (e.g., 30k - 100k). It can represent common words as single tokens but break down rarer words into known subword pieces.
Dictionary Analogy: Vocab size is like the number of entries in the dictionary the AI carries. More entries mean it recognizes more words directly, but the dictionary gets heavier.

The Trade-offs:

Small Vocab (e.g., Characters):
- Pros: Can represent any text, handles typos/new words gracefully. Smaller model embedding layer.
- Cons: Sentences become very long sequences of tokens, potentially harder for the model to capture long-range meaning. Less semantically meaningful units.
Large Vocab (e.g., Words):
- Pros: Shorter sequences, tokens are semantically meaningful.
- Cons: Struggles with unknown words ("out-of-vocabulary" problem). Requires huge memory for the embedding layer.
Medium Vocab (e.g., Subwords):
- Pros: Good balance. Handles rare words by breaking them down. Keeps sequences reasonably short. Most common approach now.
- Cons: Tokenization itself is more complex. Tokens might not always align with intuitive word boundaries.

Why does it matter?

Vocab size directly impacts the model's size, speed, memory usage, and its ability to handle diverse text. It's a fundamental design choice in building an AI model.

Knowledge Cutoff: The AI's "Last Studied" Date

Ever asked an AI about a very recent event and gotten a vague answer or information that's clearly outdated? This happens because of the knowledge cutoff.

What is it?

The knowledge cutoff is simply the point in time when the data used to train the AI model was collected. The model generally cannot "know" about events, discoveries, or information that emerged after this date because it wasn't part of its learning material.

Exam Analogy: Imagine studying for a history exam using only textbooks published up to the year 2020. Your knowledge would be "cut off" at 2020; you wouldn't know about major events from 2021 unless you got an updated textbook (or the model is retrained/updated).

Why does it exist?

Training is Finite: Training these massive models takes enormous resources (data, computation, time). The process has to stop at some point to actually release and use the model.
Static Models: Once trained, most large models don't continuously learn from the live internet (though some systems might integrate web search results post-generation). Their core knowledge remains fixed until a significant retraining effort occurs.

What's the Impact?

Outdated Info: Be critical! AI responses about rapidly evolving topics (current events, latest tech, recent pop culture) might be inaccurate if the event happened after the cutoff.
Reliability Varies: For stable knowledge (math principles, historical facts before the cutoff, core programming concepts), the cutoff is less relevant. For dynamic knowledge, it's crucial to be aware of.

Always check or ask for the model's knowledge cutoff date if you need up-to-the-minute information!

Wrapping Up

Phew, that was a lot! But hopefully, terms like tokenization, embeddings, attention, and transformers feel a bit less mysterious now. Understanding these core concepts really helps pull back the curtain on how modern AI, especially LLMs, actually works its magic with language.

These aren't the only terms you'll hear, but they form a solid foundation. Keep learning, keep experimenting, and don't be afraid to ask questions! AI is constantly evolving, and being curious is the best way to keep up.

What other AI terms have you found confusing? Let me know in the comments!

Advice for Intermediate developers

RAM PANDEY — Sun, 09 Jun 2024 15:50:55 +0000

Prologue

I wrote this blog five years ago when I was a junior developer. The tips I shared back then are still rules I follow today and have become an integral part of me. I've grown a lot as a developer, so now I want to give back to the community as an intermediate developer.
The advice mentioned here is for people who love their craft and want to get better at it, not for the sake of better compensation but for the joy of programming.

1) Love your job

I've seen people treat programming as just a job, doing it only for the money. They program to earn a living and go about their daily lives. This lifestyle is fine, its your choice. But don't be surprised if your skills don't improve and you become stagnant. To become great at programming, you have to love your job. You spend most of your day programming at your day job, and if you don't love it, you won't take the initiative to improve your skills while working.

I have a personal story to share. I once worked at a company I hated. I didn't take any initiative to improve the codebase or learn new things to enhance the application architecture. Now, I work at a job I love and treat it as my own product. This often leads me to learn new things and develop the codebase in a well-structured manner because I don't want to ruin it. If you do what you don't love, you'll do more harm than good. You can learn after work, but you would have wasted around six hours of your day and accomplished very little.

2) Be a generalist

Never put yourself in a box. Don’t think of yourself as just a frontend developer or backend developer. Think of yourself as a software developer. Great developers don't limit themselves to specific technologies they focus on solving problems, not just parts of a problem. If you limit yourself to a certain stack, you won't become a great problem solver. Software development is all about problem-solving, and if you don’t understand how to build an end-to-end product, you won’t be a good problem solver.

At the start of your career, you might have to choose a specific stack to prove yourself as a great software developer. But don't let that limit you. If you work at a good company, talk with a senior or other developers to gain insights into different teams and learn new things. Start taking responsibility for other parts of your company's codebase to transition into a more full-stack developer role. This way, you'll start thinking more about solving whole problems rather than just parts. If you are not welcomed to work with other stacks I would recommend working at another job. A company should never limit the learning of their engineers.

So, be a generalist. Don’t limit yourself to one part of the stack. Learn to solve problems as a software developer. Generalists find it easier to be good at solving specific problems because they can pick up new technologies faster since they already have a broad understanding.

3) Never stop learning new tech ( Be a tinkerer )

This is a crucial point that many developers overlook. To be a good problem solver, you must keep yourself up to date with the latest advancements in your technology. I find a lot of joy in my hobby projects, which help me develop many skills. When you tinker with new stuff, you learn a lot, and you never know when it will become useful.

For example, imagine you've been tasked with creating a blogging application for your company. They want a custom solution, not something that uses Webflow and other similar services. If you've kept up with the latest advancements, you can use modern CMS tools like Supabase or Pocketbase to develop the backend quickly. It might take just 30 minutes to set up a CMS for your blogging site, saving you from creating and managing the database and backend code. Then you can focus on the frontend according to your company's needs.

Here's a personal example: I’ve been learning Go for a month on the side. Recently, I had to write a cron job to update user metrics every 30 minutes. Knowing that Go is great and very fast for such tasks, I created the cron job in Go, built the binary, and scheduled a system daemon task with a timer for every 30 minutes. It works efficiently and consumes fewer resources. If I hadn't been tinkering in my spare time and only wrote code at my day job, I wouldn’t have come up with the best solution in a reasonable time. The cron job would have been written in Node, which would take more time as the user base grows.

So, never stop learning and creating on the side. The best way to learn is by creating and tinkering. I’ve been learning Ruby on Rails and Go on the side, and I’ve come to appreciate the different features that various ecosystems offer. This has helped me integrate new ideas into my workflow.

4) Take ownership

I recently watched a video of ThePrimagen that inspired me to write this blog. He mentioned that the best way to solve a problem or become a great software developer is to take ownership of the product. He talked about how Doom was created by just four guys who delivered such a good product because they took ownership. They knew they had no one else to rely on, so they made it their responsibility to develop the best possible software. There was no Plan B. They never felt burnout or gave up because they owned the product, not just the tasks.

To improve your skills as a software developer, you need to start taking ownership of the product you are building, not just features or tasks. You will find it much more enjoyable to work on a product when you see any feature or bug as your problem to solve, not just another task for someone else. This is the best way to beat burnout. When you take ownership, you will find joy in improving and making the product more efficient.

If you are working on a product, you can't blame others for any bugs that come up when users find them. You are part of the problem if things go wrong, so you have to take ownership to fix them and make a great product. Good, scalable products are built by teams, and if you don't take ownership, you are not a good team member. When you take ownership, you write the best possible code to create the best possible software, not just another software product.

Like the four guys who made Doom, they put in an insane amount of time to create something that was theirs, and they never settled for just another game, they created an era-defining game. The rest, as they say, is history. The same applies to you, if you want to make the best possible software, you have to start taking ownership and think of the product as your own.

Epilogue

I feel good after writing this blog and sharing my thoughts with the community. We might argue about frameworks, languages, and tools, but these debates help us improve. They push technology forward, making our community very competitive. Let's keep the passion alive!

MongoDB Replication Why and How?

RAM PANDEY — Sat, 16 Sep 2023 15:49:24 +0000

Hello, fellow developers! I trust you're doing splendidly. In this week's blog, we're delving into the intriguing world of MongoDB replication—the 'why' and 'how' behind it. Let's kick things off with the fundamental question: why replication?

The Why

Imagine your database as a bustling library of books, each book representing your valuable data. In a single-library setup, you have one copy of each book stored on a single shelf. While this might seem fine, it leaves your collection vulnerable. If, heaven forbid, that one shelf were to collapse or a book went missing, you'd lose precious knowledge.

Now, consider database replication as a strategy akin to having multiple libraries, each housing an identical collection of your books. This redundancy ensures that even if one library faces an issue—like a shelf collapsing (hardware failure) or a curious book thief (data corruption)—your knowledge remains intact in other libraries. This way, your data stays resilient, accessible, and safe.

The pitfalls of not considering replication become evident when you rely on a single library. In case of hardware failure, your entire collection is at risk, leading to data loss and costly downtime. Without replication, scaling to meet increasing demand becomes challenging, limiting your application's potential.

However, with replication, you enjoy an array of benefits. Improved data availability means your application stays up and running even in adverse conditions. Load balancing ensures efficient resource utilization, while failover mechanisms kick in when needed, ensuring uninterrupted service. Plus, you can easily scale your database to accommodate growing workloads.

In essence, replication is your fail-safe mechanism, your data's guardian angel. It's not just a best practice; it's a lifeline for safeguarding your precious data, avoiding potential pitfalls, and reaping the many benefits it brings to the table. Now that we understand the vital importance of replication for your database, let's delve into the 'how' — how to create a replication set in MongoDB.

How a Production level setup would look

Now that we've explored why replication is crucial, let's delve into how a production-level setup would appear.

In the illustration above, you can observe a primary node and two distinct secondary databases. Additionally, there's an arbiter situated on its own server, tasked with monitoring all the databases within the cluster. It continuously sends out small heartbeats to ensure all the machines remain online. In the event that the primary node experiences downtime, communication with it becomes impossible. In such a scenario, the arbiter steps in and assesses the remaining servers within your cluster. Subsequently, it will arbitrarily promote one of the secondary servers to assume the role of the new primary. This mechanism ensures the continued operation of your database even in the face of primary node failures.

For this demonstration, we will create a primary and a secondary node. However, if you intend to implement this replication setup in a production environment, I highly recommend following the complete setup outlined above.

How

Note: To follow along as we configure a replica set, it's recommended to download the latest version of MongoDB Community Edition. At the time of writing, the latest version is 7.0. For this blog, we'll use MacOS. While most commands are universal and work on both Linux and MacOS, Windows users may need to find equivalent commands.

Before diving into creating a replica set configuration, let's first grasp the basics of spinning up a MongoDB database. After all, a replica set comprises multiple databases. MongoDB relies on a configuration file called mongod.conf. The location of this file varies by operating system:

The location of mongod.conf for different OS are

OS	Path
Linux	/etc/mongod.conf
MacOS	/usr/local/etc/mongod.conf (on Intel processors), or /opt/homebrew/etc/mongod.conf (on Apple M1 processors)
Windows	< install_directory >\bin\mongod.cfg

To keep things organized and prevent unintentional changes to the default configuration, let's create new files and name it mongod.primary.conf and mongod.secondary.conf. This approach ensures that even if we make errors during the setup process, the default database remains unaffected.

To create the configuration files, we will begin by making a copy of the default configuration:

sudo cp mongod.conf mongod.primary.conf

sudo cp mongod.conf mongod.secondary.conf

Now, let's edit the configuration files to customize the settings. For each file, we will:

Set the destination for system logs.
Specify the data storage path (dbPath).
Define the port on which MongoDB should listen for requests.
Assign a name to the replication set.
Configure the MongoDB server to run as a background process.

mongod.primary.conf

systemLog:
  destination: file
  path: /usr/local/var/log/mongodb/mongo.primary.log
  logAppend: true
storage:
  dbPath: /usr/local/var/mongodb-primary
net:
  port: 27018
  bindIp: 127.0.0.1
replication:
  replSetName: demoRepl
processManagement:
  fork: true

mongod.secondary.conf

systemLog:
  destination: file
  path: /usr/local/var/log/mongodb/mongo.secondary.log
  logAppend: true
storage:
  dbPath: /usr/local/var/mongodb-secondary
net:
  port: 27019
  bindIp: 127.0.0.1
replication:
  replSetName: demoRepl
processManagement:
  fork: true

We have added the processManagement.fork option. This setting instructs the mongod server to run as a background process, allowing you to regain control of the terminal for running other commands.

You should also update the dbPath with the appropriate location for your operating system. Here are the default directories where MongoDB data is typically stored:

OS	Path
Linux	/var/lib/mongodb
MacOS	/usr/local/var/mongodb
Windows	C:\data\db

Please make sure to specify the dbPath according to your specific operating system.

The next crucial step is to create the necessary directories for storing the database data. Failure to do so may result in issues when starting the MongoDB server. Make sure to create these directories in advance to ensure a smooth setup process.

To create the necessary directories for the databases, you can use the mkdir command (for Linux and Mac). In this example, I'll create the directories in /usr/local/var/, but you should adjust the paths according to your preferences.

sudo mkdir /usr/local/var/mongodb-primary

sudo mkdir /usr/local/var/mongodb-secondary

With the directories in place, we're now ready to start up the databases.

To start the MongoDB server, you can use the mongod tool, which is included with the MongoDB installation. The command to start the server is as follows:

sudo mongod --config <config_file_name>

In this command, you specify the config file for the server to retrieve the necessary instance details.

Running this command will produce output similar to the following:

Notice that MongoDB forked as a child process. If you didn't specify the fork option, you would have lost control of the terminal. You can try running it both ways to see the difference in behavior.

If your setup was correct, you would be able to create both the instance without any issues.

Now lets get access to our databases using the mongosh tool which was shipped with the mongodb installation as well. We will write the commands in two different terminal tabs to get access to both of them at the same time

mongosh --port 27018

mongosh --port 27019

You should be able to get access and get an output similar to this

Now, if you try running any command in either of them, you will encounter an error like this:

The reason for this error is that, since we have specified the replication set name, the MongoDB instance will only be able to access the database after syncing with the other databases in the replication set. However, we haven't initialized the connection yet, so it remains in an unknown state.

To resolve this, we will start by using the primary database tab and running the following command:

rs.initiate()

The command will produce an output like this:

In this step, we haven't provided any specific configuration for the replica set, which is why you'll see an info message.

Now, let's add the secondary database to our replica set using the following command:

rs.add("<servername>:<port>")

rs.add("127.0.0.1:27019")

Running this command should yield an output like this:

With this, our replication set is now complete!

To check the status or obtain details about your replica set, you can use the rs.status() command.

Let's create a new database in the primary and verify if the changes are replicating in the secondary node or not.

Primary Database View:

Secondary Database View:

As you can see, without any additional effort, the database changes from the primary have been successfully replicated in the secondary. With these steps, we have configured our very own replication set.

I hope you found this tutorial informative and valuable. Please feel free to share any feedback or suggestions with me. Thank you for reading, and until next time!

Guide to building Multi-Tenant Architecture in Nodejs

RAM PANDEY — Sun, 10 Sep 2023 18:48:09 +0000

Greetings, fellow developers! I hope you're all doing well. Today, I'm thrilled to delve into the realm of multi-tenant architecture and share my insights and experiences from constructing a backend application that fully embraces this approach.

Let's paint a vivid picture: imagine your software as a colossal apartment complex. In this virtual edifice, every user or organisation functions as a tenant, residing in one of the apartments. They peacefully coexist within the same architectural structure, which represents your software's backend. However, they each maintain their unique, private spaces — think of them as their data sanctuaries housed in separate databases. The primary objective? Safeguarding their data, ensuring its security, and enforcing strict boundaries to prevent any unauthorised peeping into their digital "apartments." It's akin to orchestrating the management of a vast, interconnected digital apartment complex!

In this blog, we'll demystify the complexities of multi-tenant architecture, making it accessible even to beginners. So, let's embark on this enlightening journey together!

Before we commence, I'd like to extend a special shout-out to this exceptional Medium blog that served as a wellspring of inspiration for my exploration.

The Challenge at Hand

Each tenant must have their dedicated database.
Admin should possess the capability to deactivate specific tenants.
Storing user passwords in the main database is an absolute no-go.

Of course, there were other requirements as well, but we'll keep our focus squarely on these for the scope of this blog.

High Level Overview

As you can see, we have four clients, each with their own database. They all interact with the same backend, which performs its magic and connects them with their respective databases. To manage all of this, we need a super admin database that keeps track of all users and can store additional details you might need, such as pricing information or the modules to which a tenant has access.

Now that we've set the stage, let's dive into the exciting part – the code!

For this example, I'll be using Node.js, Express, and MongoDB for the database. However, you can adapt this approach to suit your tech stack. It's surprisingly straightforward once you grasp the concept.

P.S - After I completed the blog I noticed that the blog got too technical and some people might not like it and would only visit to get the gist or an idea of how it works. So I have added a section at last which gives the application overview. You can use that to just get the idea.

Code

Initialisation

To kickstart our project, let's create a directory for it. I'll name it "multi-tenant," but feel free to choose a name that suits your project.

mkdir multi-tenant

After this we initialise the npm project

npm init -y

Now, let's install the necessary third-party packages.

npm i cookie-parser express jsonwebtoken lru-cache mongoose

Feel free to initialize a Git repository as well, but for this demo, I won't cover that step.

Additionally, I'll be using ES6 modules instead of CommonJS. Please bear with me, as I work on both frontend and backend, and I prefer to keep things consistent when possible. To enable ES6 modules, you can add "type": "module" in your package.json or use the .mjs file extension.

Directory Structure

Before we dive into the code, let me explain how I organize my backend projects into different directories to keep the codebase organized. This structure is what we'll follow for this blog:

Controllers - Controllers: This is where API entry points reside, mainly for request validation.
Services - Here, the core business logic of the backend lives. It includes data manipulation, API calls, and database query building.
Repositories - These interact with the database. Services call repositories after constructing queries, and repositories handle database interactions and return results.
Utils - In this directory, I store helper functions that don't interact with clients but assist the backend, such as password hashing, JWT management, etc.
Middleware - This directory contains code executed before reaching our APIs. It's where we can write logic to determine the database to use and validate the tenant. This separation helps keep our main backend logic separate from connection logic.
Server - This holds configuration files for our server, such as Express setup and potentially CORS configuration (not used in this demo but recommended).
Routes - Code in this directory defines the routes and endpoints that clients can access.
Schema - Here, we store the database schema.

.
├── controllers
├── middleware
├── repositories
├── routes
├── schema
├── server
├── services
└── utils

Here is how our directory will look like.

Now, keep in mind that I won't go super deep into how to separate admin and tenant logic in this demo, but you can achieve it by creating separate admin and tenant directories within all the the directories to organize the logic accordingly. For this demo, we're focusing on setting up the architecture.

Creating Our Server

We'll start by creating the entry point for our server, which is the index.js file. It will initialize the Express app. The actual initialization of our Express app will be handled in the server directory, as index.js doesn't need to worry about the Express setup; its role is to require all the initialization functions, which connect to the DB, initialize middleware, set up Redis, etc.

Here's the server/express.config.js file:

import express from "express";

const ExpressConfig = () => {
  const app = express();
  app.use(express.json());
  app.set("trust proxy", true);
  return app;
};
export default ExpressConfig;

Now, let's use this configuration in our index.js, which will bind our server to a port:

import ExpressConfig from "./server/express.config.js";

const app = ExpressConfig();


const PORT = 5000;

app.listen(PORT, async () => {
  console.log(`Multi Tenant Backend running on port ${PORT}`);
});

It's important to note that for this demo, I haven't used environment variables to define the port. However, in your production project, it's highly recommended to use environment variables. This allows you to set the PORT variable and other configuration options dynamically based on your environment, making your application more flexible and secure.

Initial Middleware setup

Now, let's dive into setting up the initial middleware for our backend. Currently, we'll implement the essential middleware, and we'll revisit this section when we need to code the logic for validating and determining the database connection.

We start by creating the middleware/index.js file

import cors from "cors";
import cookieParser from "cookie-parser";

export default function (app: Application) {

  app.use(cookieParser());

}

For this demo, we've kept it simple. We're using cookie-parser to handle tokens as HTTP-only cookies. In production code, you can expand on this file to include additional middleware such as defining CORS policies, rate limiting, setting up request context (a file used to log the request-response cycle), and more. Feel free to customize it according to your project's specific requirements and security considerations.

Database schema setup

Now, let's define the schema for our data. In this demo, I'll keep it simple, covering the basic elements, but please feel free to modify it to suit your specific requirements. It's worth noting that I'm not registering my collections with mongoose.model here. I'll register them when I start working on the database connection setup. The reason behind this is to avoid registering super admin schemas for tenants and vice versa. We want to keep these schemas separate.

Let's begin with the super admin collections.

Tenant Collection

The first collection I want to build is the tenant collection. In this collection, you can store metadata related to tenants, such as the modules they've purchased, user limits, and whether they are enabled or disabled. For simplicity, I'll only store the name and database URI. Here's the code for schema/tenant.js:

import {  Schema } from "mongoose";

const tenantSchema = new Schema(
  {
    dbUri: { type: String, required: true },
    name: { type: String, unique: true, required: true },
  }
);

export default tenantSchema;

Tenant User Collection

The second super admin collection stores all the users our application has, regardless of their tenant. Again, for simplicity, I'll keep it basic. You can extend it to include fields like user roles or whether they are tenant administrators. Here's the code for schema/tenantUser.js:

import { Schema,Types } from "mongoose";

const tenantUserSchema = new Schema(
 {
   email: String,
   tenantId:{
       type: Types.ObjectId,
       ref: "tenants",
    }
  }
);
export default tenantUserSchema;

Now the tenants Database will have -

Users Collection

The third collection is a tenant collection. This will store data related to the users of that specific tenant. I've kept it basic for the demo, but you can add relevant fields to your schema as needed. Here's the code for schema/users.js:

import { Schema } from "mongoose";


const usersSchema= new Schema(
  {
    email: { type: String, unique: true, required: true },
    password: { type: String },
  }
);
export default usersSchema;

With these database schema definitions in place, you now have a general idea of the project structure and how the data is organized. Feel free to adapt and extend these schemas to match your project's specific requirements.

Repositories

In this section, we'll define simple functions that take a query and perform database operations using it. Note that the first parameter for these functions will always be a database connection object. Since we're dealing with multiple database connections in a multi-tenant setup, we need to provide the appropriate database connection to the repository so it can perform operations on the correct database.

Let's start with the repositories/tenant.js file:

import mongoose from "mongoose";

const mainSchemaName = "tenants";

const getTenantsRepo = async (
  adminDbConnection,
  findQuery = {},
  selectQuery = {}
) => {
  const data = await adminDbConnection
    .model(mainSchemaName)
    .find(findQuery)
    .select(selectQuery)
    .lean();
  return data;
};

const getATenantRepo = async (
  adminDbConnection,
  findQuery = {},
  selectQuery = {}
) => {
  const data = await adminDbConnection
    .model(mainSchemaName)
    .findOne(findQuery)
    .select(selectQuery)
    .lean();
  return data;
};

// This function is part of a service
// that involves many database calls, 
// so we'll use transactions here.
const addATenantRepo = async (
  adminDbConnection,
  tenantData,
  session = null
) => {
  const sessionOption = {};
  if (session) sessionOption.session = session;
  const data = await adminDbConnection
    .model(mainSchemaName)
    .create([tenantData], sessionOption);

  return data[0];
};

const updateATenant = async (
  adminDbConnection,
  findQuery = {},
  updateQuery = {},
) => {

  const data = await adminDbConnection
    .model(mainSchemaName)
    .updateOne(findQuery, updateQuery);
  return data;
};

export { getTenantsRepo, getATenantRepo, addATenantRepo, updateATenant };

Now, let's move on to the repositories/tenantUser.js file:


const mainSchemaName = "tenantusers";

// This function is part of a service 
// with transactions.
const addATenantUserRepo = async (
  dbConn,
  userData,
  session = null
) => {
  const sessionOption = {};
  if (session) {
    sessionOption.session = session;
  }
  const data = await dbConn
    .model(mainSchemaName)
    .create([userData], sessionOption);
  return data[0];
};

const getATenantUserRepo = async (
  dbConn,
  findQuery,
  selectQuery = {}
) => {
  const data = await dbConn
    .model(mainSchemaName)
    .findOne(findQuery)
    .select(selectQuery)
    .lean();

  return data;
};

const updateATenantUserRepo = async (
  dbConn,
  findQuery,
  updateQuery
) => {
  const data = await dbConn
    .model(mainSchemaName)
    .updateOne(findQuery, updateQuery);
  return data;
};

export { addATenantUserRepo, getATenantUserRepo, updateATenantUserRepo };

Lastly, we have the repositories/users.js file:


const mainSchemaName = "users"

// This function is part of the service 
// with transactions.
const addAUserRepo = async (
  dbConn,
  userData,
  session = null
) => {
  const sessionOption = {};
  if (session) sessionOption.session = session;
  const data = await dbConn
    .model(mainSchemaName)
    .create([userData], sessionOption);
  return data[0];
};

const getAUserRepo = async (
  dbConn,
  findQuery = {},
  selectQuery = {}
) => {
  const data = await dbConn
    .model(mainSchemaName)
    .findOne(findQuery)
    .select(selectQuery)
    .lean();
  return data;
};

const updateUserRepo = async (
  dbConn,
  findQuery,
  updateQuery
) => {
  const data = await dbConn
    .model(mainSchemaName)
    .updateOne(findQuery, updateQuery);
  return data;
};

const getUsersRepo = async (
  dbConn,
  findQuery = {},
  selectQuery = {}
) => {
  const data = await dbConn
    .model(mainSchemaName)
    .find(findQuery)
    .select(selectQuery)
    .lean();

  return data;
};


export {
  addAUserRepo,
  getAUserRepo,
  updateUserRepo,
  getUsersRepo,
};

Main connection setup

In this section, we will focus on setting up the logic for managing all the database connections and utilizing an optimal database structure, such as an LRU (Least Recently Used) cache, to efficiently manage these connections. While we're defining both initialization functions for the admin and tenant connections here, you should consider separating the logic for better organization. Additionally, note that we register the models here on the database object.

Lets start with the utils/initDBConnection.js file

import mongoose, { Connection } from "mongoose";
import TenantSchema from "../schema/tenant.js";
import TenantUserSchema from "../schema/tenantUser.js";
import UserSchema from "../schema/user.js";

const clientOption = {
  socketTimeoutMS: 30000,
  useNewUrlParser: true,
  useUnifiedTopology: true,
};

// Log MongoDB queries
mongoose.set("debug", true);

const initAdminDbConnection = async (
  DB_URL
) => {
  try {
    const db = mongoose.createConnection(DB_URL, clientOption);

    db.on("error", (err) =>
      console.log("Admin db error: ", err)
    );

    db.once("open", () => {
      console.log("Admin client MongoDB Connection ok!");
    });

    await db.model("tenants", TenantSchema);
    await db.model(
      "tenantusers",
      TenantUserSchema
    );

    return db;
  } catch (error) {
      return error;
  }
};

const initTenantDBConnection = async (
  DB_URL,
  dbName
) => {
  try {
    const db = mongoose.createConnection(DB_URL, clientOption);

    db.on("error", (err) =>
      console.log(`Tenant ${dbName} db error: `, err)
    );

    db.once("open", () => {
      console.log(
        `Tenant connection for ${dbName} MongoDB Connection ok!`
      );
    });

    await db.model("users", UserSchema);

    return db;
  } catch (error) {
    return error;
  }
};
export { initAdminDbConnection, initTenantDBConnection }

Now, let's define the code for the LRU cache manager in the utils/lruCacheManager.js file:

import { LRUCache } from "lru-cache";
import { Connection } from "mongoose";

const cacheOptions = {
  max: 5000,
  maxAge: 1000 * 60 * 60,
};

const connectionCache = new LRUCache(cacheOptions);

const setCacheConnection = (tenantId, dbConn): void => {
  console.log("Setting connection cache for ", tenantId);
  connectionCache.set(tenantId, dbConn);
};

const getCacheConnection = (tenantId) => {
  return connectionCache.get(tenantId);
};

const getCacheValuesArr = () => {
  return connectionCache.values();
};

export { setCacheConnection, getCacheConnection, getCacheValuesArr };

Now, let's code the file that is the heart of the application, the connection manager file! This file contains the logic for initializing the database and managing the collections so that our application can use them. Here is the code for utils/connectionManager.js:

import mongoose from "mongoose";

import { initAdminDbConnection, initTenantDBConnection } from "./initDBConnection.js";
import {
  getATenantRepo,
  getTenantsRepo,
} from "../repositories/tenant.js";
import {
  getCacheConnection,
  getCacheValuesArr,
  setCacheConnection,
} from "./lruCacheManager.js";

let adminDbConnection;

// This function will be called at the start
// of our server. Its purpose is to initialize the admin database
// and the database connections for all of the tenants.
export const connectAllDb = async () => {
  const ADMIN_DB_URI = `your admin db uri`;

  adminDbConnection = await initAdminDbConnection(ADMIN_DB_URI);

  const allTenants = await getTenantsRepo(
    adminDbConnection,
    { name: 1, dbUri: 1, _id: 1 }
  );

  for (const tenant of allTenants) {
    const tenantConnection = await initTenantDBConnection(
      tenant.dbUri,
      tenant.name
    );

    setCacheConnection(tenant._id.toString(), tenantConnection);
  }
};

export const getConnectionForTenant = async (
  tenantId
) => {
  console.log(`Getting connection from cache for ${tenantId}`);
  let connection = getCacheConnection(tenantId);
  if (!connection) {
    console.log(`Connection cache miss for ${tenantId}`);

    const tenantData = await getATenantRepo(
        adminDbConnection,
        { _id: tenantId },
        { dbUri: 1, name: 1 }
      )

    if (tenantData) {
      connection = await initTenantDBConnection(
        tenantData.dbUri,
        tenantData.name
      );
      if (!connection) return null;

      console.log("Connection cache added for ", tenantData.name);

    } else {
      console.log(
        "No connection data for tenant with ID",
        tenantId
      );
      return null;
    }
  }

  return connection;
};

export const getAdminConnection = () => {
  console.log("Getting adminDbConnection");
  return adminDbConnection;
};

const gracefulShutdown = async () => {
  console.log("Closing all database connections...");

  const connectionArr = getCacheValuesArr();

  // Close all tenant database connections from the cache
  for (const connection of connectionArr) {
      await connection.close();
      console.log("Tenant database connection closed.");
  }

  // Close the admin database connection if it exists
  if (adminDbConnection) {
    await adminDbConnection.close();
    console.log("Admin database connection closed.");
  }

  console.log("All database connections closed. Yay!");
};

let isShutdownInProgress = false;

// Listen for termination signals
["SIGINT", "SIGTERM", "SIGQUIT", "SIGUSR2"].forEach((signal) => {
  process.on(signal, async () => {
    if (!isShutdownInProgress) {
      console.log(`Received ${signal}, gracefully shutting down...`);
      isShutdownInProgress = true;
      await gracefulShutdown();
      process.exit(0);
    }
  });
});

Hash!!! that concludes the major part of our application. I hope you understand the code its not much just some functions to manage the db connections but it is still the heart of our application.

Lets define some util functions that will help us later in our services. We can name this file as the misc.js which contains miscellaneous functions so here is the code for utils/misc.js

import jwt from "jsonwebtoken";

const signJWT = (data) => {

  return jwt.sign(data, "random secret");

};

const verifyJWT = (
  payload
) => {
    return jwt.verify(payload, "random secret");
};

// define in your env file
const saltRounds = 10

const generateHash = async (input) => {
  try {
    const hash = await bcrypt.hash(input, Number(saltRounds));
    return hash;
  } catch (error) {
    console.error("Error generating hash:", error);
    throw error;
  }
};

const comparePassword = async (plainPassword, hash) => {
  try {
    const match = await bcrypt.compare(plainPassword, hash);
    return match;
  } catch (error) {
    console.error("Error comparing password:", error);
    throw error;
  }
};

export {
  signJWT,
  verifyJWT,
  generateHash,
  comparePassword,
};

Services

In this section, we'll dive into the core business logic of our application.

So lets start with the most intense service file! The services/tenant.js file.

import mongoose from "mongoose";
import {
  addATenantRepo,
 } from "../repositories/tenant.js";
import { addATenantUserRepo } from "../repositories/tenantUser.js";
import { setCacheConnection } from "../utils/lruCacheManager.js";

import { addAUserRepo } from "../repositories/user.js";
import { initAdminDbConnection, initTenantDBConnection } from "../utils/initDBConnection.js";

const addATenantService = async (
  dbConn,
  tenantData
) => {
  const session = await dbConn.startSession();
  session.startTransaction();
  try {
    const data = await addATenantRepo(
      dbConn,
      { ...tenantData },
      session
    );

    let userData;
    if (data._id) {
      userData = await addATenantUserRepo(
        dbConn,
        {
          tenantId: data._id,
          email: tenantData.email,
        },
        session
      );

      const tenantDbConnection = await initTenantDBConnection(
        data.dbUri,
        data.name
      );

      await addAUserRepo(
        tenantDbConnection,
        {
          _id: userData._id,
          email: tenantData.email,
        },
        session
      );

      await session.commitTransaction();
      session.endSession();

      setCacheConnection(data._id.toString(), tenantDbConnection);
    }

    return {
      success: true,
      statusCode: 201,
      message: `Tenant added successfully`,
      responseObject: { tenantId: data._id, userId: userData?._id },
    };
  } catch (error) {
    await session.abortTransaction();
    session.endSession();
    throw error;
  }
};

export { addATenantService };

The addATenantService function handles the process of adding a new tenant to the system. It follows these steps:

Adds tenant data to the super admin tenant collection.
Adds user details to the super admin tenant users collection.
Links the user to the tenant in the tenant users collection, maintaining consistency between the super admin and tenant databases.

Now, let's move on to the services/auth.js file, which contains authentication-related logic:

import { signJWT } from "../utils/misc.js";

const loginService = async (
  userData
) => {
  if (!userData || !userData|| !userData._id || !userData.tenantId)
    return {
      success: false,
      statusCode: 401,
      message: `No user with the given credentials`,
      responseObject: {
        incorrectField: "email",
      },
    };

  // Do some password matching

  const accessToken = signJWT(
    {
      userId: userData._id.toString(),
      tenantId: userData.tenantId.toString(),
    }
  );

  return {
    success: true,
    statusCode: 200,
    message: `Logged In Successfully`,
    responseObject: {
      accessToken,
      userId: userData._id.toString(),
      tenantId: userData.tenantId.toString(),
    },
  };
};

The loginService function handles user login and generates an access token, which is crucial for our application as it contains the userId and tenantId. This token will be used to verify requests and determine the appropriate database connection based on the tenantId. For this demo I didn't write the code to match the passwords but you should write your own logic based on your needs.

This concludes our Services section. Now lets move on to the Controllers and Route section.

Controllers & Routes

I've covered the Services section, and now let's dive into Controllers and Routes. In this section, I'll provide you with the necessary code for setting up controllers and routes. I won't delve into detailed explanations, as the code is straightforward. However, it's essential to organize your code effectively, so I'll provide a basic structure.

So here is our controllers/index.js file code

import { loginService } from "../services/auth.js"
import { addATenantService } from "../services/tenant.js"

export function loginController = async (req,res)=>{
  const serviceFnResponse = await loginService(req.body);

  res.status(serviceFnResponse.code).json({...serviceFnResponse});
}

export function addATenantController = async (req,res)=>{
  const serviceFnResponse = await addATenantService(req.body);

  res.status(serviceFnResponse.code).json({...serviceFnResponse});
}

In these controllers, you should add code to validate the request body, ensuring data integrity and security. Proper input validation is a crucial step in building a robust application.

Now, let's define your application's routes in the routes/index.js file:

import { Router } from "express"
import { loginController, addATenantController } from "../controllers/index.js"

const router = Router()

router.post("/add",addATenantController);
router.post("/login",loginController);

export default router;

This code creates an Express router and defines your application's routes. In a real-world scenario, you would likely have more routes, each mapped to specific controllers. For maintainability, consider organizing your routes into separate files, one per resource or feature.

By structuring your code this way, you maintain a clean and organized project, making it easier to add new features or extend existing ones in the future.

Middleware

Now, let's dive into the engine of our application – the middleware. This code will handle the heavy lifting, such as deciding the database connection and tenant for each request. We'll start with the middleware/databaseResolver.js file:

import { getConnectionForTenant } from "../utils/connectionManager.js";
import { verifyJWT } from "../utils/misc.js";

export const databaseResolver = async (req, _, next) => {
  const urlArr = req.url.split("/");

  // Skip database resolution for login route
  if (urlArr.includes("login")) return next();

  const token = req.headers.jwt;
  // Handle the logic for null checking and authorization
  const payloadData = verifyJWT(token);
  // Handle the expiry logic, etc.
  const dbConnection = getConnectionForTenant(payload.tenantId);

  // Here, we are directly populating the req object, but you can use
  // custom context managers in your application
  req.dbConnection = dbConnection;
  next();
};

We've defined the basic middleware logic that resolves the appropriate database connection based on the request's tenant information. The middleware also skips this process for the login route.

Next, let's configure the middleware to be used in our application. Create a new file called server/middleware.config.js:

import { databaseResolver } from "../middleware/databaseResolver.js"

export default function(app){
  app.use(databaseResolver);
}

You might be wondering why we use a separate file for middleware when the code is relatively short. However, in a production-level application, you'll likely have more complex middleware requirements, such as request sanitization, logging, and more. Having a separate middleware configuration file allows you to manage and organize these requirements efficiently.
Exmaple - santizing middleware for requests, Logging middleware, etc.

This file sets up your application's routes, and in a real-world scenario, you'd have multiple routes organized by resource or feature.

server/route.config.js

import router from "routes/index.js"

export default function(app){
  app.use('/api',router);
}

Now all you need to do is just require all the config in your index.js file

import ExpressConfig from "./server/express.config.js";
import MiddlewareConfig from "./server/middleware.config.js";
import RouteConfig from "./server/route.config.js";

const app = ExpressConfig();

MiddlewareConfig(app)
RouteConfig(app)

const PORT = 5000;

app.listen(PORT, async () => {
  console.log(`Multi Tenant Backend running on port ${PORT}`);
});

By structuring your code in this way, you maintain a clean and organized index.js file, making it easier to understand and extend your application as it grows.

And we're done! Congratulations on building the foundation for your own multi-tenant application. This powerful structure can be used as a base for creating complex Software as a Service (SaaS) applications. If you feel that the technical details have become a bit overwhelming, don't worry. Let's provide an overview using a diagram to visualize how the application works, giving us a refresher on all the moving parts in our application.

Application Overview

Client Request: When a user interacts with your platform, like logging in or accessing data, their request is sent to your server. This request comes with a special key called a JSON Web Token (JWT), like a secret passcode.
Middleware: The request first enters a checking point called Middleware. It's like a bouncer at a club. If the request is about logging in, it gets a quick pass. Otherwise, it moves on to the next step.
Database Resolver: Here comes the real magic. The Database Resolver looks at the JWT passcode and figures out which company (tenant) the user belongs to. It's like sorting mail to different mailboxes.
Database Connection: Once the Resolver knows which tenant it is, it opens the right door – the tenant's database. It's as if each tenant has their own room in a huge library, and only they can access their own books.
Controller: With the right room (database) open, the Controller takes over. It's like a librarian who helps you find the book you need. The Controller figures out what the user wants to do and fetches the information from the correct room (database).
Services: Inside the Controller, special helpers called Services do the hard work. They carry out tasks like checking if the user's password is correct or fetching data. These Services are like expert librarians who know where every book is kept.
Database Interaction: The Services talk to the database, ask it questions, and get answers. For example, they might ask, "Is this the right password?" or "Give me the data for this user." Each tenant's data is kept separate, so there's no mix-up.
Response Generation: Once the Services have all the answers, they create a response. It's like putting together a report with all the necessary information, and it includes a message like "Success!" or "Sorry, that's not right."
Controller Response: The Controller gets this report and prepares it for delivery. It's like putting it in an envelope with the user's address on it.
Client Response: Finally, the response is sent back to the user's device. It's like getting the report in the mail. The user sees the message and any data they requested, and they can continue using your platform.

So, in simple terms, your multi-tenant application is like a giant library with many rooms (databases). When a user asks for something, the application makes sure they go to the right room to get the right information. This way, each tenant's data stays private and secure, just like different books in different rooms of the library.

This system is great for running a cloud service where each tenant gets their own space while sharing the same platform.

If you have any further questions or need additional clarification on specific aspects of the application, please feel free to ask! Also if you have any suggestions for improvement please do let me know I would love to hear it.

An advanced setup for Travis CI

RAM PANDEY — Fri, 31 May 2019 16:08:57 +0000

In this blog post, I will guide you through the basics of the YAML file and then we will write a sample .travis.yml file which I try to make as general as possible so that you can use Travis CI more efficiently. Also, this blog post is for those people who are familiar with the concept of CI/CD and testing.

What is YAML?

YAML is a recursive acronym for “YAML Ain’t markup language”. Like JSON it is a data serialization language( i.e it can be used to store or transmit data across networks)

Some differences between YAML and JSON

YAML syntax

I will cover some basic syntax that will get you started but if you would like to learn more, refer this link

The basic syntax of YAML is

Key value pairs are separated using ":"

key: value

You can comment using "#"

# this is a comment in YAML
key: value

This knowledge is enough for our example. Now let us start with our .travis.yml file

The Travis config file

Please keep a note on this as YAML files have strict rules for indentation.

The programming tool

We start configuring our .travis.yml file with the language we will be using. For example, I will be creating this Travis file for my MERN stack app so I will choose my language as nodejs for testing my backend code.

language: node_js
node_js:
  - "11"

Now here we tell the Travis CI that we want our environment to be nodejs we also specify the specific version of it. I used v11 you can use any version you want, you can test for multiple version by mentioning them below like

language: node_js
node_js:
  - "11"
  - "12"

You can also test this for the current stable version of nodejs like

language: node_js
node_js:
  - "stable"

The Build Environment

We can specify the distro/OS/Build environment we would like Travis to execute our tests in. I would recommend using trusty as this distro is lightweight and more reliable. Although this is the default distro set by Travis CI, I will show you the config so that you can use any distro you want and is offered by Travis.

dist: trusty

Other distros offered by Travis are

Ubuntu Xenial 16.04
Ubuntu Precise 12.04

Services

Many times we have various services interacting with our project. We need to include them to completely test our projects. One option you can choose is to integrate an online set up of the services but that would increase the time taken by the tests plus if the service is paid you will have to pay the extra cost for every request to your service. So Travis ci provides a solution for this by providing you the option to install the service you need with the build and configure it according to your needs.
There are many services we can configure but this post is not about it so
I will show you how to configure MongoDB for CI.

services: mongodb

You have to use the key services for any service you want to tell Travis to install the dependencies. Now to configure it you can do it the same way you do locally eg-

var url = "mongodb://localhost:27017/mydb";

MongoClient.connect(url, function(err, db) {
  if (err) throw err;
  console.log("Database created!");
  db.close();
});

that's it you have to connect to the port no 27017 on localhost. Just follow the same default connection instruction for the service you would like to connect. You can look up here for other services.

Environment variables

You can set up the environment variables in which you would like to test your project using the env key eg -

env:
  - NODE_ENV=ci PORT=3000

cache

You can cache directories in your build so that you do not have to download it on every build as that would cost you time so you can specify the directories you want to cache using

cache:
  directories:
    - node_modules

Here I would like to cache my node_modules directory for every build so I don't have to download it on every build and when I add or delete a node_module Travis would just compare the package.json and make the necessary changes and cache it again

Scripts

There are various dependencies that you would like to be installed before or after the configuration of Travis CI. Or you may need to run various scripts before the build starts executing your tests. You can install or run them by using the keyword before_install to run commands before execution and install to install the dependencies after the Travis finishes to configure the build
eg -

before_install:
  - sudo apt-get install -y libxml2-dev
install:
  - npm install

So here before installing dependencies, I would like to add a dev library and after that, I would like to run the script to install dependencies for my project.

Also, you can run various scripts during the execution of the tests with the script keyword eg -

script:
  - npm run start 
  - npm run dev

So these are some advanced setup you can configure Travis with to make your life easy and use CI efficiently.

I would love to hear some more advanced setup from you guys that you use or your feedback for this blog I would appreciate it. Thanks for reading and I would love to hear if I was able to impart new knowledge in you through this blog post.

You can connect with me here at twitter or LinkedIn.

I live by the name of @ram2510 on the internet

How to automate your Git workflow

RAM PANDEY — Wed, 22 May 2019 12:36:51 +0000

GITESY

A node js based cli that can automate your git workflow!!

Gone are the days when you had to manually create the git repo both locally and remotely.

Now you can create the repo both locally and remotely in 3-4 clicks using "gitesy"

Installation

Type this in your command prompt/terminal

npm i gitesy -g

please note - Install this as a global package otherwise this may not work as expected

Usage

Create a new repo

To create a local and remote repo use gitesy -n command with the name of the repo

  gitesy -n test

After which if you are using this for the first time you will see a prompt for entering your credentials like this

Enter them and then you will get prompts asking details of the repo fill them and then it will automatically create the repo. Then you will get the message and start working on the project !!!!

If you are not using this for the first time you will see a prompt asking whether gitesy can use creds you entered previously It's very easy to create a new repo and start coding. I also added the none option too, just for creating an empty git repo both locally and remotely.

Add a new template

To Add a template create a template add the features and then use this command from the same directory

  gitesy -a <nameOfTheTemplate>

Epilogue

You can contribute to this project here

Things you can do

fix typos
add more general templates

Please give me your suggestions they are valuable to me.

Visit the npm page for gitesy here

If you like this npm package please share it so that we can make this a better package with suggestions. Thank you !!!!

You can connect with me here at twitter or LinkedIn.

3 Tips for new developers

RAM PANDEY — Thu, 16 May 2019 09:02:28 +0000

Prolouge

I know that there are many articles on the internet where devs share their experiences and you might be thinking why should I read them but from my personal experience I find these posts useful as I get to learn something new in every post

In this blog, I highlight Some mistakes that I did and have seen others make so that you do not make them or if you have already committed those mistake you recognise it and try to rectify them.

I am a self-taught web developer who had no mentor or a person that I can fall back to if I needed help. I have been working as a web developer for 6 months and here are the 3 tips I wish someone told me when I started my career.

1) Be curious

I have seen many of my classmates say that they have "learned" so and so programming language by attending so and so Bootcamp or taking classes online. This is a very bad sign, you just know how to use this language you cannot master a programming language in 12 weeks time. But the main problem here is it kills your hunger to learn more. Remember When you started the class you were so hungry and happy to learn the language but when you complete the class and you say that you have mastered it you instantly kill that hunger. You have just learned a part of that language, you should not stop now, always try to keep learning new parts of that language. I am in no way saying that you should not take pride or enjoy the sense of accomplishment of completing the course but don't let that pride kill your hunger. Always be curious to learn new things it is a very vast field there is always something new to learn.

Stay Hungry Stay Foolish.

– Steve Jobs

2) Don't follow the trend

Here I would like to point out the mistake that I made when I first started out programming I was riding the bandwagon of hype. I wanted to do machine learning but when someone asked me why? I would say because that's the future, not because I like it or something and guess what I was never successful in my pursuit of becoming an ml engineer, as I didn't have passion for the field. I was demotivated by this. Due to which I always procrastinated and my tasks kept on piling. Then I started HTML, CSS JavaScript and oh boy! it was love at first sight. I became passionate about this field and wanted to learn every new technology here. I never needed motivation again. I was enjoying every bit of it and I still do, I hope this continues. Again now I see many of my classmates riding the hypewave as when I ask them why you want to learn this technology? I get an answer stating that this will make me more money. I am not saying that you shouldn't try out the technology but first ask why do you want to.

The only way to do great work is to love what you do.

– Steve Jobs

3) Don't Be disheartened by errors

While Developing new software or upgrading old ones you will definitely get many errors. Some errors might even take months to patch. Some errors will be a quick fix but you should never give up on that project just because you couldn't solve that error. There is a huge community of programmers out there that are willing to help you. You just need to ask. So the next time you encounter an "not fixable" bug or error don't give up, ask for help there are many great forums on the internet like StackOverflow where people are always willing to help. You just need to ask.

The only Mistake you can make is not asking for help

– Sandeep Jauhar

Other mentionable tips

1) Always be humble
2) Always commit your changes! 😊
3) Develop soft skills

Epilouge

This is my first blog post ever and I had planned that I wanted to write a blog that would help the community. If you liked the blog or would like to share your own tips or if you have any suggestions for me please share your views in the comment box down below. I would really appreciate It. Thank you for reading!!!

You can connect with me here or at twitter , LinkedIn.