DEV Community: Rajdip Bhattacharya

Design Decision for Role-Based Access Control (RBAC)

Rajdip Bhattacharya — Wed, 21 Jan 2026 14:14:29 +0000

Every organization uses a set of SaaS products for their daily operations. Every such product is responsible for sheltering data and performing operations at large scale. So there should be appropriate checks as to who can do what with the data. And that's where Role-Based Access Control comes into play.

A bit about RBAC

Before we dig in, a bit of an explainer will help us start off at the same page. RBAC is a 1-1 mirror of how organizations work IRL. You as an employee will have a job role, and with your role, comes responsibilities and permissions. An employee who is an accountant will have access to the debit/credit book of the company. A DevOps engineer will be heading the pipeline development of the products. And so on.

Similarly in SaaS products, to control the access of users to specific actions and data, we put RBAC in place. Project admins (ofter the person who signed up for the product) creates teams, roles and assigns them to certain users within the product.

Whom this article is not for

Unconventional way to start, but since I'm writing this to address a very niche, I would like to set the expectations for the readers in here.

This article is not for you if:

You are looking for a Node-based RBAC implementation
Yes that's just about it.

But, you should read it if you:

Are a beginner and trying to grasp the fundamentals of RBAC
Trying to debug what went wrong in your design, or, attempting to improve your design
Need a handbook for initial steps
Please your seniors with a weekend blog reflection

RBAC Designs

The most basic form of RBAC system will almost always have these entities in some form or the other:

Organization: The unit under which users, resources, and roles reside. This is where the action takes places.
User: The actors, using the platform and taking actions.
Resource: Items which are being acted upon by the users. It can be docs, VMs, databases, etc. These are what we protect using roles.
Role: The gatekeeper, preventing/allowing users' access to resources.
Permission: Associated with roles, define what a role will permit its users to do.
User Group: Optionally present in infrastructures with massive number of resources. It associates with multiple roles and users.

The Design

An organization is the host. Everything in our infrastructure (except permissions) are tied up with the organization.
An organization can have multiple users → 1 x N
An organization can have multiple roles → 1 x N
An organization can have multiple resources → 1 x N
An organization can have multiple user groups → 1 x N
A role can be associated with multiple permissions, and permission can be associated with multiple roles → M x N
A role can be associated with multiple users, and a user can be associated with multiple roles → M x N
A role can be associated with multiple resources, and a resource can be tied up with multiple roles → M x N
A role can be tied up with multiple user groups, and a user group can be tied up with multiple roles → M x N
A user can be tied up with multiple user groups, and a user group can be tied up with multiple users → M x N

High-level diagram:

Detailed diagram:

Design Outcomes

Having a separate table for permissions is purely a design choice, and not a mandate. Having a separate permission table allows you to scope permissions for specific resources. For example, the role ORGANIZATION_ADMIN won't be applicable to a Document resource. So, you could have an optional list of acceptable/unacceptable resource types per resource.
A user can be tied up with both roles and user groups simultaneously.
User groups are never tied with resources directly.
There can be roles, that may not be tied with any resources at all. For example, a role having ADMIN would imply administrative access over the organization, and hence won't have any resource mapping.
We could have chosen a higher order join-table, but having binary join-tables provide a ton of flexibility at the cost of some data repetition.

The Least Privilege model

Usually, this is where I conclude my blogs with a Conclusion, but I thought about making this section a bit more informative. This is not related to how you should implement a RBAC system, but how you should use it properly.

The Least Privilege Model states that actors should be granted the least possible set of permissions required for it to operate efficiently.

Consider you have created a pipeline that will fetch your code from your SCM, build an artifact, and deploy it to AWS Lambda. So collectively, it will require the following permissions:

Read repository on SCM
Pull code from SCM
Push deployment to AWS Lambda

This is how you build a restrictive system with the least access.

Why does this matter? — Say that your deployment script gets a new change that was pushed unchecked. This can now affect other resources without your knowledge.

The only takeaway from this section would be this: your home-brewed RBAC system should be fine-grained enough so that users relying on your product can apply this principle.

Managing Secrets in Dev Tools Without Getting Yelled At

Rajdip Bhattacharya — Tue, 09 Sep 2025 06:46:15 +0000

Good day geeks! It has been a long time since I dropped my last article, so I thought about coming back with one that would actually be helpful to all sorts of developers.

So what's this blog about?

If you are reading this, chances are that you have already understood what this is about just by glancing at the title. If not, well that's what I am writing this about!

We developers have a bad habit of hard-coding secrets into our codebase, just o make sure that the bigger picture works. "It's on dev, we'll remove it before it hits our repository", we say. And then, it turns out that we never really got rid of it. The hardcoded secrets hit our SCM, stays there unnoticed, and ultimately lands in the hands of a malicious actor.

Project onboarding is surely painstaking. We add a teammate, and now we have to send them the .env file - over email / discord / slack / teams, and sometimes even over messages. That's not a very secure practice. And the problem doesn't end there. What if we update the OPEN_AI_API_KEY? We would now need to broadcast this to the entire team, and also hand them the secrets. Our servers would need the latest values too, so that we don't start hitting 500.

But why does it really matter?

You might be wondering that these are just stupid mistakes we developers do. That PR reviews should be more vigilant, that there should be to-and-fro communication whenever something fails. But hey, we are humans, and humans will make mistakes.

AI won't be so stupid!

True, but what if the person sitting behind that prompts isn't as experienced? What if they don't understand how sensitive the contents of a .env are?

If you have read so far, that means I have been successfully able to glue your attention to the screen, and you might be expecting a solution. And in case you are not impressed, let me drop some numbers.

GitGuardian's The State of Secret Sprawl 2025 quotes a whooping 23 million plaintext secrets detected in GitHub commits, 70% of which are valid and exists till date. And the numbers are still increasing! It shot up to 23.8M in 2024 from the 19.1M in 2023. The potential breaches resulting from such leaks can amount up to $4.4 million, as stated by IBM's Cost of a Data Breach Report 2025.

As you can see, that's enough to cripple an organization to its knees.

And now, the solution!

All's not lost! Problems are why engineers exist in the first place, and this is where I would like to introduce you to Keyshade

Keyshade?

Yep! Keyshade is an open-source secret and configuration management platform, built with a security-first and developer friendly mindset.

But why would I store MY SECRETS somewhere else?

Good question! Well, there are a lot of reasons:

Forget "Hey bro, can you DM me the .env file?". Keyshade helps you to stay in sync. You upload your configurations to your project, and your entire developer team taps into this project.
Data ownership is a big concern. Keyshade's approach to solve this problem is unique. It uses public key encryption to store your secrets, which means even keyshade won't be able to decrypt your secrets. You hold access to the private key, and you decide to share it with your teammates.
Your hosted applications get to use your configurations aswell! You could do so by using their platform-specific integrations or their CLI.
And the best part? It's a drop-in replacement. You don't need to change a single line in your codebase to adapt to keyshade. It just works.

They offer a generous free-tier for you to play around with.

Demo time!

Well, enough talking. Let's see how it actually works. I would be using a simple NextJS application to demonstrate the platform.

First, head over to app.keyshade.xyz and get yourself an account.

You would land up in a page like this:

Here's a bit on the terminology that keyshade uses:

Workspaces: Consider these as organizations.
Projects: This is what houses your secrets, variables, and environments
Secrets: Sensitive stuff from your .env file. These are always stored in an encrypted form. For example
Variables: Not so sensitive stuff. For example, port number.
Environments: Think of them as development environments. For example, dev, prod, stage, uat, etc.

Install the CLI

Keyshade has it's dedicated CLI to perform all operations on the platform, and also to use your configurations in your local development environment. We would need that for our demo. Here's how to install the CLI:

npm i -g @keyshade/cli

Once successful, you should be able to use:

keyshade -V

Setting up a profile

Profiles can be considered as your identity to access keyshade. The possibilities are many, but we will stick to the most basic use.

First stop, head over to Dashboard → Profile Dropdown → Profile → API Keys. Now, hit Add API Key and create a key for yourself. For demo purposes, select all the authorities.

Once generated, you should get an API key. Take a note of that.

Open up your favourite terminal, and issue the following command:

keyshade profile create -n my_key -a <your key> --set-default

Once done, you should be able to see it using:

keyshade profile list

That's it! You have now successfully logged on to keyshade from your local machine. To verify everything works, try listing your workspaces:

keyshade workspace list

The NextJS project

This is what I have in my project:

In src/app/page.tsx, I have the following content. You can replace yours with mine:

export default function Home() {
  return (
    <div className="bg-white h-screen w-screen text-center text-black flex flex-col">
      <div><span className="font-bold">Secret (NEXT_PUBLIC_API_KEY):</span> {process.env.NEXT_PUBLIC_API_KEY}</div>
      <div><span className="font-bold">Variable (NEXT_PUBLIC_PORT):</span> {process.env.NEXT_PUBLIC_PORT}</div>
    </div>
  );
}

Things to note:

I don't have a .env file
We are reading NEXT_PUBLIC_API_KEY and NEXT_PUBLIC_PORT from the environment.

If you fire up this project now using npm run dev, and head over to https://localhost:3000, you would see something like this:

Create a project

To start adding secrets and variables, we would first need to create a project. Head over to Dashboard → Create Project.

Access level: Defines who get to see this project. Since it's just you who would be seeing it, set it to PRIVATE (default)
Store private key: This is what gives you ownership. Keyshade recommends you to not store your private key with them. Set the slider to off (default)
Environments: We won't need too many environments, so this is good.

It should ultimately look like this:

After hitting Create Project, you would be greeted with a new dialog.

Copy the command
Head over to your local project
Fire up your terminal
Paste it and hit enter

Wallah! Now you have successfully tied up your local project with keyshade.

Creating secrets and variables

We are all set. The last lego block would be to add in the values for NEXT_PUBLIC_API_KEY and NEXT_PUBLIC_PORT. As the name suggests, the first one would be a secret (secure by nature), and the second is a variable (not so secure)

Adding `NEXT_PUBLIC_API_KEY`

Head over to your project
Click on Secret tab
Click on Add Secret
Set the name as NEXT_PUBLIC_API_KEY
Fill the Note if you want to, i'll leave it blank
Enter abc as the value for default (the environment we selected when creating the project)
Hit Add Secret

You should see something like this:

Adding `NEXT_PUBLIC_PORT`

The steps are exactly the same, just that you would be adding this one under Variable tab. I'll be using 1234 as the value for default. Once you are done, you should be able to see this:

Seeing the magic in action

To harness the power of keyshade, we need to use their CLI to run our scripts. So,

npm run dev

would now become

keyshade run -- npm run dev

You should see this in your terminal:

From the output, note that keyshade has fetched the secret and variable that we created in the project before:

ℹ️ INFO: Fetched 2 configurations (1 secrets, 1 variables)

So now, when you head over to https://localhost:3000, you get to see the actual values for your configurations!

Closing thoughts

I'm sure you might have found this article useful, especially if you and your team have been struggling with secrets management. Keyshade covers up for most of the problems we developers face in our day-to-day jobs.

When should you not use keyshade?

You might have got the intuition that this platform is pretty useless if you are a solo developer, working on a project locally. Keyshade also isn't helpful if your are building CLI applications, or mobile applications, or anything that would be running on an external environment (the client's device).

How keyshade employs the use of sockets in distributed environment

Rajdip Bhattacharya — Mon, 06 May 2024 17:30:21 +0000

Any web application that wants to integrate real-time responsiveness into their operations, will surely use web sockets in some way or the other. Socket programming is, nonetheless, difficult to implement, hence a library of softwares exist to ease our life a little bit.

But there are scenarios when you can't use these tools due to the lack of flexibility, and you need to manually write your socket code.

We were challenged with a similar problem at keyshade!

The problem

At the very core of our project, we have real-time notifications to client applications. This means, if your application is configured with keyshade, it will receive live updates regarding changes that happen in your configurations. This implies that your application will need to maintain a socket connection to our servers, which, by no means, is feasible (even if possible) to externalize our sockets. And so, our sockets are home baked.

But, the plot thickens further!

Adding to the problem

But before we could implement this, we were met by yet another (major) blocker. Consider this scenario:

We are having an API cluster that has 3 instances. This cluster sits behind a load balancer, through which all connections are multiplexed (which I have dropped deliberately to reduce complexity).

In this case, your client application (where you want to get live configuration updates) has connected to Server A. But, when you are making changes, they are sent to Server C.

This raises the concern about the scalability of web sockets in a distributed environment. As you can infer, sockets that are connected to Server A won't receive any updates from Server C.

This brings us to our next section: the solution.

The solution

After spending an entire day searching for the correct solution, I found none. There was not a single way on that I could "share" socket connections among various servers. So, we brought Redis' PubSub into the picture.

The very fundamental of this approach was this: whenever a server started up, we would make it register to the change-notifier topic of Redis.

async afterInit() {
  this.logger.log('Initialized change notifier socket gateway')
  await this.redisSubscriber.subscribe(
    'change-notifier',
    this.notifyConfigurationUpdate.bind(this)
  )
}

Next up, whenever a configuration's (secret or variable) value was changed, we would push an even to this channel:

await this.redis.publish(
  'change-notifier',
  JSON.stringify({
    environmentId: variable.environmentId,
    name: variable.name,
    value: dto.value,
    isSecret: false
  })
)

This flow allowed us to achieve an architecture that is both scalable and ensures high availability!

Contributing to the cause: Doing it the open-source way

Rajdip Bhattacharya — Sun, 24 Dec 2023 18:20:38 +0000

Hello there!

Lately, I have been getting quite a few messages about how one should get started with developing open source projects, or contributing to projects out there. This blog aims to address those queries with the fullest of my knowledge! I tend to clarify the what's and why's of open-source development before actually jumping into the action.

Happy reading!

Intro

One of the most important rule of software development (and any other kind for that matter) is this - Do not re-invent the wheel. This means, we should stay away from working on developing something that already exists, and should reuse it. Therefore, developers often find themselves using a plethora of tools along with their projects to make their work (and lives) easier!

But have you ever wondered where these tools come from? We surely don't pay for using any of these development tools (at-least most of them!).

Well, these code (and also the binaries, sometimes) for these tools are hosted in a free, available to all and developer friendly space called Open Source.

So, what's Open Source?

If you do a Google search, it will give you the following answer:

denoting software for which the original source code is made freely available and may be redistributed and modified

Now, if you have eagle eyes and can understand what that means, feel free to move on to the next section :D, For the likes of me (affinity towards elaborations), do follow on.

Breaking down the definition, it tells us these few things:

We can host our software code in a "space"
This "space" is free to use and available to the world
Our code (and the software) can be distributed among other people, and also be reused by them.

Now, I'll be honest, this is too much an abstraction to build our understanding on. So let us give some life to it.

The "space" that we have been referring to, can be any of the following: GitHub, GitLab, BitBucket, and many more. For this blog, we will stick to GitHub, since that's more beginner-friendly.

GitHub

GitHub is a place where developers can create their accounts and host their code, completely free of cost. Other users (and non-users) would also be able so see the code (until and unless you make it private). It is highly recommended that you check out their platform (and preferably create an account) before you move any further.

Now that we have the stage set up (literally!), we are ready to delve deep into the "why's".

Why use Open Source?

So why all of this fuss, you might ask. To be very honest, this was my first question when I was introduced to open source.

Setting up the scene: Before I started using Open Source to develop my tools, I used to do the following:

Write and save my code locally.
Whenever I had to make any major changes to the codebase, I used to make an archive of the existing code, give it some version number, and then get started with writing the updates.
I used to share the zip files with my friends (who were from non-CS backgrounds) and asked them to MANUALLY set it up using the instructions.
When it came to deployment, I used to upload the code to the server manually and make the deployment run there.
And lastly, I was storing all of these zip files in my Google Drive

Now, any novice can see that there is too much of a manual intervention required in all the steps. To be precise, here are the drawbacks:

Reverting the codebase is impossible if you don't create the versions MANUALLY.
Distributing your code is a cumbersome task.
Collaborating with your friends is next to impossible.
Visibility of code is as poor as the fog in winter.

Back then, I believed that these were more than enough reasons for me to switch my old habits and learn something new. Not to mention, in the longer run, I found the switch helpful in more ways than I could possibly imagine.

Here are a few solutions that Open Source development gives you for the above problems:

All open sourced software use Git, a tool (which is open-sourced as well) that helps you in keeping a track of all the changes, giving you the ability to roll back, reset and fast forwards to any point of time.
Since open sourced software are hosted free of cost, everyone is able to view your code.
Collaboration is a cakewalk: git gives you the ability to develop your features on separate branches, while keeping the rest of the code intact.
Additionally, it provides us with automated notifications, builds and releases if we decide to opt into it.
Many companies open source their software to create a level of transparency with their user base. This helps them to gain trust.

If it's still unclear as to why people choose to develop their software for free, out in the open: It is because of the support of developer pool that you get

Getting started

I'm quite sure that you must be wanting to get your hands dirty, now that you have came across all of it. But, before you could move any further, you should be familiar with the concepts of Git. It is the building block of every open source project. Now, since this blog isn't explicitly about git, I will attach a few links for you to get started:

Now that you are well versed with some of the git terminologies, let's proceed to the actual stuff!

Finding your repository

The first step of this journey is to find yourself a decent project. You can search these projects from any of the following links:

To be very honest, selecting a project can be very tricky. There are a multitude of ways to shortlist a project, but here are some ways that I use:

Look for the programming stack they use. Are you comfortable with it? Do you have any prior hands-on experience with it?
Does the project align with your interest? Is it a tool, or a platform? Does using it feels good? - This matters because the emotional attachment with a project will stir a natural will in you to contribute to it.
Is it being actively developed? - Active projects will give faster feedbacks, and your work will be known to the people.
Are there any active issues in the Issues tab? If no, you might consider moving on to another project.
Can you understand the documentation? - You would be expected to set up the entire project locally without any help (most of the time). So it is super important that you can understand the docs.

Getting accustomed to the project

After you have made a decision, it is now time to start developing. Almost all the repositories follow the same pattern, and I will outline it for you.

The flow

Fork the repository into your account
Clone the fork into your local device
Create a new branch like: feat/feature-name or fix/fix-name

git checkout -b feat/add-logout-button

Develop on the new branch

The first step is to set up the project. All projects vary in this step, but gladly there is a common thread:
- Go through the README.md file. Most of the time it contains info about quick setup.
- Go through the CONTRIBUTIONS.md file. This will contain detailed information about the setup, and might also provide links to other docs.
- Lastly, scan for a docs/ folder, this might contain all the help you will ever need.

The next step would be to familiarize yourself with the codebase. Most of the repositories use monorepos for organizing and managing their code. A rule of the thumb here would be to make yourself familiar with what component lies in which place. It is next to impossible to understand the entire codebase at once. For starters, you can:

Understand how the environmental variables are used
Where does the UI/API/SDKs live
Where does the project fetch its configs from

Making your contributions

Now, we are ready to head over to the issues of the project. Start small - pick up issues that have a relatively smaller scope of impact, like updating something in the docs, making minor UI fixes, making changes in the file names, etc. Some projects use the good first use label to track such issues.

After you have picked your issue, now it's time for you to get assigned to it. For this, consider joining the community of the project - Discord / Slack / Telegram. Many projects encourage you to ask the maintainer to assign you to an issue before you start working on it. Many ask you to put up a PR directly. Most of this is mentioned in the CONTRIBUTIONS.md file. Again, the rule of the thumb in this step is:

Search for an issue
Ask the maintainers to assign it to you over the issue comments.
Optionally, you can provide a suggested way to fix the issue (this increases your chances of getting assigned)
Once done, start following the code guidelines and doing what you do best!
Always remember, deciding on the features with the help of the maintainers is a BIG PLUS. Although, keep in mind that you should not spam them.

Submitting a PR

After you feel that the feature is working as expected, you should always add tests or update them. Making sure your tests are up-to-date and passing makes the merge process effortless. Before you commit the code, do make sure you have gone through the commit guidelines of the project. Almost all projects use this pattern.

Finally, it is the time that you make the pull request (PRs). This is how you submit your code. To do this,

Go to your fork
Make sure your code is not outdated
Select the branch that you were working on
Make the PR from under the contribute section.

This stage requires the least effort. Generally, if you have communicated your way through the implementation, your PR should be merged in a short time.

Your gain from open-source

We humans are outcome-oriented creatures. So, for every action that we do, we expect some sort a reward. And, in case of open-source there are "many" rewards.

You will be surrounding yourself with a like minded peer group who think and act alike. This will enhance your skill set and also, help you become a better human.
You get to showcase your merged PRs on your portfolio! They will give you an upper hand in your itnerviews.
You can get an offer from any of the project you are working on. Most of the open-sourced companies look for full time employees from their pool of open source contributors. Although a bit difficult to achieve, it is certainly possible!
Lastly, you can go for bounties! Yes, bounties! Once you have quiet familiar with contributing, head over to Algora and start getting paid for your work!

Conclusion

To wrap up, I would say that, open-source is an ability that only software developers have. You get the opportunity to work with top class engineers, developing industry grade applications, all while improving your network. No hassle of interviews, no rat-race, no nothing! All that's needed is your time and dedication.

P.S.

A note that I would like to leave here on behalf of all the maintainers out there, which I believe most will be in unison with - When a regular developer thinks about making himself a maintainer of a project, the responsibility comes with a great many challenges. That person is responsible for not only developing the crucial aspects of the project, but also in managing a swarm of developers. Not to mention the fact that, the maintainer choose to open source their product so that world can see and lend a hand in its growth (also, making it free to use). It is my request to all of you who's reading this, to treat them with the respect they are deserving of. Open source is not all about being good at coding, it is also about becoming a better human, and to develop a deeper sense of understanding and sharing responsibilites.

Open source, AI, and CodiumAI's PR-Agent

Rajdip Bhattacharya — Sat, 09 Dec 2023 06:10:14 +0000

Hola geeks!

Development, undoubtably, is a vast ocean. We can barely define a starting point and definitely no ending point. Like we all know, "Necessity is the mother of all inventions", our journey this far is no exception either. It all started with Sir Charles Babbage inventing the first computer, followed by a massive breakthrough when Sir Alan Turing developed the Turingery during WWII to decipher the Enigma code. This decade and the last have seen the creation of the most prolific systems and pieces of software that lay the foundation of modern day computing and software development. Throughout this entire journey, we have always been, and are, fascinated by one technology in particular - Artificial Intelligence.

In our day-to-day lives, we interact with multiple software that employ AI to ease our work. Not only that, developers have put in efforts to bring AI to our loved IDEs for our ease of development!

Undeniably, most of the development happen on GitHub. Every developer in his/her life has come across GitHub to look for a solution or look at the code of some of the greatest open-source repositories. GitHub says that there are a whooping 94 million developers who are actively contributing and developing in the open-source industry!

This brings us to an important concept - a PR

So what's a PR?

For any developer out there, who is a contributor to any OSS, certainly know about pull requests (PRs)

In case you heard this term for the first time (or perhaps you want a refresher), whenever we contribute any changes to an open source project, we do it via a pull request (PR). For doing this, you first write your code in a separate branch, then you head over to the repository to create a PR. This is done with the intention of giving the maintainers a glance into the changes you made and test the correctness/feasibility of your changes.

This brings us to a gradual conclusion that, for our PRs to be accepted, we need the maintainers to have a full picture of the changes we made. As humans, we are bound to make mistakes, the top ones being:

Inaccurate/incomplete description about the changes you made
Bugs/errors/potential security issues in your code base
And most importantly, the lack of energy to write a concise PR.

These errors are mostly fixable, but there's someone (some"thing") that could make our lives a lot easier: CodiumAI's PR-Agent

A quick glance into CodiumAI

CodiumAI is an AI based product that helps developers in doing their stuff. Their plugins sit inside your IDEs, helping you out in your code by doing a significant heavy lifting. They provide automated tests for your code, that not only makes your code more resilient, but also aids in ramping up your efficiency and productivity. And the best part is, CodiumAI is entirely open-sourced!

PR-Agent by CodiumAI

The PR-Agent is an AI-based Git plugin developed by CodiumAI. It helps us in our open-source journey by accomplishing the following:

Auto-Description: Automatically generating the description about the entire PR that includes name, summary and code walkthrough.
 PR Review: Automated smart feedbacks about possible errors or bugs in our code
Question Answering: Helps us in querying the different aspects of the PR.
Code Suggestion: Suggestions about possible improvement scope for our code to polish our PR.

Finally, the hands-on!

Skills are pointless until and unless you practice using them for your own good! And the fact that we are learning about an absolute game changer, this blog would be incomplete without a hands-on section.

We would take a wild ride doing the following:

Installing PR-Agent
Creating a pull request
Creating a detailed description of what the PR is about.
Tagging our PR
We will ask the PR-Agent to give us a review on our PR (oh, how badly I strive to get a review!)
Of course, we are bound to make mistakes. We will detect the errors in our code.
We will take a glance into how PR-Agent helps us in fixing the error.
Making further improvements by adding documentation.
Generating a CHANGELOG

You can follow this article throughout the blog to have a reference.

NOTE: Custom configs to be covered in the later half!

Installing PR-Agent

Here is the nice part about PR-Agent: It enables you to use the tool without even any installation. All you need to do is @CodiumAI-Agent <command> (commands to be discussed later), and it will do its magic!

But there are some drawbacks as well:

The first being, you have to put a mention tag (@CodiumAI-Agent) in front of every command.
Not all commands will work with the mention tag. That being said, it's safe to say that you get a sort of "trial" of the tool without installing it.

Now, to further extend its capabilities, an installation is necessary. For doing that, head over to this link

One important thing, before you proceed, make sure you are logged on to GitHub.

As you can see, it's free for individual developers! Next, scroll down and hit on Install it for free

This will take you to a billing page, where you have to enter your billing details. Rest assured, you won't have to add any banking details, nor will you be charged a single penny.

Once everything is successful, you would be redirected to the integration page of your GitHub profile.

In case you want to visit this page later on for some reason, you can do it by visiting Profile Icon -> Settings -> Integrations -> Applications -> CodiumAI PR-Agent

Creating the PR

There are numerous ways to get started with this. I won't be digging into creating the PR, but here's an article that will help you to follow along: https://docs.github.com/en/desktop/working-with-your-remote-repository-on-github-or-github-enterprise/creating-an-issue-or-pull-request-from-github-desktop

For the context, I have created a PR to my own portfolio repository, where I have intended to add some code to include my Twitter account in my portfolio. Again, the code base doesn't matter, since the PR-Agent works with any and every code base.

Here is a glimpse of the initial PR:

It looks pretty empty (and useless). Let's give it the shine it deserves.

Describing the scope of the PR. (/describe)

As you saw in the above image, our PR looks pretty barren. Creating PRs have a sole purpose of integrating the changes we made into the existing code base. For that to happen, we want our reviewers to have a clear understanding of what the PR does. Thankfully, our AI buddy makes it a breeze of an effort! Let's see how.

Head over to the comment section, and enter:

/describe

If you have set up everything correctly, Codium-AI will "see" your comment (literally). Notice the "eyes" reaction:

Now, the tool will do the following things:

Add a comment stating it has updated the PR description and title
Add the PR description and change the PR title

Tagging our PR (/generate_labels)

GitHub (and every other platform) provides us with the ability to search issues and pull requests using tags. Tags help us categorize the issues. Here is a glimpse of the PR-Agent issues in GitHub.

PR Agent gives us the ability to generate meaningful labels of our PRs. Here's how you do it:

/generate_labels

As you can see, the label enhancement was inferred from the changes we made.

This doesn't end here, as we can inform the AI to use custom labels for pre-defined changes. More on that later.

Feedbacks are important (/review)

Often times you will find yourself struggling to get a concise feedback on your work. Feedbacks enable us to further improve on our code and make it much more efficient. Along with it, you also get to know about the compatibility of the new features that you introduced with the existing code.

However challenging this might sound, fear not, because our AI partner has got our backs!

All you need to do is, type in:

/review

As you can see, the review cover a great details. It has even got the Estimated effort to review [1-5] scale that comes in super handy to the maintainers.

Now that we got a brief about the internals of the code, let's see if our code works as expected or not.

Looking for scopes of improvement (/improve)

As humans, we tend to make mistakes. The PR-Agent gives us a handle /improve to scan for possible errors / bugs.

We run this, (and all of the commands to follow) in the same way we ran the previous command:

/improve

As you can see, we do have issues in our codebase! I have mistakenly (deliberately) used the wrong URL for my twitter account.

Now, let's see how we can fix them.

Got questions? All you need to do is `/ask`!

Although we got the error, we still don't know what's causing it, or what we should do to fix it. In such scenarios, we can use the ask command. The usage of this command is similar to the previous one, just that it takes a question as the parameter:

/ask <your question>

So, let's ask our AI buddy "how should we fix this?"

And we see a neatly formatted and concise message, explaining us what we should do to fix our issue.

Now we can fix the errors, and can make a push.

Scope for adding documentation (/add_docs)

Documenting our code, although might seem boring, but is an extremely important task in software development. Generally, any software that gets written, has a bulk of documentation associated with it.

Often times, while implementing some fix or adding some feature, we miss out on docs. PR Agent helps us in finding those spots where we can add documentation.

We do this by:

/add_docs

As you can see, the tool gives us a bunch of things that can be documented.

Now, I am that lazy developer who never writes the docs. So I'll as my AI buddy to do it for me :D

And as you can see, it gave me the things I need to change, all I need to do is copy and paste.

Generating a CHANGELOG (/update_changelog)

Every project has a file called CHANGELOG.md. In case you are wondering what that is, it is a ledger that keeps track of all the changes that were made in a specific version. It mosly contains these things in this format

[version]

- things added
- things fixed
- things removed
- anything else

Changelog is an important component that helps us in visibility, hence it becomes extremely important to manage it properly.

PR Agent helps us in this by generating a CHANGELOG for all the changes made in the particular PR. We do it by:

/update_changelog

We do have a small change that we made, and hence the log file is small, but this does demonstrate us the capability of the tool in assessing our file!

Custom configuration - going a step further

We have already crossed two big steps:

Using the commands via @CodiumAI-Agent mention
Using the commands via local installation

While we feel we might have everything that we need, this approach might not fit bigger projects. Such projects have a more "focused" way of tracking their changes, and might want the PR-Agent to be customized as per their wishes.

But fear not, because we do have complete authority over how and what CodiumAI PR Agent does!

The local installation comes with a pre-defined configuration file (.pr_agent.toml) that hosts all the settings that PR Agent will use. This article explains you how you can get a customized PR Agent at your fingertips.

Since the article covers a lot, and the configurations can seem overwhelming at the beginning, I will try to break it down to you by demonstrating how to generate custom labels based on the changes you made (now we're getting at it!).

One important note, you might have to close your existing PR before your do this. PR Agent expects you have your configurations in place before you make the PR. (Gah... sorry for breaking it to you this late).

Customize your AI buddy

We will get started by creating a .pr_agent.toml file in the root directory of the project. Note that this should be the main branch of your project.

All the custom configurations are mentioned in the individual tool guides

We want to do the following:

Label our PR as Link Update if any link is updated in our code
Label our PR as Code Update if any code changes.

Head over to the custom labels guide to see what all things you can change. We will insert these configs into our .pr_agent.toml file:

[config]
enable_custom_labels = true

[custom_labels."Link Update"]
description = "Whenever any link in the code is updated"

[custom_labels."Code Update"]
description = "Whenever any code is updated"

Let's break this down.

[config]
enable_custom_labels = true

This tells the PR Agent that we want to use custom labels. What this means to the tool is, besides using its own rules for evaluating our PR labels, it will also look throught the .pr_agent.toml file to find user-defined rules for generating labels. User-defined labels have higher priority, which mean, given 2 rules - user-defined and pre-defined, PR Agent will use our rule for generating the labels.

The next two blocks have a simple format:

[custom_labels."Assign this label"]
description = "When this condition satisfies"

[custom_labels."Link Update"]
description = "Whenever any link in the code is updated"

This block tells the PR Agent to assign the Link Update label whenever any link in the code is updated.

[custom_labels."Code Update"]
description = "Whenever any code is updated"

This block tells the PR Agent to assign the Code Update label whenever any code is updated.

Once you have committed the changes to the main branch, you can open up another PR to test these settings.

Let's give it a run:

/generate_labels

As you can see, we were successful in generating custom labels for us.

Closing thoughts

PR-Agent is certainly a game changer in the world of open source (and also closed source). It is an all-in-one package for developers and maintainers, allowing us to lift the weight of manual review and documentation off our shoulders. Doing this gives our productivity an extra polish, thereby leading us to develop with speed and accuracy.

The fact that the plugin is open-source and free of cost is a cherry on the top, setting no bounds for developers.

But why stop in using the tool? Why not contribute to the cause? Jump right in!

Automating Python Deployments with GitHub Actions, AWS ECR, and AWS Lambda

Rajdip Bhattacharya — Sat, 14 Oct 2023 07:38:03 +0000

Greetings everyone!

Welcome to the second part of this blog. As promised earlier, I told you that I'll be setting up a deployment pipeline using GitHub Actions for our Python code. And here we are!

A brief revisit

For those, who haven't read the previous blog or would just like to brush up their knowledge, here is what we have built so far:

A python code base specialized for a lambda function that extracts a field called name from the lambda event and returns us a greeting.
A Dockerfile for containerizing our code.
AWS ECR for hosting our image.
AWS Lambda for hosting our function.

Additionally, we employed the use of Terraform as IaC to deploy our AWS infrastructure.

Roadmap

This part covers just a single, but a critical aspect - creating the build and deploy pipeline for our Python code. Here's what we are going to do:

Create a GitHub repository
Write the code for our pipeline
Configure the secrets in our repository
Push the code, sit back, and watch the magic in action

You can find the code for this blog in here.

So, let's get started!

Creating a GitHub repository

I shouldn't be saying this, but, you do need to have a GitHub account to continue further. Once you have it, go here.

I'll be using the above configurations to get started.

Next, you should have a brand-new repository ready at your disposal!

Building our pipeline

This is perhaps the most critical part. For getting started, we need to create a file in our project as:

.github/workflows/ecr-lambda.yaml

The folder naming convention is necessary since GitHub treats all files under the workflows folder as a pipeline.

Next, we create the pipeline.

on:
    push:
        branches: [master]

name: Lambda ECR Deployment

jobs:
    deploy:
        if: github.ref == 'refs/heads/master'
        name: Deploy to ECR and update Lambda
        runs-on: ubuntu-latest

        steps:
            - name: Checkout
              uses: actions/checkout@v2

            - name: Configure AWS credentials
              uses: aws-actions/configure-aws-credentials@v1
              with:
                  aws-access-key-id: ${{ secrets.ACCESS_KEY }}
                  aws-secret-access-key: ${{ secrets.SECRET_KEY }}
                  aws-region: ap-south-1

            - name: Login to Amazon ECR
              id: login-ecr
              uses: aws-actions/amazon-ecr-login@v1

            - name: Build, tag, and push the image to Amazon ECR
              id: build-image
              run: |
                  # Build a docker container and push it to ECR 
                  echo "Registry: ${{ vars.ECR_REGISTRY }}"
                  aws ecr get-login-password --region ap-south-1 | docker login --username AWS --password-stdin ${{ vars.ECR_REGISTRY }}
                  docker build -t ${{ vars.ECR_REGISTRY }}/${{ vars.ECR_REPOSITORY }}:latest .
                  echo "Pushing image to ECR..."
                  docker push ${{ vars.ECR_REGISTRY }}/${{ vars.ECR_REPOSITORY }}:latest
                  echo "name=image::${{ vars.ECR_REGISTRY }}/${{ vars.ECR_REPOSITORY }}:latest" >> $GITHUB_OUTPUT

            - name: Update AWS Lambda function
              id: update-lambda
              run: |
                  aws lambda update-function-code --function-name ${{ vars.LAMBDA_FUNCTION_NAME }} --image-uri ${{ vars.ECR_REGISTRY }}/${{ vars.ECR_REPOSITORY }}:latest

I'll explain what this code does step-by-step. It is recommended that you copy and paste the code into an IDE to follow along the description, as I will be using line numbers.

Lines 1-3 specify which branch this pipeline would activate on. Right now, the pipeline will be activated only when we push to the master branch.
Line 5 states the name of the pipeline
Lines 7-42 define the jobs that will run
Line 8 defines the only job that will be run by this pipeline.
Line 9 ensures to run this pipeline only when the code is pushed into the master branch.
Line 13 lists the steps to be followed in this job
Line 15 uses the community GitHub checkout action to check out the repository on the pipeline whenever a new push is made.
Lines 18-26 tries to log into our AWS account using the credentials provided in lines 20-22.
Lines 31-37 builds, tags, and pushes the docker image to the ECR repository.
Line 42 updates the lambda function's code to deploy the most recent version of our code.

Configuring our GitHub repository

Note that in our workflow file, we have used some special syntax like ${{vars.ECR_REGISTRY}} and ${{secrets.ACCESS_KEY}}. These are GitHub Action Secret and Environmental variables. Using this approach gives us two benefits:

We don't need to change our code when some configuration changes.
Our secrets stay safe when it's not in the code.

To add your own secrets and environmental variables, head over to your repository and then go to Settings->Secrets and Variables->Actions. Here, we find the following:

Next step, add the secret (ACCESS_KEY, SECRET_KEY) in the Secrets tab and the rest in the Variables tab like this:

There are some limitations on using the Variables though, you can read about them from here.

Showdown

Now that we have all the bits in their places, we are ready to jump into action.

To do this,

Initialize a git repository in your project
Make sure you refactor/remove the access and secret keys from your Terraform config (Only if you have been following since the last blog).
Push the code.

After a successful push, you can head over to your repository and click on the Actions tab. Here, you will find your pipelines.

Now, whenever you make a change to your python code, your changes would be automatically deployed into AWS Lambda when you push your code into the repository!

Conclusion

With that, we come to the end of this blog. I hope you have enjoyed reading through this and didn't actually fall asleep. I hope to bring more such blogs related to DevOps very frequently, so, I would encourage you nice readers to ping my up with ideas that you would like to see in action! Until the next time, happy hacking!

Building Serverless Python Apps with AWS Lambda and Docker

Rajdip Bhattacharya — Mon, 09 Oct 2023 14:52:38 +0000

Greetings geeks!

Welcome to another blog on how to get your stuff done (the easy way)! In this blog, we will be taking a look at how to deploy dockerized Python code on AWS Lambda. Not only that, we will be setting up our AWS environment using Terraform to reduce the chances of blowing up our servers without our knowledge. So, let's get started!

Why Lambda?

This question is not uncommon. In case you have felt it as well, you have come to the right place! When we use Lambda functions in place of traditional EC2 servers, we are taking a big leap into the world of serverless computing. This technique enables us to focus only on the business logic of our product, while the cloud provider does the heavy-lifting of setting up the environment for us.

An use-case of Lambda

AWS Lambda has a special use case in the industry. Consider this simple example:

This diagram demonstrates a caption generation API that does the following (The abstraction of intricate details is deliberate):

Take in a video
Store the video
Generate its captions
Store the caption along with the video

There is one design flaw in here. We might have a case that the Upload API might have authentication and authorization to filter out who can upload videos. But we can easily point out that, the video generation doesn't require any authentication, i.e, it can operate autonomously. This is where lambda can help us. We are also increasing the workload on Upload API in this flow. The next diagram shows the optimized design:

As you can see, we have separated the caption generator to another component called Caption Generator. You can think of this as a component. It does the following:

Retrieve the video from storage (eg- S3)
Process the video and generate captions
Return the captions.

Through this design, we achieved the following:

Implemented separation of concern and single responsibility principle.
Reduced workload on Upload API.

Now, you might be wondering, how do you actually host this Caption Generator. Well, you have your answer - AWS Lambda! Lambda functions are meant for exactly such use cases, where you can isolate autonomous functionalities into separate services and host them. The Upload API can then invoke the Caption Generator when it will need to.

Our roadmap

Implementing the above service can take quite some amount of coding. To keep this tutorial simple, we will implement a Hello World function for starters.

Create Python code for the Hello World function.
Dockerize the code.
Initialize Terraform
Create ECR.
Deploy our code.
Create Lambda function.
Test our function.
Cleaning up.

You can find the code for this blog in here.

Did you say Python?

Yes, Python! Perhaps one of the easiest languages for getting started. Lambda supports a lot of runtimes. Do check them out if you are interested. Here is a glimpse:

First, we need to create a file named lambda_function.py. The naming convention is necessary since we will be using a base image that utilizes this name.

Short note on using containerized build.

Lambda supports three sources of code:

Single file code: used for smaller use cases such as testing and demonstration and is not often recommended for production environment.
Zipped up code from S3: Generally used along with pipelines where the artifacts from the build stage of the pipelines is uploaded into a S3 bucket and the lambda function uses this archive.
Containerized code from AWS ECR: Perhaps the most stable one, this gives you the flexibility to use custom runtimes with the help of Docker. The code is built into a docker image and is pushed into ECR which is then used by Lambda.

The base image that we will be using dictates us to name the entry file as lambda_function.py and the entry function as handler. Hence, the names.

# lambda_function.py

def handler(event, context):
    name = event["name"]

    return {"statusCode": 200, "message": f"Hello {name}!"}

The code is pretty simple. It extracts a parameter called name from the lambda event and returns a JSON response. We will come to the event format at the end.

Docker up!

That's step one done. Next, we move towards dockerizing our code.

Create a file named Dockerfile in the same directory with the following contents:

# Dockerfile

FROM public.ecr.aws/lambda/python:3.10

# Copy function code
COPY lambda_function.py ${LAMBDA_TASK_ROOT}

# Set the CMD to your handler (could also be done as a parameter override outside of the Dockerfile)
CMD [ "lambda_function.handler" ]

Initialize Terraform

For starters, Terraform is an open-sourced tool developed and maintained by HashiCorp that allows us to build our cloud infrastructure from code. This approach is popularly known as Infrastructure as Code(IaC). If you would like to read more, I suggest you go through their docs (and perhaps some YouTube?).

Before proceeding forward, make sure you have these prerequisites checked:

Now, we are ready to get started with terraform.

First, create a folder named terraform in the root folder of your project. This isn't a rule, but will allow you to properly organize your code files.
Next, create these files in the folder:

# terraform.tf 

terraform {
  required_providers {
    aws = {
      source  = "hashicorp/aws"
      version = "~> 4.0"
    }
  }
}

provider "aws" {
  region     = var.region
  access_key = var.access_key
  secret_key = var.secret_key
}

This file is used to hold all the configurations required by our project. To simplify, we will be using AWS, hence we mentioned aws in the required_providers block. Next, we set up our aws provider, where in we mention our AWS settings.

# variables.tf

variable "region" {
  default = "your_aws_region"
}

variable "access_key" {
  default = "your_aws_access_id"
}

variable "secret_key" {
  default = "your_aws_secret_key"
}

This file contains the AWS option values. It is usually a good practice to extract all sensitive values from your terraform code into the variables.tf file and inject the values at runtime to enforce and maintain security of your code.

Once done, run the following:

terraform init

This will initialize your project with the dependencies.

Create ECR

With the previous step done, we are ready to move into the actual stuff. Now, we will be creating our repository (AWS ECR). This will allow us to host the custom python image that we will be building shortly. Add this file inside the terraform folder that you created just a while back.

# ecr.tf

resource "aws_ecr_repository" "greetings_repository" {
  name                 = "greetings-repository"
  force_delete         = true
  image_tag_mutability = "MUTABLE"
}

Note that the name of our repository is greetings-repository. You can change it if you want to.

After that, we will want to spin up our repository:

terraform apply

This command will create your ECR. You can check it from your AWS Console by going into Search > ECR > Repositories > greetings-repository

Deploying our code

Now that we have our repository set up, we are ready to build and push our code. To do this, click on the View push commands button on the top right of the repository page.

Go into your root directory of the project and run the commands one by one.

After running the commands, your output should look something similar to this. To confirm you have pushed the image, you can go into your repository in ECR and verify that it's there.

Create the Lambda function

Before we deploy our lambda function, we need to create a Lambda execution role that our lambda function will use. This allows the function to operate on its own.

# role.tf

data "aws_iam_policy_document" "assume_role" {
  statement {
    effect = "Allow"

    principals {
      type        = "Service"
      identifiers = ["lambda.amazonaws.com"]
    }

    actions = ["sts:AssumeRole"]
  }
}

resource "aws_iam_role" "iam_for_lambda" {
  name               = "iam_for_lambda"
  assume_role_policy = data.aws_iam_policy_document.assume_role.json
}

Here, we created a role called iam_for_lambda that we will attach to our lambda function.

Let's buckle up and get that function deployed.

# lambda.tf

resource "aws_lambda_function" "greetings_function" {
  function_name = "greetingsFunction"
  role          = aws_iam_role.iam_for_lambda.arn
  package_type  = "Image"
  timeout       = 10
  image_uri     = "${aws_ecr_repository.greetings_repository.repository_url}:latest"
}

We did quite a few things in there, namely:

Create a lambda function named greetingsFunction
Attach the iam_for_lambda role that we just created
Specified that our lambda function will be using an image as its source.
Specified the greetings-repository url.
Specified a timeout for the function.

With that, we come to the finale. All we need to do now is, run:

terraform apply

After this, we should see our lambda function up and running! You can check it from your AWS Console via Search > Lambda > greetingsFunction

Testing the lambda function

In case you forgot, our lambda function is extracting a key called name from the event and returning us a JSON in the form of:

{
    "statusCode": 200,
    "message": "Hello <name>!"
}

To test the function, we will be using the console. Head over to your lambda function and go into the Test tab. Scroll down to find a text editor. Paste the following content in there:

{
    "name": "John Doe"
}

Once satisfied, click on the Test button. You should see the following content if your execution was successful:

Congrats! You have successfully created a serverless, containerized python application! Don't be so harsh on yourself, give yourself a pat on the back!

Cleaning up

It's always a good idea to dispose off your resources when you don't need them. Gives us the ability to delete everything with just once command. So, when you feel that your are done playing with, run:

terraform destroy

Conclusion

In this blog, we had a deep dive into how you can set up your own container, deploy code to lambda, and create a fully managed AWS environment. In the next part, we will be looking at setting up a CI/CD pipeline using GitHub Actions that will allow us integrate changes into the lambda function at ease. Till then, happy hacking!

SOLID Design Patterns for Spring Boot Developers

Rajdip Bhattacharya — Tue, 05 Sep 2023 18:47:05 +0000

Hello geeks!

Welcome to yet another new blog on the tidbits of software development. In my previous blogs, I have touched upon topics such as Docker and System Design. While these topics are no doubt a good to have skills, what is more important is how you write your code. Imagine this scenario where you have a top-class secure pipeline set up for your application, all ready to go into deployment, your manager is super happy with you, and he/she hires more people to assist you in your work. But now, they are introduced with a nightmare since your code isn't clean. No, I'm not saying that you don't know how to code, all I'm saying is, even if you do, you might not be following the world-class SOLID Principles!

This is the core motive of this blog, to introduce you to this magical concept of clean coding and make you stand apart in the crowd! So, let's get started without further ado.

A note: Most of the references in this article has been taken from the book Clean Architecture by Robert C. Martin.

What is SOLID?

Before we start, make a note of this: The essence of design and architecture is that, it never gave us anything new, but always took from us what we already had.
Let me simplify this. We live in a society where we are bound by some laws enforced upon us by our government. Violating any such law would impose a penalty upon us. What laws essentially are: barriers. Laws draw the line between the dos and don'ts of our society. This is absolutely necessary because us humans have the capability to do absolute wonders in the planet. To maintain the stability of our society, laws are our only hopes.
Similarly, to maintain the sustainability and developability of code, coding best practices are the only way forward.

Now, to begin with SOLID, it focuses on arranging data and functions into classes. Note that in this context, class doesn't necessarily mean the class we know from OOP, but rather containers that allow us to containerize our code.

Goal of SOLID: Creating mid-level software programs that:

Tolerate change
Are easy to understand
Can be easily used in many software systems.

The term mid-level signifies that we use SOLID principles while working at module level. To further simplify, we use solid principles while developing, for example, business logic and APIs for our software.

As you might have already guessed, SOLID is a short form. It consists of:

(S)ingle Responsibility Principle (SRP)
(O)pen Closed Principle (OCP)
(L)iskov Substitution Principle (LSP)
(I)nterface Segregation Principle (ISP)
(D)ependency Inversion Principle (DIP)

We will look at each of the constituents and analyze them in the following sections.

Single Responsibility Principle (SRP)

First, let's see what we do in general. Consider that we have a class named UserService. It does the following things:

Register a user
Send promotional mails

Upon registration, a user will receive a mail from us denoting their successful registration, followed by a promotional email.

While this might look ok, it isn't optimal. We can clearly spot two different functionalities stashed inside the same class. What this will do is, any changes in the registerUser() function will also recompile the promotionalMail() function. Plus, if there were two teams working on the two different functions, a merge conflict is inevitable.

For this very reason, we would like to use two separate classes for this: UserService and EmailService. This obeys the following principle: A module should be responsible to one, and only one, actor. Actor, here, means the changing factor. In simple words, the reason for which a class needs to change. The lesser the changing factors for a class, the more stability it gains.

The main takeaway of this principle is: make sure that you segregate your code based on the factors that changes them.

Open Closed Principle (OCP)

While the name might sound intuitive, what it really means is, we want to develop code where in to make changes to our code, we would essentially extend the code rather than updating the existing code.

Consider the following code:

@Service
public class ShoppingCartService {

    public double calculateTotalPrice(List<Item> items) {
        double total = 0.0;
        for (Item item : items) {
            total += item.getPrice();
        }
        return total;
    }
}

The calculateTotalPrice() calculates the total price of our order based upon the cost of each item. Later, let's say we need to implement a discount mechanism in the code where the discount percentage is variable.

One way to do this is to directly insert the math in the same function. While we can give ourselves a pat on the back for doing this so simply, we might be digging our own graves in the longer run.

The optimal way to solve this problem is by creation another service class that will calculate the price for us based upon the discount. We can then segregate our code based on the SRP, which narrates us to separate our code based upon the actors affecting it.

We are going to do the following:

Create a new interface named DiscountStratergy with a single method named applyDiscount

public interface DiscountStrategy {
    double applyDiscount(double totalPrice);
}

Create an implementation of this interface named PercentageDiscount

@Component
public class PercentageDiscount implements DiscountStrategy {

    private final double discountRate;

    @Autowired
    public PercentageDiscount(@Value("${app.discountRate}") double discountRate) {
        this.discountRate = discountRate;
    }

    @Override
    public double applyDiscount(double totalPrice) {
        return totalPrice * (1 - discountRate);
    }
}

Hook up this interface to do the calculation for us in ShoppingCartService

@Service
public class ShoppingCartService {

    private final DiscountStrategy discountStrategy;

    @Autowired
    public ShoppingCartService(DiscountStrategy discountStrategy) {
        this.discountStrategy = discountStrategy;
    }

    public double calculateTotalPrice(List<Item> items) {
        double total = 0.0;
        for (Item item : items) {
            total += item.getPrice();
        }
        return discountStrategy.applyDiscount(total);
    }
}

Wallah! We now have SRP, OCP and DIP & LSP (coming soon) in place. Now, let's say we choose to move from percentage based discount to loyalty based discount. All we need to do is create another implementation of this interface and comment out the @Component tag in the current implementation. This approach also makes sure that, for any change in the discount implementation, there aren't any changes in the ShoppingCartService class.

Liskov Substitution Principle (LSP)

We have already covered the usage of LSP unknowingly, but let's look at it properly in this section.

Let's take an example of the Linux kernel. Don't worry, you don't need to be a Linux geek to understand this example! Linux uses files to represent everything, be it a device or an actual file. This enables the kernel to use uniformity across its code for doing I/O based operations. Namely, there are 4 functions that Linux use to operate on file or I/O devices: open(), read(), write(), close(). What this means is, any I/O device that is attached to the Linux kernel, must provide their own implementations for these methods. The kernel doesn't bother itself with the details of the implementation. This is what LSP enforces.
LSP promotes the use of interfaces instead of concrete definitions. This allows relaxation of code and lesser dependency.

Here is an example of the interface promoted by the Linux kernel.

import java.io.IOException;

public interface FileDevice {
    void open(String filename, String mode) throws IOException;
    int read(byte[] buffer) throws IOException;
    void write(byte[] data) throws IOException;
    void close() throws IOException;
}

Any device that wants to talk to Linux should implement these methods (It is the device drivers that do this and not the device. Kept it simple for ease of understanding).

Interface Segregation Principle (ISP)

The name already tells us what it does. ISP tells us to not couple classes together that are not used.

In the above diagram, we can see that the 3 User classes directly depend on the OPS class. Note that User1 uses ops1 method, and the same goes for the rest 3. As you can see, it makes a clutter if stash unrelated code. We solve this issue by using interfaces to segregate the use.

This diagram demonstrates how we can change the hierarchy of classes and interfaces so that each module talks to another module that is made for just a specific use case. The green boxes are interfaces.

Dependency Inversion Principle (DIP)

The last and perhaps one of the most tricky principles to understand. Let's expand on the previous examples of using interfaces in place of classes. Following this approach helps us to not depend on concretions, but rather, abstractions. If we depend on concrete classes, any change in the class would also mean recompilation of dependent classes. Not only that, this approach makes the code highly dependent and reduces code reusability. This is what DIP puts emphasis on.

DIP dictates that the most flexible systems are those in which source code dependencies refer only to abstractions, not to concretions. Abstraction, in the context of OOP, can mean interfaces or abstract classes.

These abstractions are of classes that are highly volatile. A point to note is that, it is perfectly normal to depend directly on classes that do not change often or at all (eg. String in Java).

The bottom line of this principle is, never ever depend directly on volatile classes.

Let's take the example of JPA repository.

This is the dependency graph we follow when we are creating a simple CRUD application using springboot.

The Service class contains business related logic that uses the JPA Repository to do its work.
The JPA repository provides an interface of the functions that might be used by its consumers
The DB provides drivers that implements these methods to connect all the dots

Here is a simple java code for the same:

public class UserService {
    private final UserRepository userRepository;

    public UserService(UserRepository userRepository) {
        this.userRepository = userRepository;
    }

    // Rest of the code
}

Note that the userRepository object is injected by SpringBoot at runtime.

Conclusion

With this, we come to the end of this blog. I'm sure if you have read so far, you would have found it really helpful. As developers, it's our responsibility to maintain the software we develop. There is no better way to do this than starting from such a granular level.

Once again, in case you feel I have missed something, please do let me know.
P.S. You should definitely get Clean Architecture if you liked this article :D!

Effortless Containerization: Deploying Spring Boot and MySQL with Docker and Docker Compose

Rajdip Bhattacharya — Thu, 17 Aug 2023 08:03:17 +0000

Greetings!

In the increasing complexity of software development, there is a general shift from manual configuration to automation. Be it development, or deployment, seamless development and integration is the word of the town.
Hence, in this blog, I plan to crack open a critical aspect of DevOps: Containerization.

What is Containerization?

Before we dive into that, first let us understand the old school way of doing it. Make a supposition that you are developing an application. You know every inch of this application. Be it the databases or the environments, you have everything set up. Now consider that this application is done with development. You now want to host it someplace. For instance, let's say you decided to use Amazon EC2 for its ease of use. Now let's look at the steps that you would perform to get this application running (at least).

Clone the application into the EC2 instance
Install required dependencies (Java, NodeJS, etc.)
Set up the required environmental variables
Test the application for bugs or errors.

While this might look trivial, it becomes a pain when you have to do it over and over again, for all of your applications. There is always a chance that some OS feature would break your application, some dependency won't be installed, some environmental variable might not be configured. It becomes extremely difficult to debug such applications. Here is where containerization comes to our rescue.

A container includes everything needed for an application to run: the code, runtime, system libraries, and settings. This self-contained unit ensures that the application behaves consistently regardless of the environment it's deployed in. Containers are lightweight, portable, and can be easily moved between different host systems or cloud platforms without significant modifications.

Key features and benefits of containerization include:

Isolation: Containers are isolated from each other and from the host system, preventing conflicts between dependencies and runtime environments.

Consistency: Containers ensure that an application behaves the same way in every environment, reducing the "it works on my machine" problem.

Portability: Containers can be moved between different systems or cloud platforms with minimal effort, making application deployment and migration easier.

Resource Efficiency: Containers share the host OS kernel, resulting in lower overhead compared to traditional virtualization.

Scalability: Containers can be rapidly scaled up or down to accommodate varying workloads and demands.

Version Control: Container images can be versioned, allowing teams to track changes and roll back to previous versions.

DevOps Enablement: Containers facilitate DevOps practices by enabling continuous integration, continuous delivery, and automated deployment.

Roadmap

In this blog, we would be doing the following things.

Create a simple SpringBoot application to manage a Person entity
Set up MySQL using Docker
Create a Docker Image of our application
Develop a run-and-deploy of the entire infrastructure using Docker Compose

To follow along, clone this repository (also leave a star maybe?)

So, let's get started!

Creating the SpringBoot application

We would start with our SpringBoot application. Head over to Spring Initializr and create a project using the following settings and dependencies.

To summarize, we are using:

Lombok: For annotation based POJOs.
Spring Web: For our web application.
Spring Data JPA: For Hibernate ORM
MySQL Driver: Enables our application to talk to a MySQL database.
Spring Boot Actuator: For health checks

Once you are satisfied, generate the project, extract it, and open it with your favourite code editor.

We would be creating the following classes:

Person: Base entity for holding the person.
PersonPayload: Contains the request body for a Person object.
PersonDTO: Contains the DTO of the person object when the API returns data.
PersonService: Contains business interface for managing persons.
PersonServiceImpl: Contains the implementation of PersonService.
PersonController: Exposes the endpoints for managing Persons.
PersonRepository: Enables us to use JPA

Here are a list of endpoints we would be developing in PersonController:

POST /api/person/: Creates a person
PUT /api/person/{personId}: Updates a person with the given data
GET /api/person/all?page=<page_index>&size=<page_size>: Gets the list of all persons on the database. page and size helps us to control the index of a page and number of items in a page in pagination.
GET /api/person/{personId}: Gets a person by their ID
DELETE /api/person/{personId}: Delete a person by their ID

Our final folder structure should look something like this:

Since this blog is focussed on getting Docker set up, I would be skipping the code explanation in here. You can always clone the repository and check the code.

Let's go through the application.properties file

As you can see, I have injected environmental variables. This would allow us to resolve the values at runtime. It gives us the flexibility to configure our application without actually touching any of the code. Any change that you would like to make to the environments, all you need to do is tweak the values as per your wish in the system's environment.

I will be using a .env file to feed the values into the application. Nearly every IDE has the support for doing so. In case you can't figure out how to include a .env file into your execution environment, you can try setting those values in the environment of your OS. Alternatively, you can let that untouched, since all the keys have a default value assigned to them.

.env.docker: Used when deployed using docker compose.
.env.local: Used when using just docker.

Now, we have our application ready. Before we are actually able to run it, we need to launch MySQL, which we will do next.

Launching MySQL using Docker

To begin with this step, first, make sure that you have docker installed. This article provides an excellent guide in letting you set up docker. Once it is set up, we are ready to move forward.

We will be launching a MySQL docker container using this command:

docker run --name springboot-test -p 3306:3306 -e MYSQL_ROOT_PASSWORD=root -d mysql

Here is a breakdown into what this command does:

It creates us a container from the mysql docker image
It assigns a name to that container using --name springboot-test
It exposes the standard mysql port of the docker container to the host's network using --p 3306:3306
It sets the root password of the docker image to root using -e MYSQL_ROOT_PASSWORD=root
Lastly, it tells the container to run in detached mode, meaning, it won't be attached to the console where we are writing this command using the -d flag

With that, we have a brand new mysql container up and running which you can check using:

docker ps

At this point of time, we are ready to launch our springboot application. Go to the root of the project, and run

If you have maven installed:

mvn spring-boot:run

If you don't have maven:

mvnw spring-boot:run

You can verify that the application is running using:

curl http://localhost:8080/actuator/health

Now we are ready to finally dockerize the application!

Dockerizing the application

We use Dockerfile to containerize any application using docker. This article provides all that you need to know to get started. For starters, I'll run down through the docker image that I'm creating.

Go to the project's root
Create a file named Dockerfile
Paste the following content in the file

# The base image on which we would build our image
FROM openjdk:18-jdk-alpine

# Install curl and maven
RUN apk --no-cache add curl maven

# Set environment variables
ENV DB_HOST=${DB_HOST}
ENV DB_NAME=${DB_NAME}
ENV DB_USER=${DB_USER}
ENV DB_PASS=${DB_PASS}

# Expose port 8080
EXPOSE 8080

# Set the working directory
WORKDIR /app

# Copy the pom.xml file to the working directory
COPY pom.xml .

# Resolve the dependencies in the pom.xml file
RUN mvn dependency:resolve

# Copy the source code to the working directory
COPY src src

# Build the project
RUN mvn package -DskipTests

# Run the application
ENTRYPOINT ["java", "-jar", "target/application.jar"]

Notice that I'm still not hard coding the environmental variables. Also, note that, the last line mentions using the command java -jar target/application.jar to launch the container. For this to happen, we need to first set the build name to application in the pom.xml.

To optimize the docker build process, I have first copied the pom.xml and then resolved the dependencies before actually copying our soruce code. This is done with the purpose of reducing the number of layers docker rebuild during its build process. Source code is bound to change often. Putting that at the very top would mean all the subsequent layers would be rebuilt.

Docker networking

Before we get started with running the application, let's first get a few points right about networking in docker. When we run a docker image, the container boots up into a separate docker network that works in isolation to our host network and other docker containers. Hence, container A can't ping container B if they are running on different networks. In our case, we would be running the springboot application and MySQL database. So if we let them run in different networks, our containers won't be able to intercommunicate.

There are two ways to address this issue:

Create a custom network and attaching our containers to that network.
Attaching the containers directly to the host network.

Using custom network

Let's start by creating a docker network

docker network create dummy-network

Once done, you can verify this using

docker network ls

Now, we need to migrate our MySQL container to this network. We do this by:

docker network disconnect bridge springboot-test
docker network connect dummy-network springboot-test

Now, we can verify that these commands work by inspecting the Containers section in the output of this command:

docker network inspect dummy network

Using host network

Recall that we used the flag -p 3306:3306 while creating our MySQL container. This flag creates a channel in the network of our MySQL container that allows us to communicate with the container's 3306 port via our hosts 3306 port.

We can bypass this by instructing docker to run the container directly on the host's network. This can be done by:

docker run --name springboot-test --network host -e MYSQL_ROOT_PASSWORD=root -d mysql

Notice that I replaced the -p flag with the --network flag. When we are using the host network driver, port mappings are neglected by docker.

Now that we know the fixes, let's move towards making the application work.

Running the SpringBoot application

We have created our Dockerfile in the previous sections. Now, first, we need to create a docker image out of that file. To do this, go to the root directory of the project and run:

docker build -t application:latest .

Next, run

docker run --name temp --rm -p 8080:8080 --env-file .env.local --network dummy-network  application:latest

This command will do the following:

--name temp will set the name of the container as temp
--rm will remove the container once it's stopped
-p 8080:8080 will map the container's port 8080 to the host's port 8080
--env-file .env.local will read the environments from the .env.local file
--network dummy-network will associate the container to dummy-network
Lastly, it will launch the docker container

In case you get any error stating that the DB connection fail, I would like to point you to the .env.local file. In there, we have a key called DB_HOST with the value set as springboot-test. This is the name we used when launching the MySQL docker container. The above command will only work when these conditions are satisfied:

The MySQL container is named springboot-test
Both the database and the application are in the same network.

Alternatively, if you want to use some other name for the MySQL container, you can should the name in the .env.local file aswell.

Now that we have everything up and running, we can verify our network again using the docker network inspect dummy-network command.

Using docker compose

No doubt that most of you have felt that this is too much of configuration. Yes, configuration comes as the cost of making applications reliable and secure. But don't feel demotivated, docker compose is here to rescue!

docker compose is a plugin that reads deployment configurations from a file (typically named docker-compose.yaml) and deploys the entire infrastructure at one click!

For doing this, let us first create the docker-compose.yaml file in the root directory of the project.

Then, paste the following into the file.

version: '3.8'
services:
  mysql:
    container_name: mysql
    image: mysql
    ports:
      - "3306:3306"
    environment:
      - MYSQL_ROOT_PASSWORD=admin1234
    networks:
      - stack
    healthcheck:
      test: ["CMD", "mysqladmin", "ping", "-h", "localhost", "-u", "root", "-padmin1234"]
      interval: 30s
      timeout: 10s
      retries: 3

  application:
    container_name: application
    build:
      context: .
      dockerfile: Dockerfile
    ports:
      - "8080:8080"
    env_file:
      - .env.docker
    networks:
      - stack
    depends_on:
      mysql:
        condition: service_healthy

networks:
  stack:
    name: stack
    driver: bridge

As you can see, we have created a network named stack. We have created two services - application and mysql. Both of these services come under the stack network. In the .env.docker file, I have set the DB_HOST to mysql. This name corresponds to the name of the service. In the docker file, I have added a dependency of mysql in application service. This means that the application service wont start before the mysql service reaches the service_healthy state.

Once done, shut down your previous containers using:

docker stop springboot-test
docker stop temp

Now, we can fire up the entire infrastructure using:

docker compose up

This command will take some time to start up. A few flags that might come in handy:

-d: Starts in detached mode
--build: Rebuilds the images. Useful when you have made any changes to the source code.

When you are done playing around with, you can shut down the entire thing by using:

docker compose down

Conclusion

So that was all about using docker to make your lives easier. I hope you have quiet a few tricks by now. Feel free to leave a comment in case you find something off.

How to develop a scalable, resilient and highly available URL shortener

Rajdip Bhattacharya — Wed, 09 Aug 2023 20:20:36 +0000

Greetings everyone!

In the world of software development, we often face an overwhelming amount of information. Every application has its complexities hidden beneath the surface, which can be difficult to understand upfront. When I started my career in development, I was mainly focused on learning programming languages and familiarizing myself with a specific tech stack. Over time, I began to realize the importance of understanding the inner workings of applications. This shift in perspective occurred as I observed the performance of my applications in real-world scenarios and recognized the need to dig deeper into the details. I realized that to date, I was only staring at the tip of the iceberg.

In this blog, my intent is to present to you my thought process while developing an application. I will try to guide you through the critical aspects of system designs, how to assess your application and finally, how to arrive at a happy showdown. This blog might seem boring/irrelevant to some of the readers because this is purely theory-based. Don't worry, I'll be coming back with the practical version of this blog soon. But if you are willing to understand the details, this would be the one-stop shop for you.

Okay then, let's begin!

Roadmap

Here is the list of things we will be doing in this blog:

Get introduced to URL Shorteners
Decide upon the implementation of the idea
Take a deep dive into its working
Figure out the working of the backend

If you have come this far, I'm sure you will enjoy this:")

What are URL Shorteners?

Have you ever gone to Facebook profile via the web? Did you ever try to share your LinkedIn profile? If yes, then you can certainly feel where this is going. If not, though, let me tell you.

Here is the link to my GitHub profile: https://github.com/rajdip-b

This link might look like it's easy to memorize. Well, it is. But, if we ramp up the URL, let's say I show you my LinkedIn URL, you would get the idea.

Here goes: https://www.linkedin.com/in/rajdip-bhattacharya-581119232

As you can see, it's no joke to remember this URL. This is where URL shorteners come into play. You can think of URL shortness as a mathematical function that takes a string as input and generates another string based on the input. Let's see this in action with an example:

Notice how the big URL gets shortened into a much smaller URL. Now that we know what a URL shortener does, let's see how it does it.

Setting our goals

To make sure that our application stands out in the crowd, first, we need to make sure that we have a clear conception of what we are trying to build. It is in this step that we outline our application.

Working overview

First, let's see how our application will function.

The above diagram sketches out the working of our URL Shortener. To reiterate,

The client sends a request to our server to generate a shortened version of a URL.
The server generates this URL, stores it in the DB (which will be covered in the next sections) and returns the shortened URL.
The client, later on, requests the shortened URL to our server.
Our server sends a 301 Permanently Moved\ response along with the location.
The client gets redirected to the actual server (in here, google.com).

Note that in step 4, we are sending 301 Permanently Moved\. In case you need to brush up your knowledge, here is a great resource to learn more about HTTP status codes. There are two status codes that we use for redirecting:

301 Permanently Moved
302 Found

There is a subtle difference between the two:

301 states that the resource has moved permanently. Browsers are programmed to cache such responses so that when we make subsequent requests to this URL, we are redirected to the actual location instead of the former location.

302, on the other hand, merely redirects us. This means the browsers won't cache these responses, and we will always be hitting the URL that sends us the redirected location.

To summarize, considering that we have requested a server that gets us redirected, for the subsequent requests,

301: Browser -> Our Server -> Actual Location
302: Browser -> Actual Location

Where "Actual Location" can be thought of as the long URL.

Planning

Next stop, we will make some assumptions on the use case of our application. This can be considered the most crucial point in any application design because implementation and resource allocation highly depend upon the output of this state. So here they are:

Capacity: 100 million URLs generated per day
Uptime: Should be running for 1 year at least.
Data: URLs containing alphanumeric characters.
Data modification: Just writes are allowed.

From the given design, we can infer that our application would have more reads than the number of writes. Let's say, reads:writes would be 10:1

Calculations:

Writes per second: 100 mil / (24 * 60 * 60) = 1160

Read operations: 10*1160 (since 10:1 is read:write) = 11600

Number of records that needs to be stored: 100 mil * 365 = 36.5 billion records

Considering an average of 100 bytes per record, the storage required for 1 year: ~= 33 TB

From the above calculations, we can infer the following requirements:

A database that can scale at large. Hence, scalability is ensured.
The database should have great indexing support for decreasing delay.
Caching the most frequently accessed URLs would decrease delay.
Creating multiple servers and load balancing would make the application resilient.

Designing the heart of the URL Shortener

Now that we have established the requirements, we are ready to move forward. The next stop is deciding the function that will actually generate the shortened URL. Note that we will have to store 36.5 billion records, which means we will have 36.5 billion unique shortened URLs. Therefore, we need to find out the minimum size of the shortened URL. We know that we will be operating on alphanumeric input, which has a size of 62. Since our shortened URL would also contain alphanumeric, we will have 62 different characters in 1 position and 62^n different combinations, where n is the size of the URL. So if our URL is of size 4, then n=4. So the number of combinations becomes 62^4 = 1,47,76,336 ~= 14 million. Therefore, we need to find an n such that 62^n >= 36.5 billion. I'll spare you the calculation. The lowest n that satisfies this equation is 7. Therefore, our shortened URL would be size 7.

Next, I'll be discussing 2 approaches to solving this problem.

Hash Function

A hash function can be defined as a complex mathematical function that takes in an input (hashValue) and generates a hashOutput based on the input. SHA256, SHA512, MD5 are some renowned hashing algorithms to name a few.

The above diagram illustrates SHA256 converting a string into absolute gibberish. Behind the scenes, the hash function uses mathematical functions to map characters to certain calculated values. The output is very long and according to our requirements, we only need 7 characters. We can truncate the output to be 7 characters long, but then it will mean that we might have duplicate elements in our records. To mitigate this, we will use hash + collision detection.

The diagram illustrates how we will be generating our shortened URL. Note that the output shortUrl\ has 7 characters, and this is done by truncating the rest of the output.

Base 62 conversion

Base 62 is a commonly used encoding technique that uses 62 characters-long set. This set consists of all characters in the alphanumeric family. For generating a Base62 short URL, we follow the following steps:

Character Set Selection: We map all the alphanumeric characters to decimal numbers, as represented in the above table.
Converting to Decimal: The entire long URL is scanned through, and each character is replaced by its decimal representation
Conversion to Base62: The output of the previous step is continuously divided by 62 and the remainder obtained at each step is mapped back from the table. For example, at any step, we got a remainder of 20, then would write K, because the table maps alphanumeric K to decimal 20.
Padding and truncating: The output obtained from the above stage might not be always 7 characters long. If the output length is lesser than 7, pad it with 0s else, truncate it if it's greater than 7.

The steps involved are much similar to that of hashing, just that we don't use any hash function but encoding functions.

With our URL Shortener function defined, we are ready to move forward to the "web" aspects of our application.

Deciding the backend

Nearly any application is always backend-heavy. In our case, we need to figure out how we want to put the pieces together to make our URL shortener functional.

Firstly, we want to establish how our DB operates.

The records in the database would be of the format:

id: number (auto generated)
shortUrl: string (7)
longUrl: string

Next, comes the high-level architecture of the overall application.

The client requests the server requesting for the long URL for a short URL.
The load balancer receives the request and sends forwards it to one of its healthy servers in the target group.
The server first checks the cache. If it is present, it returns the short URL to the client (5).
If it is not present in the cache, the server queries the DB and fetches it. It stores it in the cache and sends it over to the client(5).
This is a 301 Moved Permanently response, that will ultimately make the browser go to the destined location.

Conclusion

With that, we've concluded this blog. While it delved into theory, it has provided you with a clear understanding of the inner workings of such applications. What's even better is that you're now equipped to put this knowledge into action and apply the thought process to your projects!

This blog might not be 100% accurate, so, if you see any window for improvements, you're always welcome!

Logical Volume Manager: A Beginner's Guide

Rajdip Bhattacharya — Mon, 07 Aug 2023 10:14:24 +0000

Hey there, everyone!

I'm super excited to kick off my very first blog! Today, I want to take you on a journey into the awesome realm of Linux Volume Manager (LVM). Don't worry if you're new to this – I'll start from the very basics and walk you through the entire process of creating and managing your own LVM. And guess what? I promise to keep things fun and easy to understand. So, buckle up, and let's embark on this exciting adventure together! Ready? Let's go! 😊

Prerequisites

Before we get started, we would need a few tools at our disposal.

🖥️ A Linux machine with root access (VMs would suffice as well)
📀 A USB drive (preferably 8 GB and above)
☕ A cup of coffee (can't miss this one!)
🐧 Some understanding of Linux is always a plus!

Physical Volumes

As the name suggests, we would be dealing with volumes, to be precise, storage. For us to understand LVM better, we would need a firm understanding of how Linux stores data without LVM in the first hand. Let's start by inspecting how a regular hard disk drive or solid state drive (block devices)

The diagram above illustrates how a physical device or block device is partitioned and used by our Linux machines. At the top, there's a partition table, which is just a reflection of how the device is internally partitioned. The next three blocks represent the partitions, named in ascending order for easy understanding. Each partition has a filesystem, a special database that stores the file and folder hierarchy details**. Common examples of filesystems are ext4, NTFS, and FAT. Don't worry; we'll dive into filesystems and partitions in the hands-on section later!

Linux organizes its file system like a tree structure, with the / directory as the root. All files and folders reside next to it. In our setup, partition 1 is mounted to /, partition 2 to /tmp, and partition 3 to /home. When we store data in any directory, it goes into the corresponding partition.

However, using physical volumes directly comes with some problems:

When the / directory runs out of space, extending it can be tricky.
Incorporating a new storage device with your /home directory requires adjustments.
Replacing a slow storage device is a cumbersome task.

That's where LVM comes to the rescue! It simplifies these processes and provides more flexibility. Let's explore LVM further in the upcoming sections.

Okay, so, what actually is LVM?

LVM, which stands for Logical Volume Manager, becomes more understandable now that we have a basic understanding of storage devices and their utilization. LVM acts as an intermediate layer, allowing us to manage our storage seamlessly. It adds flexibility and simplicity to the storage management process, making it easier to handle partitions and filesystems. With LVM, we can efficiently allocate, resize, and move logical volumes, providing us with a more dynamic and adaptable storage solution

As the diagram shows, LVM utilizes a Volume Group (VG) to abstract the Physical Volumes (PVs), which represent the raw storage devices like HDDs and SSDs. The VG is typically created with one or more PVs, and from this pool of storage, Logical Volumes (LVs) are carved out. The beauty of LVM lies in its ability to freely adjust the size of these LVs, thanks to its dynamic data management on the physical volumes. In a sense, the VGs act as a versatile and flexible data pool.

You may have noticed the block labeled /boot. This partition holds the data related to our system's boot process. At boot time, Linux cannot read data from the LVs directly, so a separate raw /boot partition is necessary.

It's important to note that not all LVM-based systems offer the same level of flexibility. Some systems may restrict us to just one PV and two LVs (for swap and root), limiting the benefits of LVM's true potential. However, when we have the liberty to leverage LVM's full capabilities, it opens up a world of possibilities for efficient and customizable storage management.

Operations supported in LVM

LVM supports the following operations:

Add more physical volumes (PVs) to volume group (VG)
Remove PVs as long as there is enough space to accomodate the existing logical volumes (LVs) into the VG.
Resize LVs

Enough chit-chat, let's get to the real stuff!

Now that we have the theory out of the way, let's get our hands dirty by actually creating a LVM on our systems. We would be doing the following stuff:

Partition our USB drive to have 2 partitions (2 GB + ~6 GB).
Create a VG from 1 partition and, later on, extending it with the other partition.
Displaying the details of the VGs.
Create 2 LVs (1 GB + 2 GB)
Display the LVs
Create file systems on the LVs and mount them to our systems
Extend the 1 GB LV by 2 GB and re-adjusting the file system.
Removing the VGs and LVs.
Fixing out pendrives.

Just to make a note of it, some commands that I will be using might not be already present on your distro. In that case, please be sure to read the error messages and install the required binaries.

So turn on your Linux machine, get a sip of your coffee and get started!

Partitioning our USB drive

Just so we don't actually blow up our computers trying to do anything fancy, we would do this experiment using a USB drive. I am using an 8 GB HP stick for this. Just make sure to back up any data that is present in the USB drive and then we are good to go.

The first step is to partition the USB drive.

Make sure you are in a root shell. If not, use

sudo -i

Next, let's grab some details about our block devices.

lsblk

I have truncated the irrelevant details from the output. You can notice that there are 2 devices called sda and sdb. And under the TYPE column, you can also see that these are...disk. But wait, where did these names come from? Let's find out how.

Block devices use SCSI (Small Computer System Interface) to talk to processes on one side and hardware on the other side. Hence, block devices have device names (since everything on Linux is a file, so device files) starting with sd*, where s = SCSI and d = device. During boot time, the kernel detects block devices one by one and name them in the form sd[a-z]. So in my case, my vmware disk got detected first and hence, it was named sda. Next, after boot, I attached my USB stick and the kernel named it sdb. As I mentioned earlier, all disks have one or more partitions. In the above output, sdb1, sda1, etc. are names of partitions.
Note that USB devices use SCSI as a layer of abstraction over its USB interface. That's the reason why it gets detected as block devices.

We can fetch more details about our block devices using this command:

fdisk -l /dev/sdb

Now, we would partition the device into 2 halves. For that, we first need to unmount our device:

umount /dev/sdb1

To verify that the device is unmounted, you can run the following command:

mount | grep sdb

The output of this command should be empty.

Next, we will use fdisk to manage the partitions. Here is what we will do:

Delete the existing partition
Create a new partition of 2 GB
Create another partition of the remaining size
Save the changes (fdisk doesn't save the changes on the go)

To start, type the following command and refer to the output for more:

fdisk /dev/sdb

Commands used:

d: Delete existing partition
n: Create new partition
p: List all partitions

Once done, you can verify the newly created partitions using:

lsblk /dev/sdb

Creating a volume group

Now that we are done with partitioning our USB drive, it's time to create our brand-new VG.

Before running any of the commands below, make sure you have installed the lvm2 package:

sudo apt install lvm2 -y

Next, we create our VG:

vgcreate myvg /dev/sdb1

Listing VGs

We can list the existing VGs in our system using the command:

vgs

vgdisplay

The first command lists all the VGs in a compact manner, while the second command gives us a detailed view.

VG: Name of the volume group
#PV: Number of physical volumes associated with the VG
#LV: Number of logical volumes inside the VG
#SN: Number of snapshots of the VG
VSize: VG Size
VFree: Amount of unallocated space on the VG

If you have come this far, give yourself a pat on the back and have a sip from that cup of yours!

Adding PVs to VG

We want to add our /dev/sdb2 partition to myvg aswell.

vgextend myvg /dev/sdb2

Notice that if we list out our VG now, we would see that its size has changed:

vgs myvg

Creating logical volumes out of `myvg`

Our VG has been created. Now, we want to create volumes out of it so that we can start using it.

We will create a 1 GB volume named mylv1:

lvcreate --size 1g --name mylv1 myvg

And similarly, we will create another LV of size 2g, named mylv2:

lvcreate --size 2g --name mylv2 myvg

If we try to see our VG config now, we will see that the VFree has reduced now:

vgs myvg

Listing LVs

We have 2 LVs created and we would like to see how they look like. The command is similar to that of listing VGs:

lvs

Using our LVs

Okay, so now we have 2 LVs created and now we want to use them. Lets see how we can do that.

Previously, we saw that the kernel detects devices during boot time and assigns them the names accordingly. This means that our LV "device" names will also vary with each boot. Initially, the kernel creates the logical volumes under /dev folder with a naming convention dm-*, where * is an integer. To make sure we can map our LVs by the name we know them to these dm-* device files, the kernel also creates symbolic links under the /dev/mapper and /dev/<vgname> directories.

Contents of /dev/mapper:

Contents of /dev/myvg:

Now that we know what device files to use, let's actually create a file system:

mkfs -t ext4 /dev/myvg/mylv1

This command creates an ext4 file system on our mylv1 LV. Now its time to mount it to a location.

First, I'll create a mountpoint:

mkdir /mnt/mylv1

Then, I'll mount the LV:

mount /dev/myvg/mylv1 /mnt/mylv1

Next, I'll verify my mount point using:

mount | grep mylv1

As we can see, everything is set up exactly how we wanted it to be. If I were you, I would have tried to do the following things:

Do the same for mylv2
Take a break
Take another sip from my cup

Using our newly created LV

First, we need to move to our mount point:

cd /mnt/mylv1

Next, we can inspect the size using:

df -h .

Lastly, we can start creating files. I'll leave this to you. But remember, this space is created by the root so no other user would have write access to it. If you want to give the users write access, then run chmod a+rw . while you are still in that directory.

Resizing the LV

Remember our mylv1 LV just has 1 GB of storage. We now would like to ramp it up to have 3 GB. Lets do that.

lvresize -L +2g myvg/mylv1

This command adds 2 GB of storage to mylv1.

But now if we run df -h /mnt/mylv1, we notice that the size didn't change. Did we make a mistake? No.
Remember that file systems are the managers of every partition? When we created a partition on mylv1, it's size was 1 GB. Just increasing the volume size doesn't mean that the filesystem would also be synchronized. So, to do that, we need to run the following command:

fsadm -v resize /dev/myvg/mylv1

Try running df -h /mnt/mylv1 and you will see that the size has increased.

lvresize provides us with a -r flag to automatically do the file system resizing for us. So, the command would be:

lvresize -r -L +2g myvg/mylv1

Removing LVs and VGs

We have almost reached the end of this long blog now. Before we finish, we would like to perfrom a clean-up.

First step is to unmount the LVs:

umount /dev/mylv1

Make sure you unmount mylv2 if you did mount it earlier.

Second step is to delete the LVs:

lvremove myvg/mylv1
lvremove myvg/mylv2

And at last, we will be deleting the VG:

vgremove myvg

I made the mistake of removing a LV before unmounting it, so I got an error.
After this command, you can issue vgs and lvs to confirm that everything is deleted.

Fixing our USB drive

Since I have already talked about partitioning and file systems, I won't be reiterating over them.
To get started, do:

fdisk /dev/sdb

Wallah! Your USB would now be fixed. With this, we come to the end of this blog.

Conclusion

In conclusion, Linux Volume Manager (LVM) serves as a crucial tool for flexible storage management. It abstracts raw storage devices (PVs) into logical volumes (LVs) within volume groups (VGs). LVM empowers us to resize LVs dynamically and handle storage with ease, providing a versatile storage solution. While some systems may limit its potential, utilizing LVM opens up possibilities for optimizing storage usage and system performance. Embrace the power of LVM in your Linux journey for seamless and efficient storage management. Happy computing!

DEV Community: Rajdip Bhattacharya

Design Decision for Role-Based Access Control (RBAC)

A bit about RBAC

Whom this article is not for

RBAC Designs

The Design

Design Outcomes

The Least Privilege model

Managing Secrets in Dev Tools Without Getting Yelled At

So what's this blog about?

But why does it really matter?

And now, the solution!

Keyshade?

But why would I store MY SECRETS somewhere else?

Demo time!

Install the CLI

Setting up a profile

The NextJS project

Create a project

Creating secrets and variables

Adding NEXT_PUBLIC_API_KEY

Adding NEXT_PUBLIC_PORT

Seeing the magic in action

Closing thoughts

How keyshade employs the use of sockets in distributed environment

The problem

Adding to the problem

The solution

Contributing to the cause: Doing it the open-source way

Intro

So, what's Open Source?

GitHub

Why use Open Source?

Getting started

Finding your repository

Getting accustomed to the project

The flow

Making your contributions

Submitting a PR

Your gain from open-source

Conclusion

P.S.

Open source, AI, and CodiumAI's PR-Agent

So what's a PR?

A quick glance into CodiumAI

PR-Agent by CodiumAI

Finally, the hands-on!

Installing PR-Agent

Creating the PR

Describing the scope of the PR. (/describe)

Tagging our PR (/generate_labels)

Feedbacks are important (/review)

Looking for scopes of improvement (/improve)

Got questions? All you need to do is /ask!

Scope for adding documentation (/add_docs)

Generating a CHANGELOG (/update_changelog)

Custom configuration - going a step further

Customize your AI buddy

Closing thoughts

Automating Python Deployments with GitHub Actions, AWS ECR, and AWS Lambda

A brief revisit

Roadmap

Creating a GitHub repository

Building our pipeline

Configuring our GitHub repository

Showdown

Conclusion

Building Serverless Python Apps with AWS Lambda and Docker

Why Lambda?

An use-case of Lambda

Our roadmap

Did you say Python?

Short note on using containerized build.

Docker up!

Initialize Terraform

Create ECR

Deploying our code

Create the Lambda function

Testing the lambda function

Cleaning up

Adding `NEXT_PUBLIC_API_KEY`

Adding `NEXT_PUBLIC_PORT`

Got questions? All you need to do is `/ask`!

Creating logical volumes out of `myvg`