DEV Community: Adrian Piętka

Monorepo - when to choose it for your project? 5 heuristics worth considering

Adrian Piętka — Thu, 09 Feb 2023 10:54:00 +0000

Only a few years ago, I kept everything that was connected to the project in one repository. But then dividing projects into smaller parts became the norm. It was a fascination with microservices - every service was put into a new repository.

This approach caused several problems like versioning and compatibility, duplication of the code, dependent implementations, and multiple coding standards, even though we kept all the repositories as one, etc...

Nowadays, more and more frequently, we go back to the monorepo pattern - only one repository. It’s not in pursuit of the trend, rather for me, it’s an additional technique that I can use when it will be useful.

Should we choose the monorepo pattern in a new (or already existing and under development) project? In this article, I will try to explain five heuristics worth using for discussion and decision-making.

But be careful!

Don’t instantly use this list with your team! My intention is to give you inspiration for preparing your own list. Then to develop it, transform it. In the end, various conditions can affect the decisions.

Usually, it’s not easy to answer the question “is monorepo suitable for my project?”.

With the below list, you’ll be a step ahead when you don’t know where to start.

Do I have technological consistency?

During the requirements analysis and architecture design phases we are able to see the boundaries between particular parts of the project very quickly. It can occur for various reasons eg.
different business context another scaling model a specific way of data processing

This is the moment to extract particular parts of the project and map them to applications with specific purposes. It’s also worth asking yourself a question at this moment...

Is it still going to be the same programming language?

If the answer to this question is yes, then we have technological compliance. It’s important information and if we have it, it’s worth keeping the applications together.

Going further, it’s worth asking another question...

Is it still going to be the same framework?

If the answer to this question is yes, then we have perfect technological compliance.

In this case, we will probably be using the same tools: CLI applications (supporting code generation), package manager, statistical analysis of the code tool, the code formatting tool, etc...

There is no need to introduce the same configurations or change them later for a few repositories. Let’s keep them together.

Do I share the code between applications?

Let me tell you a story. A project that I was a part of started struggling with sharing API interfaces. We wanted to be common and reflect the current API state.

In the beginning, we could have taken the submodule principle. The ones we know from Git. But it’s still a separate repository that you need to take care of. Many times someone forgot to introduce an appropriate fix after the changes in API... It simply didn’t work.

Later on, code fragments appeared that were so big they could have been external libraries. It’s still the same problem. We add a new function to the library and we need to update it everywhere to the newest version.

Some new Merge Requests, appropriate scaling order… Eh… Pain in the neck.

Private libraries, published and applied on the basis of external dependencies generated more general service than the value added from the function.

When I see code duplication between applications I usually want to extract it to the independent library. The idea is good but not without, often unforeseen, flaws. Cycling of versions, backward compatibility (because we don’t want to use a newer version yet), and bouncing up versions in applications that use the library. It’s quite a lot of things to take care of.

Monorepo gives a faster service for this kind of shared code. Especially if we are the only ones using it. An actual duplication shouldn’t be in a library that is used outside our project.

It’s just a reusable code in our system. Nowhere else.

Do I have a similar implementation model (CI/CD)?

In certain moments a few of our applications are implemented into a test or production environment.

Perhaps the implementation process is identical and for example, it may consist of the below steps (example strongly generalized) for all of our applications:
Build a Docker Image with an application
Publish an image in Docker Registry
Update service to the new version in Amazon ECS

When the implementation process matches between applications, then we have a good control point. We will be able to use automation scripts again, without the need for copying them between repositories.

One Jenkins Job was able to implement every application from a big system. A joint process for all of the applications was prepared. Thanks to that, when a change in the process appeared (eg. adding a notification on Slack) then it was immediately used in every implementation.

Do I need to sustain compatibility between applications?

Sustaining compatibility between communicating applications (with API, and queues) is not an easy challenge. When a new structure of data appears or the existing one changes, there is a need for adequate preparation - eg. supporting new and old structures in a temporary manner.

The matter may be even more complex when the structure isn’t dependent on us and it needs to be suited to a cycle.

Implementation of the all necessary changes between projects at the same time becomes easier. For example, adding one necessary box. We can deliver everything at the same time and be certain that the actual state of the code as it allows us to use the change in all of the applications.

Is there only one application development team?

I worked for the teams that were sustaining polyrepo for the project. Every microservice had its own repository.

Maintaining attention with constant changes being introduced by the team was exhausting. I had seven repositories that I needed to watch. I was constantly looking for changes, updating each of them, rebuilding the application, or installing new packs…

Not to mention introducing new people to repositories. I can remember the look on their faces when there was a need of installing seven different applications so that the project can operate in its entirety. Sometimes there was no need to do that but as a team, we each looked after all of the applications, so in the end everyone could program everywhere else.

In monorepo it’s easier to organize the automation of that kind of process. Installing the project, starting it, or updating it after the implemented changes, go much smoother. Especially when we use the tools supporting working with monorepo eg. rush.

Monorepo - is it a good idea?

Remember that every project, team, or business expectation affects decision-making. If after a few months of work, we start to feel uncomfortable working with monorepo / polyrepo then it's worth having retrospective thoughts and thinking of improvements.

These problems can always be solved differently. It can occur that CI/CD implementation looks a little different, but it’s still worth putting the application in with the rest in monorepo.

Sometimes creating suitable scripts or doing an additional pipeline in CI/CD is all that it takes. And sometimes we are gonna be forced to change our approach.

My propositions are not set in stone. They are points to be discussed. Their main goal is to encourage team talks about it.

And what do you put attention to? Do you have any control points?

Code development guide: working with small chunks

Adrian Piętka — Fri, 18 Mar 2022 08:47:48 +0000

Final work with the code can be a real pain in the… neck. It is often a long and tedious process of going through and verifying all the changes which can bring you to a moment when frustration is directly proportional to the time spent on writing the code. Is there a way to tidy up the work and avoid those tears under the shower at the thought of another implementation? Yes, there is. You can work with small chunks.

What’s the prob, dev?

Imagine waking up after tens of minutes (or even hours!) of coding trance and seeing 50 changes in your editor. Changes were made in totally different places of the application.

You look at this code-battlefield and ask yourself *the same question the Ant-Man did. *“What the hell happened here?”... To make matters worse, not everything is working and you have a lot more to do because you poked your thumb in too many pies. Part of the changes has to be verified again and it will pull further changes. On top of that - you haven’t corrected even one test.

You know that means one thing - a need to go through and verify all the changes, remove the unwanted code, and fix the tests. Well, you could probably say that you’re starting iteration number 2. The problem is, you are tired and you lean towards leaving work for tomorrow. After all, you did a great job today. Isn’t it enough? ;)

Eventually, that approach drives me to the moment I don’t feel like working at all. It’s like complaining about other people’s work and making a much worse mess than the others. Frustrating, isn't it?

That’s why I try to organize my process of creating software by the little approach. Small, but thought through and effective steps.

Let me guide you through all the stages of the work and point out its important elements.

Let’s start with a good plan

Think over carefully how to approach the problem to finish the task as a whole. Focus only on your goal and only those changes that will lead you to achieving it.

What’s important - I rarely pay attention to the tasks I’ll be working on next week or in two weeks' time. I focus on it when the time is right. Not once and not twice I had prepared implementation for the future changes and finally, there was no need of doing all that work because requirements changed in the meantime. All the work that was done turned out to be a waste of time.

It’s hard because you’re often overwhelmed with tasks. You hear about SOLID, DRY, and the rest of the acronyms that make the whole thing messy and abstract. You end up wasting your time instead of just working on a specific piece of code.

This is the moment when you need to start making decisions. For instance, you need to decide if you’re implementing the strategy pattern or not. But on top of all that is if in the future you will be able to easily change/ replace the elements you are currently working on. Without deleting half of your code. I focus on coupling the changes with the rest of the code.

I tend to point out the work phases on a sticky note. After completing each task I can cross it out. But the real moment of little glory comes when all the tasks are completed and the crushed paper lands in the bin.

Of course, you can use the virtual sticky notes and have the satisfaction coming from clicking the delete button… but frankly, it’s just not the same ;)

The plan isn’t a carbon copy with the changes I’m implementing. It is because you can subdivide them into even smaller steps.

It is my high-level work plan and not the plan for the particular changes.

One change at a time

This is where it gets interesting. After preparing the plan it’s time for a serious fight - decomposing the work into smaller pieces.

Examination of borders and drawing a line where I start and finish the work is just a heuristic. In that matter, I rely on two factors.

A type of change is the first factor. I’ve defined a catalog of 5 different potential changes:

Change in the file and directory structure - changing names and moving them
Refactorization - preparing the ground in the accordance with Uncle Bob’s 'Clean Code' principle
Implementation - the most atomic and independent part of the function you’re working on
Fixing an error
Deleting an unnecessary, unused part of the code

I always try to make only one change at a time. The exception to this rule is when the changes are mutually exclusive, for example - changing the file name that fixes the error.

Unfortunately, there is a trap here because it’s impossible to describe decomposition in just five easy points.

Second factor. I verify if the change meets the assumptions below:

To be atomic
To be as independent of the future changes as possible. What I mean - it doesn’t require something that doesn’t exist yet
To be dependent on things I added earlier. What I mean - it can use previously added functions
To change one specific function
To apply only one specific file (ideal situation)
To easily describe the changes in just a few words (it makes creating commit messages a lot easier)
To start from the deepest changes, the ones at the end of the calls chain

Let me show you an example:

I add a function to calculate the X. It’s not used anywhere yet (new MathCalculations function)
I add a new endpoint that I’ll be taking actions in (only Controller action and route)
I add the validation of data coming to an endpoint
I add business logic (e.g. in UseCase)
I connect triggering logic with the endpoint

Theoretically and practically I am able to merge my changes with the main development branch. Nothing should happen because various elements are independent of the next steps I plan to take.

This is, of course, just an example and we may find that to point 4 we would need to add a new entity, repository, or generate a new table in the database. I would think about these as separate changes.

It’s important to remember that theoretically, one single change can require dividing it into many smaller pieces and each one of them follows exactly the same process. I was thinking about 3 changes… and I ended up making 7 :)

Decomposition is the key part here. Either you have it or the process itself won’t get you anywhere.

Earning the ability to do that won’t always mean it is going to be easy (but also no one promised it will be ;) ). You often need to know the system you’re modifying, its architecture, code structure, and the main principles it follows. Without them, working little by little is a lot more difficult. What’s important is that each new task is knowing the code and how the software was built. This is what makes us move around much quicker and with higher confidence.

In this step, I’m not afraid to use comments // @ TODO. But only to make my work in the next step easier and finally to replace it with an appropriate implementation.

Everything needs to work

Everything needs to work after the changes you made. Every added, deleted line or modification can’t have any side effects, e.g. damaged automated tests, defective SCA, or malfunctioning software. Everything needs to be compiled correctly.
What’s the next step? To check if everything works. I verify not only the app but also everything needed to implement the next change in the environment.

I tell more about the changes testing approach in my 'Programmer, Test!' article. You’ll find a list of points worth remembering in the process of changes verification.

Write it down!

Preparing the storytelling about changes in the right words is the final step. Remember, you need to explain what you have done and why.

I add files to the Staging Area (git add)
I make the Git Commit and the created history I put into the Commit Message

This is where I can make the next step. Part of the work I consider done has been secured and I can safely take the next step in this case

What Have I achieved?

First of all, using only one tool (git) I can track/monitor the progress of my work. Even if I had a difficult weekend due to my friend’s bachelor party I could still easily remember what I had done on Friday. And after getting rid of the hangover I am able to quickly get back to work on Monday morning… ;)

And what’s important - I can easily undo my experiments. All I need to do is to get back to a specific commit.

In my opinion, it’s important to take care of proper history creation used in Commit Message. You get this feeling of sweet harmony in your work. You know what you changed and most importantly why you had done it.

But on the other hand, if you don’t care about this you can use Git Squash to simply merge a few commits in one. That way you will describe the final result, not how you achieved it.

Obviously, as always, the right approach depends on the project you’re working on or simply your preferences.

Do I push every commit created?

It depends. Sometimes you co-work with someone on the same branch and your partner is waiting for your changes. The other time you will just upload them from time to time.

Over time you will get this on your list of habits and you will realize that the work is getting done faster. Next thing you know you have a bunch of tasks marked with the green ‘done’ tick. In the end - practice makes perfect, or rather “practice makes practitioner” ;)

In the end, I would like to draw your attention to 'Commit Often, Perfect Later, Publish Once: Git Best Practices'. Actually, I think this is where my adventure with the little by little work and Git approach had begun.