Growing fast at RazorpayX
RazorpayX is a neobank for businesses that combines the best of B2B SaaS and B2B Banking, to make finance simpler and more efficient. Its state-of-the-art Current Account combined with award-winning Payroll, Corporate Cards, Vendor Payments, Tax Filing and Accounting solutions, helps businesses manage all their financial activities from one place
We experienced exponential growth at RazorpayX during the pandemic when online payments became the norm. With more businesses going online, business owners saw RazorpayX as their one-stop shop for all of their banking needs.
As a result, RazorpayX’s frontend developer count has more than doubled. Simultaneously, to serve a broader range of businesses, we introduced significant products such as the Chartered Accountant Portal and Vendor Portal. Our codebase had become a giant monolith and developer productivity as well as the speed of delivery of features was taking a hit.
To give you an idea of how challenging it had become to maintain the monolith, the upgrade from React 16 to React 17 took more than two weeks to complete. Due to the large codebase, we had to upgrade React all at once for the whole dashboard, which meant fixing all breaking changes between the two React versions in one go and risking regression when pushing such a large change.
Why did we reach such a stage?
- Lack of a reusable Auth
The authentication system was not designed to be reusable and support multiple apps.
- Inability to quickly spin up infra for projects
Creating new codebases required the creation of CI pipelines and other deployment infrastructure on AWS. This was not a priority during a time when getting the product to market quickly was critical.
- Lack of a reusable component library
We had UI components built in the same repo rather than a library. As a result, we were unable to easily share UI component code between products.
Issues due to the monolith
We began to encounter some issues as the codebase grew in size:
- Long build times
As the size of the codebase increased, it took longer to start the development server and for HMR to reflect developer changes.
- Long-running unit tests
Time to execute Unit tests grew exponentially with more and more apps being powered by RazorpayX’s repo.
- High cognitive load
Since code for multiple apps resides in the same repository, it increased the surface area of code a developer needs to understand thereby increasing the cognitive load on a developer.
- Lack of independent CI/CD
Because the same codebase that powers the RazorpayX dashboard for merchants, also powered the dashboard for vendors, chartered accountants and tax payments, any small change in one app required the deployment of all the apps.
So we began to consider what we could do about it: can we break our codebase into smaller independent modules, where each module could be developed, tested, and also deployed independently from the others?
Finding the right way to do it
The first project to be moved away from the X Dashboard codebase was Vendor Portal. We chose Vendor Portal because, if not for the above explained constraints, it would have been an independent project. It was less coupled with other code in the codebase and would serve well as an experimentation project.
Before adopting a potential solution, we devised a set of criteria that it must meet.
The criteria for the solution
In addition to solving the above discussed pain points, the ideal solution would check all of the following boxes:
- Should have good developer experience.
- Should not require drastic changes to the codebase.
- Should be relatively easy to configure.
- Should have a gradual learning curve for developers.
Potential solutions
After a lot of brainstorming around the problem statement, we came up with a few things that could be implemented:
- Publish each app as an NPM package
Each app could be published as a package to NPM. The package version in the host can then be updated, and the host must be redeployed
The limitation of this approach is that we would have to redeploy the host every time we published a new version of a micro app. This method works for libraries that require versioning, but it slows down deployments for projects that require continuous delivery
- Bundle each micro-app individually and load with a Script tag
Each micro-app is bundled and uploaded to S3 or a similar store. The host app loads the script via the URL
This method is very crude. Deduplication of dependencies, for example, is completely manual and must be configured for each project so that a dependency is loaded only once. In the case of some packages, such as React, the library should only be instantiated once as a singleton which adds more complexity
- Using micro frontend frameworks
There are numerous micro-frontend frameworks like Single SPA, Open Components, Mosaic, and so on
The configuration for these tools can get complex and difficult to manage as we add more apps since they require configuration at each layer — routing, dependency management, and development. This would complicate our tooling and hence, would not work for us.
- Using Module Federation with webpack 5
Module federation is a webpack5 feature that allows a JavaScript application to run code from another build. It allows us to reference code from a different project at runtime. You can read more about the concept here.
Module Federation is a run-time concept, and other tools, like Jest and Typescript, lack the same functionality. Making other tooling operate with Module Federation is complicated and requires a lot of configuration
Among the solutions we tried, Module Federation turned out to be the solution that ticked all the boxes.
What is Module Federation?
In simple words — it allows you to share code between different webpack builds at runtime. It enables you to expose files from one build and consume them in another. It is a method of achieving a micro frontend architecture.
Module Federation creates a very thin but well-defined contract layer between apps. It allows you to be flexible with what you want to slice out of the codebase and if the transition doesn’t work well, it also makes it easy to bring the application piece back in.
Our proposed architecture
The architecture we are aiming to build is a shell driven federation. The shell contains the logic for navigation, lazy loading of routes, instantiates a common Redux store to be shared between all apps, and exposes code shared between all micro apps. Each team can deploy their changes independently.
An important thing to note here is that we do not want to share any code between apps. This can create a mesh of dependencies which might result in future bugs.
After the migration of the Vendor Portal to a federated bundle, the system currently looks like this (simplified)
The monolith (shell) exposes UI components that are used in the Vendor Portal codebase. The shell and Vendor Portal have their own independent deployment pipelines and are stitched together at run-time by webpack in the browser.
Migration and Challenges
Moving to a federated architecture wasn’t easy. It took us a little less than 3 months to properly migrate Vendor Portal completely. We faced a few hurdles along the way and built tooling and processes that worked for our teams.
As a part of the migration process, we also migrated the Vendor Portal code to a monorepo because it makes solving the below challenges easier.
Now, let’s deep dive into some of the challenges we faced and how we solved them.
Sharing dependencies
A common problem with all micro frontends is how to de-duplicate dependencies. This is important because we don’t download the same dependency twice
Examples:
- MomentJS: Both, X Dashboard and Vendor Portal, use moment for date and time, and we want it to be loaded only once in the browser. At the same time, we want to ensure that the version loaded is compatible with the modules that rely on it.
- React: Dependencies that have an internal state, like React, should be instantiated only once and then reused across all federated modules as a singleton.
We wrote a utility function to solve this problem:
const dependencies = require('../package.json').dependencies
const getSharedDependencies = () => {
const singletonPackages = [
'react',
// other singleton dependencies here
]
const sharedDependencies = {}
Object.keys(dependencies).forEach((dependencyName) => {
sharedDependencies[dependencyName] = {
singleton: singletonPackages.includes(dependencyName),
requiredVersion: dependencies[dependencyName],
}
})
return deps
}
Then in the configuration for Module Federation plugin:
new ModuleFederationPlugin({
// ...
shared: getSharedDependencies(),
}),
Testing in federated modules
We run tests on the pre-commit hook using Husky. Since the code in Vendor Portal is dependent on UI code from X Dashboard, it must be present when tests are run (this is not an issue if the code is in a monorepo).
We leveraged jest’s moduleNameMapper to rewrite certain imports by Jest. This allows us to use actual code for running tests and not mocks as mock code in our tests would mean that tests will pass even after a breaking change if the dev forgot to update the mocks.
const xRepoBasePath = '<rootDir>/../../razorpayx'
module.exports = {
// ...
moduleNameMapper: {
// ...
// module federation related config
'^razorpayx/store': `${xRepoBasePath}/src/js/store`,
'^razorpayx/ui': `${xRepoBasePath}/src/js/ui/index`,
'^razorpayx/views': `${xRepoBasePath}/src/js/views/index`,
'^razorpayx/model': `${xRepoBasePath}/src/js/model/index`,
'^razorpayx/modules': `${xRepoBasePath}/src/js/modules/index`,
'^razorpayx/api': `${xRepoBasePath}/src/js/api/api`,
},
// ...
}
Type-checking
At RazorpayX, we have been migrating towards Typescript and want to be able to use the types defined for our UI code. We want type safety and auto-completion for developers writing their code locally. On CI, we want to run type checks to make sure that there are no TS errors.
For this, we use Typescript’s Project References feature. It enables a project to reference types from a different project:
{
"compilerOptions": {
// ...
"paths": {
"razorpayx/ui": ["<razorpayx>/src/js/ui/index"],
"razorpayx/views": ["<razorpayx>/src/js/views/index"],
"razorpayx/model": ["<razorpayx>/src/js/model/index"],
"razorpayx/modules": ["<razorpayx>/src/js/modules/index"],
"razorpayx/api": ["<razorpayx>/src/js/api/api"],
"razorpayx/store": ["<razorpayx>/src/js/store"]
}
},
"references": [{ "path": "<razorpayx>" }]
// ...
}
The above example is tsconfig.json file of Vendor Portal. In the references array, we add a path to the RazorpayX Dashboard codebase. It tells TypeScript about which other projects to reference for types. In the paths property, we assign types to the code exported from the RazorpayX dashboard using their path. This does require you to have both codebases on your local machine as well as CI, although this is not a challenge if your code resides in a monorepo.
Note: If your code is not inside a monorepo, you could take a look at the recently open-sourced Types Federation Plugin. One drawback of this plugin is having to manually add types to imports which the above approach does not suffer from.
CI actions with multiple repositories
We use GitHub actions for running tests and type-checking on PRs. Since not all of our code is in a single repository, for the time being, we clone both the repositories we need in the workflow. Checking out code is fast on GitHub actions so it has not affected the CI times in a significant manner.
We then run the tests command and type-checking on the Vendor Portal codebase on every push to the repo against the master branch of the RazorpayX Dashboard codebase.
What has this effort resulted in?
After the migration, vendor portal has its own codebase, independent CI/CD pipelines, independent dependency management. All this has led to the following improvements:
| Before (entire dashboard) | After (independent module) | |
|---|---|---|
| Build time | 180+ seconds | 14 seconds |
| Starting development server | 90+ seconds | 12 seconds |
| Running tests | 8-10 minutes | ~2 minutes |
| Deployments | The entire codebase has to be built and all apps will be deployed. | Only Vendor Portal code is built and a single app is deployed |
| Cognitive load on developers | High cognitive load owing to the large codebase size | Limited codebase size ensures faster onboarding and ease of development |
| Dependency management | Change in dependency version affects all apps powered by the codebase | Dependency versions can be scoped to apps which ensures that each app loads its supported versions |
Where do we go from here…?
Overall, Module Federation has solved a lot of problems for us and we are migrating towards a federated architecture to enable developers to ship faster.
We are working on solving the developer experience hurdles that a distributed micro frontend can create and at the same time, ensuring that the code we ship is resilient.
Having infrastructure pieces that support this architecture has been of tremendous help. Check out Devstack on GitHub which allows us to use different commits when we develop Vendor Portal.
If you want to understand our decision-making and reasoning about Module Federation in more detail, you can watch this talk we gave about the same on YouTube
This article was originally published at Razorpay's Engineering Blog
Get updated when I next post, subscribe at burhanuday.com
Top comments (0)