Most of the organizations include the one for whom I had work has some successful products (applications) that are more than 5 to 6 years old now. Although applications are running considerably ok, all either become slow or facing a hard time extending features. That is the time when product stakeholders need a new version of the same application created from scratch. Sometimes its also becomes Compulsion. That will give a more satisfying experience to the end-users.
A similar situation occurred in one of a business for which I worked in recent past.
A product is yet to be refactored and going to change some business logic and UI. I was one of the developers assing to be part of that team.
After spending some time understanding the business logic, I feel that the application needs a critical refactoring of the heart of the application. But stakeholders are not considering that as part of reengineering.
The application had some critical problems that has to take care of. A feature of the app creates a zip file every time file modifications are made in sub-modules. The sub-modules are deployable to the customer hosting provider.
Whenever changes are made in any file, a zip archive of complete the submodule is created. All those zip files are stored in multiple cloud service providers like Google Cloud Platform, AWS, and Linode. The application has generated almost 30TB so far in the last 5 years. Now managing that 30 TB is not only pain also very costly. Company spending around 500USD/month to manage those files.
Submodule deployment uses FTP/SFTP and SSH methods to ship the code to the hosting service provided by the customer. The size of the submodule on average is 20MB, but it can be up to 200MB. Deploying 20MB code has taken almost 20 minutes every time. Deployments are not real-time. Whenever a request comes for the deployment of the module, that save in the database and executed by cron job. Those are not concurrent processes. Requests are executed one at a time. So throughput is very less in pick time.
There is no status of progress for deploying the module. There is no real-time notification for the same.
Here I came with a solution that promises 90% faster deployment and a 30% cost-efficient. But the challenge is to convince stakeholders. I know they are reluctant to take the risk as always.
Now I thought to create a prototype and make a demo for them. It takes almost 7 to 8 days to make ready the basic prototype. My team lead encourages me and gives me a chance to present that to the higher management. At the very first presentation, they have not shown any interest in the solution as I expected. I again try after few days, these time they show interest and convinced. I took the responsibility to transform the idea into implementation.
I decided to transform the monolithic application into a microservice architecture. And separate the sub-module management from the actual application. The proof of concept was developed with NodeJs, ReactJs, RabbitMq, and MongoDB. But the main application is about to be developed with NodeJs, Angular, MySql, and RabbitMsq tech stack.
2 services have been developed for the main two tasks.
API Service: to interact with UI and get the instruction of file save, delete, update. Request for the submodule version and deployment of the sub-module, real-time notification, and many others.
Batch Process Service: Interact with API service using message queue (RabbitMq) to get deployment instruction, execute the deployment process, and provide real-time progress notification. It is a standalone system not exposed in any port.
I have decided to change the file storage system from file-based to git base. Choose GitHub as a provider, which charges 29USD / month as an enterprise edition.
As of now, we moved from a file-based storage system to a git-based. Managing file versions and change records are become easy. Few API calls will give all the details. Earlier those are saved in the database.
Git-based architecture solves the problem of saving files in a cloud-based file-hosting management solution and manage headaches.
As a batch process service has been introduced the deployment has been done in real-time. No need to wait for a corn job to take a request and execute it.
The batch process can handle concurrent deployment up to a limit that the server configuration permits. Even does not limit there, the batch process is a cloud-ready scalable and standalone application. Which can be run in any environment. It takes 1 to seconds to get ready to execute the instructions. Which allows creating multiple instances of the batch processor in pick time to get things done quickly. That brings down the throughput time of deployment from an average of 20 minutes to between 40 sec to 1 minute. Insane !.
The batch process feeds a real-time update of the status of the ongoing task. It helps the user to understand how much time is left to finish off.
So as Promised, the throughput brings down almost 90% better than the legacy application. The cost is not going to be that much as that of the legacy application. And One Most Important managing File and Application is now far better than legacy application. The new architecture will also provide features like real-time notification. Less overhead deployment. Better code editor than legacy one and many more scope for future development.
Top comments (0)