DEV Community: Vincenzo Marragony

Saving 62% on EC2 costs ($41k to $15k)

Vincenzo Marragony — Fri, 28 Apr 2023 14:11:38 +0000

Scenario

During the first few months of my new job, I created a cloud infrastructure to host a large eCommerce site running on Magento 2. I containerized the app and wrote a Terraform configuration to launch and configure everything needed for the service to work, including VPC, EC2 instances, ECS, S3, CloudFront, EFS, and more.

The issue with a Magento eCommerce site receiving an average of 10 million requests each month is that the load tends to slow down the infrastructure, requiring a powerful server as a base starting point. After load testing, I found that I need a c5ad.2xlarge compute-optimized instance with 8CPUs and 16GB of dedicated RAM to run Magento smoothly.

In addition, Magento requires additional services such as MariaDB, Redis, and Elasticsearch to function. To address this, I’ve chosen to use a dedicated machine for each service. Currently, I am not using RDS due to its pricing, but I’m considering whether the risk of sudden downtime from one of these instances is worth it.

I’m referring to this risk as we experienced a service disruption in a previous project due to AWS maintenance on one of our running machines. We received an email notifying us of a hardware fault and the need to replace the current instance with another one within two weeks. Unfortunately, AWS shut down the machine immediately after the notice.

With RDS we could start a database with two running instances, decreasing the risk of downtime in case of hardware failure.

While this might be worth writing about, let's focus on our cost-saving strategy for now. As of today, it has been two years and we have not experienced any downtime related to these machines in this specific project.

Problem

In total, we have a minimum of six instances running in production, including two base instances for ECS, service instances, and a bastion host. The instances are configured by the following sizes: 2x c5ad.2xlarge for ECS, 1x m5ad.xlarge for Redis, 1x c5ad.xlarge for MariaDB, 1x c5ad.large for ElasticSearch, and 1x m5a.xlarge for the bastion host.

The monthly on-demand costs for each instance are 2x $288 for ECS, 1x $180 for Redis, 1x $144 for MariaDB, 1x $72 for ElasticSearch, and 1x $180 for the bastion host, totaling $1,152/month.

This amounts to $13,824/year and $41,472/three years.

How can we save money from this?

Solution

It actually depends on your situation, and in my scenario, I have an eCommerce that is running for more than five years, so it makes sense to get reserved instances for three years paying all upfront.

Here there’s a Google Sheet with aws-pricing-sheet-addon to get a quote with the latest pricing, comparing reserved instances with different payment methods.

Sample picture

Read-only URL

Costs Evaluation spreadsheet

Results

With this cost evaluation, I managed to save up to 62.40% in EC2 costs, going from $41,472.00 to $15,593.42.

It’s clear that you will need to pay $15,593.42 all upfront, but depending on your situation, this can be a proficient way to save on cloud costs.

On a personal level, it was the first time that I’ve spent more than $15.000 in one shot for someone else. It’s two years now that we’re saving and managers are happy about it 😃

Conclusion

This article is part of a series where I write about production use cases as a cloud engineer. The focus is on meeting business needs, working as a team, and personal growth.

If you find this article useful or want to share your thoughts about it, feel free to write a comment. I would be happy to read it.

As a side note, I'd like to add that this article was written with the help of AI.

See you in the next chapter!

Saving 30% on costs and improve infrastructure reliability with profiling

Vincenzo Marragony — Mon, 24 Apr 2023 13:13:47 +0000

Scenario

As the technical co-founder of MyStoryViewer, a SaaS product that automated Instagram tasks for customers, I was responsible for full-stack web development and cloud engineering. However, as we began to scale and reached 600 concurrent workers, the product's performance suffered, and our customers began to complain.

If you want to read more about the underlying infrastructure, I've written this post about how I've managed to set-up workers.

Problem

The software responsible for processing customer data was a NodeJS (Typescript) application running concurrently on large servers.

To troubleshoot the problem of degraded performance, I profiled the application at runtime to determine whether there were any issues causing the degradation.

Solution

I started using ClinicJS Doctor to profile my software, and as shown in the results, there were issues with memory usage and event loop delays.

The problem was clear: something was using too much memory.

Since I was also the developer, I knew that a function I had written was retrieving data from the Instagram API. The API provided information about a profile, and I was storing all of the retrieved data in an array. Since I was processing 500 profiles for each iteration, I had megabytes of unnecessary data stored in memory. The solution was to only save relevant data from the API in memory.

Results

With a single fix, I managed to cut costs and stop receiving complaints from customers.

The workers were running in parallel on big servers, and with this improvement in memory usage, I could run more instances per server, reducing costs by around 30%. Additionally, since the workers stopped crashing, I was able to improve infrastructure reliability and customer satisfaction.

On a personal level, I discovered a new way to troubleshoot and improve performance issues.

Conclusion

This article is part of a series where I write about production use cases as a cloud engineer. The focus is on meeting business needs, working as a team, and personal growth.

If you find this article useful or want to share your thoughts about it, feel free to write a comment. I would be happy to read it.

As a side note, I'd like to add that this article was written with the help of AI.

See you in the next chapter!

Hundreds of bot instances to the back-end using Redis

Vincenzo Marragony — Fri, 08 Nov 2019 18:06:23 +0000

Summary

In this story I'll write about the solution that I've implemented to authenticate and manage more than six hundred instances of a distributed Node application (the bot) to a back-end. At that time I was working for my start-up that was serving an automation tool for Instagram. I needed to constantly exchange data between the bot instances and the associated user profile.

The Case Study

The Software as a Service allowed clients to sign-up, configure their working profile and control when to start or stop it. When started, a bot would have taken their configuration parameters and would have started to work on it. For each one of my clients I needed one bot that was processing their data.

The main challenge that I've faced for such system is to connect a bot to a profile. Who's working to what? I also wanted to understand how many bots were available at any given time, how many of them were currently working and if all the servers had the correct amount of bots running. This way I was able to understand if there was some kind of failure somewhere in the system.

It was clear that I needed a way to authenticate those bots, a method to update their state and check if they're available to work, a two-way binding to associate a bot to a user profile and a cronjob to check if everything was okay.

The idea about writing an additional software implementing a client-server model solution seemed to be the best thing to do. A single server instance would have taken the responsibility to manage the bots and the data received by them and the connection to the master database would have done the job in terms of data processing. As you might understand, after managing a front-end written in React, a back-end written in Laravel, all the DevOps on AWS and a bot written in NodeJS, I didn't want to add an extra piece to this stack, mostly because I hadn't enough money to hire an additional developer, I had to do it by myself.

This is why I've still implemented something very similar to a client-server model... using REST APIs!

Endpoints

`/bot/auth`

This endpoint had the responsibility to authenticate the bots. After a successful authentication a token would have been generated and returned by the request. This way the system had something to remember the new bot instance. A single string parameter was required, the hostname of the server hosting it.

`/bot/hertbeat`

This one had the responsibility to give the bot instructions on what to do next. By checking both the bot's own state and the system's state, the back-end was able to understand if there were waiting profiles ready to be assigned, if the current profile has been stopped for whatever reason or if the bot could have kept running. This is why this one was the most critical endpoint: a bot would have called it every thirty seconds.

Another reason for this endpoint to exists is that after each request, I was saving an extra field to acknowledge that the current bot has done a request here. This way I could check if a bot was still alive or not.

`/bot/push_data`

Finally, this one was used to get data from the bot. There's nothing special to say about it, the required token parameter was what the back-end needed to understand who was the bot sending the data and to which profile was it bound.

System health checks

What if a bot crashed for whatever reason? Every instance was built inside a Docker image with supervisor installed and configured to restart the instance in case of unexpected exits. But, you know, when you've got paying clients it's always better to double (or triple) check your job.

This is why I've thought about a back-end health check. The responsibility for it was delegated to a cron-job that had the task to check the existence for the heartbeat. If it was expired, the bot was considered as dead and deleted from the authenticated bot list.

Storing bot data using Redis

Bots are volatile and the system must be scalable to the infinity. I don't need any permanent storage solution, the RAM is all that I need: high speed and volatility.

The choice to use Redis has been made because its hashes data sets and the key-value store. The first solution came in handy because given a single hash you can get all the values present inside, very useful to get the state for a single server. On the other hand, the second one has been used to store the last heartbeat by expiring keys.

Speaking about the bot's data, it has been stored as a JSON string. From its class implementation to the plain object and converted back as needed.

Commands that have been used are the following:

HSET to save a bot instance, used after successful authentication in /bot/auth
HGET to get a bot instance, mostly used by /bot/heartbeat and /bot/push_data endpoints
HVALS to get all the bots in a server, used by cronjob and status page
HDEL used by the cronjob to delete a bot when dead
SETEX to reset the heartbeat
EXISTS to check if the heartbeat still exists

Drawing the system

Here's a simple drawing about the structure. Each request interacts with some parts of it and it has been uniquely coloured.

Actually coding it

If you wish to see the actual implementation... you're a lucky guy! I've taken part of my PHP application and I've re-written it in NodeJS. Everything is hosted in a GitHub repository. Feel free to have a look at it and play with it. Suggestions are welcome if you see something wrong!

Repository url: MrMavin/article-bots-to-backend

What are you going to get when playing with it?

___

Thank you

If you've got there, thank you! Please take a moment to give me a feedback. I would like to know what do you think about this solution and if you've enjoyed reading this article :)