DEV Community: Rahul

How to perform realtime transformations on S3

Rahul — Tue, 25 Oct 2022 04:55:57 +0000

Learn a new feature of AWS S3 which can be used to do realtime transformations like compression, resizing images etc.

Let’s discuss a new feature of S3, S3 Object Lambda, which can do realtime transformations of files, as you retrieve them.

Recently came across an S3 feature, where you can attach lambda to any AWS S3 bucket and do your conversions, transformations, altering on the fly and also use all the features that S3+CloudFront provides.

Use cases

Compressing or decompressing files as they are being downloaded.
Resizing and watermarking images on the fly.
Converting across data formats, such as converting XML to JSON.
Implementing custom authorization rules to access data. Many more (left to the devs’ creativity).

Steps

Create an Access Point for S3 bucket.
Create an Object Lambda Access Point, and attach above created Access Point and a Lambda, with required permissions like GetObject.
Write a code in above created Lambda, to retrive and modify the files according to your needs. Example Code

Example

Let’s use the example of image transformation on the fly.

User asks for sunset_600x400.jpg
Lambda function will get invoked.
If sunset_600x400.jpg is available, return 200 success to continue downloading the file.
If sunset_600x400.jpg is not available, will remove suffix _600x400 and look for an image named sunset.jpg
Resize it to fit the maximum width and height as described in the file name and save it in S3 as sunset_600x400.jpg
Return 200 success to continue downloading the file.

Caveats

For every S3 file access, lambda will get invokes and thus need to pay extra for Lambda.

In Conclusion…

With this new feature from AWS, you can do file transformations, image resize/compressions, limit access etc on the fly.

Let me know if you find any other use cases.

That’s it! Please let me know about your views and comment below for any clarifications.

If you found value in reading this, please consider sharing it with your friends and also on social media 🙏

Also, to be notified about my upcoming articles, subscribe to my newsletter below (I’ll not spam you 😂)

Blog of Codes | Rahul | Substack

Articles about cloud architecture and programming. Click to read Blog of Codes, by Rahul, a Substack publication. Launched 2 months ago.

blogofcodes.substack.com

You can find me on Twitter and LinkedIn ✌️

How to build caching service with 10% of your Redis, DynamoDB costs

Rahul — Mon, 17 Oct 2022 08:11:17 +0000

Learn how to utilise FREE cloud services and cache like Redis, DynamoDB with 10x reduction in costs

Before we dive in, this is just a theory. No one has implemented it till now. So, use it only if you understand the numbers.

So recently, I was reading a blog saying that S3 isn’t more expensive than DynamoDB, if you afford the extra latency.

Then I got an idea of why not use some cloud providers’ FREE services and build a caching service.

Limitations

Latency of 100ms.
No custom invalidation.

Services

Backblaze
- Similar to S3
- 10 GB of FREE storage
Cloudflare
- Service is completely FREE
- Provide edge caching.

Here’s are the steps

Create a storage bucket in Backblaze.
Connect your Cloudflare DNS with above bucket via some route.
Enable Cloudflare edge caching to this route.
Now you can upload to bucket with filename as key and file data as value.
Get/serve your value/file from Cloudflare route, to act as cache.

Cost Estimation

Let’s take an example of 10 billion entries of 1KB each with 100 billion reads (I know it’s massive)

DynamoDB

On-demand pricing of DynamoDB
- $0.25 per GB of storage
- $1.25 per million write requests
- $0.25 per million read requests
Calculations
- 10 billion * 1 KB = 10 TB of storage * $0.25 per GB => $2,500
- 10 billion inserts * $1.25 per million inserts => $12,500
- 100 billions reads * $0.25 per million reads => $25,000

Total = $2,500 + $12,500 + $25,000 = $40,000

Backblaze + Cloudflare

Backblaze cloud storage
- $5 per TB of storage
- Unlimited FREE inserts/uploads
- $0.004 for 10k reads
Considerations
- Cloudflare DNS and edge caching is completely FREE
- 90% of reads will be from Cloudflare cache, which is FREE. So only 10% will hit Backblaze bucket and get charged
- Cloudflare, Backblaze are in bandwidth alliance, which gives us zero egrees fees for these files.
Calculations
- 10 billion * 1 KB = 10 TB of storage * $5 per TB => $50
- 10 billion inserts * Unlimited FREE inserts => $0
- 100 billions reads * $0.004 per 10k reads * 0.1 (10% of reads) => $4,000

Total = $50 + $0 + $4,000 = $4,050

$40,000 => $4,000

10x decrease 🤯

In Conclusion…

I know this is completely theoretical. And also I didn’t take into account many things while calculating numbers, provisioned capacity, FREE tiers and etc.

But I would say, if you would leave it aside, and if you can afford little higher latency, and some caveats, then I think this can work as cache, alternative to DynamoDB, Redis.

That’s it! Please let me know about your views and comment below for any clarifications.

If you found value in reading this, please consider sharing it with your friends and also on social media 🙏

Also, to be notified about my upcoming articles, subscribe to my newsletter below (I’ll not spam you 😂)

Blog of Codes | Rahul | Substack

Articles about cloud architecture and programming. Click to read Blog of Codes, by Rahul, a Substack publication. Launched a month ago.

blogofcodes.substack.com

You can find me on Twitter and LinkedIn ✌️

How to schedule jobs at scale

Rahul — Mon, 10 Oct 2022 12:59:08 +0000

Let's discuss how to schedule and process millions of jobs.

Use-case:

Build a social media scheduler, where users can schedule their posts, like everyday at 7:30 PM and post them to their social media accounts automatically.
Build a product where users can trigger their ETL jobs or some code every 10 min.
Build a status page product to check your product health, speed etc for every 1 min.

Now let's dive into some solutions and their cons.

Cloudwatch/Eventbridge rules

Add a rule in eventbridge for every customer need and this rule takes care of hitting that particular job.
Easy to setup but has a hard limit of 300 rules per account, so not scalable.
Although this solution doesn't scale for your customers, but can be used for your own product jobs.

Code it

Take all the jobs which run in the next 10 min
Add it to "to-be-processed" table with exact time.
Continuously run a code/executor, to get all the jobs which have execution time <= current time and not processed, and execute them.

Example:

Let's say you are running this scheduler cron every 10 min and current time is 7:28 PM. And you have job A scheduled at 7:32 PM.
Now get all the jobs which needs to be run in the next 10 min that is 7:28 PM to 7:37 PM, which includes job A.
And then push to "to-be-processed" table with exact time.
Then your executor code will be getting all the jobs which have execution time <= current time and not processed, which also includes job A, and then executes them.

Issues:

Not scalable with 1 executor.
If working with multiple executor threads, need to handle concurrency and not execute the same jobs multiple times.
This can re-run failed jobs. But should write a logic to stop after certain number of failures.

SQS

Using delay feature in SQS. This enables us to send messages now and consume them after some time.
Now take all the jobs which run in the next 10 min.
Add it to SQS queue with it’s specific delay.
Attach a lambda consumer/executor to the SQS, which will consume messages, when available and execute the jobs.

Example:

Same example. Let’s say you are running this scheduler cron every 10 min and current time is 7:28 PM. And you have job A scheduled at 7:32 PM.
Now get all the jobs which needs to be run in the next 10 min that is 7:28 PM to 7:37 PM, which includes job A.
And push to SQS with it’s specific delay. So for job A, which needs to be run at 7:32 PM, which is 5 min after 7:28 PM/current time. Push job A with 5 min delay.
Lambda consumer will get all the messages, when available. So job A will be available to consumer after 5 min, that is 7:32 PM and executes it.

Does it solve previous solutions issues?

Since SQS, lambda is server less and highly scalable, this solution is scalable too.
Since we can tell SQS to send message to consumers only once, this doesn’t execute the same job multiple times.
If failed and we don’t delete message from SQS, this will re-run after certain time automatically (specified by us).
Multiple time failed jobs can also be removed from further processing by using/pushing to dead letter queue automatically.

In Conclusion…

With this architecture, you can schedule a very large number of jobs. Let me know if you have a better solution in mind.

That’s it! Please let me know about your views and comment below for any clarifications.

If you found value in reading this, please consider sharing it with your friends and also on social media 🙏

Also, to be notified about my upcoming articles, subscribe to my newsletter below (I’ll not spam you 😂)

Blog of Codes | Rahul | Substack

Articles about cloud architecture and programming. Click to read Blog of Codes, by Rahul, a Substack publication. Launched a month ago.

blogofcodes.substack.com

You can find me on Twitter and LinkedIn ✌️

How to avoid top 5 mistakes in SQL

Rahul — Mon, 03 Oct 2022 15:04:32 +0000

Let’s discuss some tips and tricks to avoid major mistakes in SQL

Never use Select *

select *, gets the data from all the columns, which increases the latency and is expensive for huge data.
Instead, get only the required fields, which limits the size of each record

Use EXISTS() Instead of COUNT()

If you want to know whether a record/result exists or not, it is better to use exists(), rather than count()
count(), will browse the entire table to get you the number of records.
Whereas, exists(), will get back to you, when it finds the first record for the query, saving you time and computing.

Functions on indexed columns is useless

Using functions on indexed columns, while querying, will make the indexes remain useless.
In order to use indexes, we need to avoid adding functions on the indexed columns.

LIMIT Statements

Most common used pagination is LIMIT, OFFSET, which is not optimal.
It is fast for smaller and immediate sets like “LIMIT 0, 10”
But when OFFSET is changed to 1000000, it takes too long, since database doesn’t know where 1000000th record exists, it starts from scratch till it finds 1000000th row.
Better approach would be to add another filter, mostly indexed column, like below

Use GROUP BY Instead of DISTINCT

The distinct is an expensive operation, and doesn’t use indexes if available.
Faster and easier way to do the same is to use group by.

In Conclusion…

I haven’t covered many major SQL pitfalls. But these are some of the top mistakes, even made by experienced developers. Now you get to avoid them 😁

That’s it! Please let me know about your views and comment below for any clarifications.

If you found value in reading this, please consider sharing it with your friends and also on social media 🙏

Also, to be notified about my upcoming articles, subscribe to my newsletter below (I’ll not spam you 😂)

Blog of Codes | Rahul | Substack

Articles about cloud architecture and programming. Click to read Blog of Codes, by Rahul, a Substack publication. Launched a month ago.

blogofcodes.substack.com

You can find me on Twitter and LinkedIn ✌️

How to save upto 50% of your AWS costs

Rahul — Mon, 26 Sep 2022 08:32:24 +0000

Let’s discuss some tips and tricks to reduce the cost of the most used AWS services - RDS, EC2, S3, DynamoDB, CloudFront.

RDS

Migrate to latest graviton based instances. According to AWS, Graviton2 instances provide up to 35% performance improvement and up to 52% price/performance improvement.
Opt for reserved instance, if you know your load. Can save upto 60% when compared to normal On-Demand database.
Upgrade to latest SQL version, for higher performance, speed and also security.
Don’t use multi A-Z, if you don’t need it.
Downgrade or auto shutdown staging, testing instances, when not in-use.
Disable performance insights for non-prod database.
Auto delete snapshots that are too old.

EC2

Migrate to latest graviton based instances.
Opt for reserved instance, if you know your load. Upto 72% savings compared to normal On-Demand EC2 instances.
Move to lambda for infrequent batch jobs, instead of running EC2 all day long. 1 million executions FREE per month.
You can also pick spot instances for testing, staging environments. Upto 90% discount when compared to normal On-Demand EC2 instances.
Upgrade to latest EC2 versions.
Downgrade or auto shutdown staging, testing instances, when not in-use.
Use smaller instance, instead of 1 large instance, so that you can scale up and down when necessary.
Choose EC2 in same region as other service like S3, DynamoDB to avoid unnecessary data transfers.

S3

Enable S3 Intelligent-Tiering to automatically move infrequently accessed files to lower priced S3. Can save up to 95% on storage costs for data that is not accessed for months, or even years, at a time.
Store compressed formats, if possible.
Enable CloudFront and serve through it, to reduce requests and data transfer cost.

DynamoDB

Check your minimum load and use provisioned capacity. 25 Write Capacity Units and 25 Write Capacity Units are FREE every month.
Use on demand for lesser used tables.
Prefer queries over scans.
Use shorter attribute names.
Avoid strongly consistent reads and transactions, wherever possible.

CloudFront

Auto Compress via CloudFront to reduce data transfer.
Go for savings plan, if you know your load. Can save upto 30% when compared to on-demand.

In Conclusion…

I haven’t covered many AWS services and tips to reduce cloud costs. But these are some ways by which you can reduce your AWS cloud bill upto 50%.

That’s it! Please let me know about your views and comment below for any clarifications.

If you found value in reading this, please consider sharing it with your friends and also on social media 🙏

Also, to be notified about my upcoming articles, subscribe to my newsletter below (I’ll not spam you 😂)

Blog of Codes | Rahul | Substack

Articles about cloud architecture and programming. Click to read Blog of Codes, by Rahul, a Substack publication. Launched a month ago.

blogofcodes.substack.com

You can find me on Twitter and LinkedIn ✌️

How to host your product for FREE

Rahul — Mon, 19 Sep 2022 05:00:04 +0000

Let’s discuss various cloud providers and their FREE services to host your product for FREE 😁, till 50k monthly active users 🤯. Be it frontend, backend, database, authentication, emails etc.

Authentication

Firebase Authentication

Can be used by up to 50k monthly active users for FREE.
Supports Google, FB, Phone, and Email.

AWS Cognito

FREE till 50k monthly active users.
Supports Google, FB, Phone, and Email.

Although you could use Google, and FB SDKs to implement their logins. It would be much easier to implement and maintain users using Firebase Auth or AWS Cognito.

Frontend

Cloudflare Pages

With Git integration, you can directly deploy your website with every commit.
Can preview every commit in every branch before going live.
No CI/CD setup is required.
FREE plan supports up to 500 builds per month. That means you can push 16 times a day, preview and deploy to prod for FREE.
Unlimited bandwidth. So you can have unlimited visitors to your website for FREE.

Other notable services are AWS Amplify Hosting and AWS S3+Cloudfront.

Media

Backblaze + Cloudflare

Store assets on Backblaze and server them through Cloudflare CDN.
Backblaze provides 10 GB of storage for FREE and Cloudflare CDN is completely FREE to use.
Also, the bandwidth alliance between Backblaze and Cloudflare gives us zero egrees fees for these media.

Other notable services - AWS S3+Cloudfront/Cloudflare.

Backend

AWS Lambda

Serverless - Can deploy your backend code on Lambda and server your API without worrying about the scale.
Lambda even has a function URL, thus you can directly use it like an API, without any API Gateway
AWS Lambda has a very generous FREE tier of 1 million invocations (with 128 MB RAM - 3 sec time) per month, which translates to 33.3k API hits per day

AWS Beanstalk

AWS gives 1 EC2 server with 1 GB RAM, and 2 vCPUs with elastic load balancing for 750 hours FREE, which means FREE for the entire month.
With this config, I was able to serve 50 req/sec without any issues (using Golang 😁)

Database

AWS RDS

Postgres or MySQL with 1 GB RAM, 20 GB SSD, and 2 vCPU is FREE for an entire month.
Can easily handle 50 req/sec (if query latency is less than a second 😄)

AWS DynamoDB

DynamoDB is NoSQL key-value database.
25 GB of storage, 25 provisioned Write and Read Capacity FREE each month.
This is enough to handle up to 200 million requests per month, with 25 write/sec up to 1KB and 50 reads/sec up to 4KB.

AWS Elasticache

Redis or Memcached with 0.5 GB, 2 vCPU can be used for an entire month FREE
For 500 B for each key, this can handle 1 million keys in RAM.
Can use AWS MemoryDB for Redis too.

AWS DocumentDB

AWS version of MongoDB.
With 4GB RAM, 2 vCPUs and 5 GB storage, you can host MongoDB for FREE.

Monitoring

AWS Cloudwatch

Monitor your server, log your requests etc.
FREE till 10 custom metrics, 10 alarms, 5GB of log ingestion
If you set the logs to expire for 7 days, mostly this limit would never be touched 😊

Email

AWS SES

Send 62k emails per month for FREE. That means 2k emails per day 🤯

Notifications

Can use both Firebase messages and Onesignal notifications.
Send unlimited notifications every day, completely FREE.

In Conclusion…

There are many cloud providers and their FREE services, which I didn’t cover in this article.

But you get the point, that you can host your entire product tech stack, to 50k monthly users, for FREE without opening your wallet.

That’s it! Please let me know about your views and comment below for any clarifications.

If you found value in reading this, please consider sharing it with your friends and also on social media 🙏

Also, to be notified about my upcoming articles, subscribe to my newsletter below (I’ll not spam you 😂)

Blog of Codes | Rahul | Substack

Articles about cloud architecture and programming. Click to read Blog of Codes, by Rahul, a Substack publication. Launched a month ago.

blogofcodes.substack.com

Django (python) or Golang?

Rahul — Fri, 18 Feb 2022 05:16:06 +0000

I'm starting a side project. I’m confused what to use for backend.
I’m an expert in Golang, but also love Django’s ease of development

Any suggestions?