DEV Community: Prithvi Jethwa

CloudFormation Template does count whitespaces in Template Size...

Prithvi Jethwa — Sat, 21 Feb 2026 14:05:08 +0000

Context

We hit the limit for AWS CloudFormation Template Size of 1 MB which caused our deployments into alpha, staging, and production to fail. Ideally the suggestion is:

Use Nested Stacks
Create more CloudFormation Templates
Remove unnecessary resources, or consider consolidating IAM Roles and Policies

We were going to follow our next steps to move the monitoring resources (example: AWS CloudWatch Alarms) into the Nested Stack, but doing a quick analysis using Claude led me to an interesting yet questionable find.

Analysis

I found that 50% of the CloudFormation Template Size was just whitespaces, which seems innocent but considering the 1 MB limit, that's a lot.

To experiment, I ended up running jq to remove all the whitespaces using

jq -c '.' cloudformation-stack-update.json > minified.json

Within the Infrastructure Composer, I copied both the CloudFormation Templates, and clicked on the Validate Template button. The Regular Template showed that the file has exceeded the 1 MB limit, meanwhile the minified CloudFormation Template successfully validated.

Next Steps

The Platform Team has a Deployment Pipeline that handles all the packaging of CloudFormation Templates using the Serverless Framework, so I ended up making a Feature Request about the very issue about minifying things.

While they would get to it eventually based on priorities, I had an inner rush to resolve it. I used to be a part of that Platform Team, so it was conveniently easy to get back and contribute to those Deployment Pipelines.

Opened a PR into the Repository, added in my testing notes and got it reviewed and merged. After retrying our service related builds, we saw a 50% cut on the CloudFormation Template Size, helping us unblock our deployments! 🎉

Why is this important

It is an interesting gotcha, cause you would not expect whitespaces to eat into the CloudFormation Template Size, but I ran into this issue, and I am hoping others do not run into it. So consider minifying those templates.

While I haven't seen an option in the Serverless Framework, adding a provider.minify would be cool to have.

Reference

https://docs.aws.amazon.com/AWSCloudFormation/latest/UserGuide/cloudformation-limits.html

First contribution to Open Source - charmbracelet/huh

Prithvi Jethwa — Fri, 18 Oct 2024 01:14:35 +0000

Context

I always wanted to contribute to an Open Source Project, but I could never find out a project where I could drop in and understand the issues posted.
I did have a hard requirement for the project to be in Go, and then came along this very cool project called Charm.

The project provides tools and libraries to build out Terminal User Interfaces (TUIs) using Go. It helps to create terminal applications that improve Developer productivity which I highly appreciate.

How did I start

I joined their Discord channel a while back, and mostly lurked around seeing all the TUIs built by other users, and how contributions were being discussed. It led me to try out the examples provided within the repositories and reading the source code for it. After feeling slightly comfortable for it, I took a look into the GitHub Issues created and found this particular one https://github.com/charmbracelet/huh/issues/367 .

First steps involved reading through the issue, and attempting to write minimal code to reproduce the issue. I started looking at the related functions to figure out how and why the output is being rendered.
After trial and error, created the necessary changes to help fix the issue, and requested a PR Review!

It got merged in PR-427 (technically PR-425), and I'm very glad that it did! 🎉

Why it's important to me

I'm finally in a position that I'm able to contribute back to Open Source, that hopefully benefits 1 developer atleast. Having the Source Code available helps me learn how and why the changes were made. It provides an opportunity for making a mental model of the given system, which is a good challenge. Finally it's in Go and I enjoy writing in Go.

I'm grateful that the my first Open Source Contribution is towards Charm and looking forward to helping out more!

Next Steps

I'll continue watching for GitHub Issues across the multiple tools that they provide, and consider trying to solve those challenges mentioned.

References

Parse UserParameters sent from AWS CodePipeline to AWS Lambda in Go

Prithvi Jethwa — Wed, 02 Oct 2024 17:59:54 +0000

Context

I was trying to setup the UserParameters configuration within the AWS CodePipeline Template being generated,

Name: ...
Actions:
  - Name: Invoke-Lambda
    ActionTypeId:
      Category: Invoke
      Owner: AWS
      Provider: Lambda
      Version: '1'
    Configuration:
      FunctionName: exampleLambdaFunction
      UserParameters: '{"example":"user-parameters"}'

While testing it out on an AWS Lambda, written in Go, it took a bit longer than usual to find out the Function Definition for the Handler, to parse the AWS CodePipeline JSON Event that would be sent, For Example:

{
    "CodePipeline.job": {
        "id": "11111111-abcd-1111-abcd-111111abcdef",
        "accountId": "111111111111",
        "data": {
            "actionConfiguration": {
                "configuration": {
                    "FunctionName": "exampleLambdaFunction",
                    "UserParameters": "{\"example\":\"user-parameters\"}"
                }
            },
            "inputArtifacts": [
               ...
            ],
            ...
        }
    }
}

Solution

Use the github.com/aws/aws-lambda-go/events package link which contains the events.CodePipelineJobEvent that helps unmarshal the AWS CodePipeline JSON event being sent

package main

import (
    "context"
    "fmt"
    "github.com/aws/aws-lambda-go/events"
    "github.com/aws/aws-lambda-go/lambda"
)

func Handler(ctx context.Context, event events.CodePipelineJobEvent) (string, error) {
    fmt.Printf("received codepipeline event function name: %+v\n", event.CodePipelineJob.Data.ActionConfiguration.Configuration.FunctionName)
    fmt.Printf("received codepipeline event user parameters: %+v\n", event.CodePipelineJob.Data.ActionConfiguration.Configuration.UserParameters)
    return "cool", nil
}

func main() {
    lambda.Start(Handler)
}

References

Importing CloudFormation Resources to help fix deployments to Production

Prithvi Jethwa — Sun, 31 Mar 2024 16:33:15 +0000

Context

A Frontend Developer in another Team, had merged their PR with the following changes:

# serverless.yml

+ tags:
+    coolTag: coolValue
+    coolTag2: coolValue2
- resources:
   Description: Cool Description
   Resources:
     CoolS3Bucket:
       Type: AWS::S3::Bucket
       ...

     CoolS3BucketPolicy:
       Type: AWS::S3::BucketPolicy
       ...
...

It was a simple PR to add in AWS Tags for helping out with visibility of the resources created through AWS CloudFormation and also for cost exploration purposes.

It also removed 1 key called resources. Serverless Framework enthusiasts will know that resources [1] contains the AWS CloudFormation Resources that we'd like to deploy. Since the key got removed, it also removed any resources that were being managed, namely the CoolS3Bucket and CoolS3BucketPolicy. It contained objects that can only be accessed through AWS CloudFront, using custom headers [2].

After the PR was merged, the new CloudFormation Template was deployed to Alpha, Staging and then Production, and caused the website to go down for a little while 😅

While it eventually got fixed by adding in the Bucket Policy manually to the S3 Bucket, we now have a scenario where if we reintroduce the resources key (reverting the PR is a quick way), it would cause deployments to Production to fail, because the CloudFormation Template wants to "create" a new resource called cool-s3-bucket but fails to "create" it, since it already exists. For fun, I tried to help fix their issue.

Analysis

The first thing I checked was when the new CloudFormation Template (without any resources) was deployed to update the CloudFormation Stack, did it throw an error when attempting to remove a S3 Bucket that contained objects? Turns out based on the Events Tab for the CloudFormation Stack, it did try to delete the S3 bucket, but could not since there were objects within it, which is expected. However the CloudFormation Stack just outright removed the S3 Bucket and S3 Bucket Policy Resource that it managed. This felt weird because there was an error that occurred with the Deletion of the S3 bucket but the CloudFormation Stack Update silently succeeded stating Update successful. One or more resources could not be deleted. I expected an UPDATE_ROLLBACK_* to appear, but it didn't 🤷.

Two options came to mind to help fix the current issue:

1) Create, Update, Delete, Recreate

Create a new S3 Bucket cool-s3-bucket-2 and S3 Bucket policy entirely
Update AWS CloudFront to refer to the new S3 Bucket
Delete the cool-s3-bucket Resource
Recreate the S3 Bucket named cool-s3-bucket with the Policy
Update AWS CloudFront to refer back to the old S3 Bucket
Delete the new S3 Bucket cool-s3-bucket-2 so that we're back to the same position we were before the PR was introduced

With this approach, it just works and while there would be a bunch of resource deletions, it should bring things back to normal. I was confident about it, since it looks pretty standard in implementation, while avoiding any downtime.

2) Import Resources

I would glance by the Import CloudFormation Resources feature, but was suspicious about it, mainly in terms of does it actually work? I had refrained from using it thinking that it would mess up the CloudFormation Resources that were being imported versus the CloudFormation Resources intended to be deployed through serverless.yml. But it was an option and so to help alleviate the mystery of Importing CloudFormation Resources, looked at understanding it by creating a simple POC to see if it works.

I setup the following for my POC:

Manually created the S3 Bucket named test-cool-s3-bucket
Manually setup a random S3 Bucket Policy
Crafted and deployed a CloudFormation Template with minimal resources (example: Log Group Resource)

I clicked on the Stack Actions -> Import Resources into stack option and just read through the simple instructions provided. From there, crafted a template that contained S3 Bucket, and S3 Bucket Policy

# cloudformation-template.yml converted to json after

Resources:
  TestCoolS3Bucket:
    Type: AWS::S3::Bucket
    Properties: ...

  TestCoolS3BucketPolicy:
    Type: AWS::S3::BucketPolicy
    Properties: ...

The Import Wizard asked for the BucketName to import and after verifying the list of resources being imported, clicked on the Import Resources Button. It was able to successfully import my manually created resources into the CloudFormation Stack. I tried updating values within the CloudFormation Template, and was able to successfully deploy an updated CloudFormation Template.

Results

🎉 Since the Import Resources method POC worked, I suggested a plan of steps that needed to be taken. I Crafted the CloudFormation Templates for all our stages/environments (Example: Alpha Staging Production etc.) and we followed those steps for Alpha. Once it had successfully imported the resources, we tried to rebuild the Jenkins pipeline and the job was successful. The Alpha CloudFormation Stack was back to normal!

We then asked our OPs team to execute those steps to import the CloudFormation Resources into the CloudFormation Stack for the remaining stages/environments, and verified the Jenkins Deployments were successful all the way till Production!

Things that make you go `hmmmm`

Q) Were there any quality gates as such that would have caught this?

Ans) From what I had observed, seems like the PR being reviewed was the only quality gate. There weren't any diffs being generated for the current CloudFormation Template with the proposed CloudFormation Template. Action items were created later on, to help create more quality gates for seemingly sensitive infrastructure changes.

References

Saving 90% of our AWS Cost using ECR Lifecycle Rules

Prithvi Jethwa — Sat, 16 Mar 2024 18:28:41 +0000

Context

For fun, I logged into an AWS account that is used for development purposes. Due to it's purpose, there was a high chance that resources may have been dangling around and not being cleaned up.

Exploring in the AWS Costs Explorer, I took a quick look at the different services, and I came across the ECR Service, which was around $16 per day (around $480 per month). Switching to Dimension: Usage Type seems like most of the cost was for an attribute called TimedStorage-BytesHrs.

I opened random ECR Private Repos that were manually setup before through a bash script. I found one in particular where the ECR Repo had around 9000+ Images, and most of it was prefixed with tag named as dev-*. Just scrolling through each image size was around 600 MB, and existed as far back as 2021. Oh boy, I had to find out what it costed us.

Analysis

I had to get the size of the Repository, and hence used the AWS CLI command to retrieve the information using AWS CloudShell

aws ecr describe-images --repository-name 'repo-a' --query 'imageDetails[*].imageSizeInBytes' | jq '.[]' | awk '{sum+=0} END{print sum}'

Copied over the Bytes Value, and converting into GB

4,236,629,104,429 Bytes/1024 = 4,137,333,109.79 KB
4,137,333,109.79 KB/1024 = 4,040,364.36 MB
4,040,364.36 MB/1024 = 3,945.66 GB

Yeah that's a lotta GBs. Taking a quick look at ECR Pricing, it's $0.10 per GB / month, so doing some quick maths

3,945.66 GB * $0.1 GB per month = $ 394 per month potentially

This was just for a single ECR Private Repository, which seemed super weird that we were paying this for a long while. Oh man, if this is for 1 ECR Repository, what about the others?

Quick ChatGPT queries later, had got the AWS CLI commands to call

 #!/bin/bash

 repositories=$(aws ecr describe-repositories --query 'repositories[*].repositoryName' --output text)

 for repo in $repositories; do
         size=$(aws ecr describe-images --repository-name $repo --query 'imageDetails[*].imageSizeInBytes' | jq 'if length == 0 then 0 else .[] end' | awk '{sum+=$0} END{print sum}')
         leng=$(aws ecr list-images --repository-name $repo | jq '.imageIds | unique_by(.imageDigest) | length')
         echo "Repository: $repo, num-of-images: $leng, Total Size: $size Bytes"
 done

Called this script within AWS CloudShell and piped the output

bash cool.sh | tee cool.txt

From here I wanted to get a list of ECR Private Repos sorted by Size so I used the following

 awk '{sum=$7/1024/1024/1024} {cost=sum*0.1} {printf("%s\tnum-of-images: %s\t%s GB\t\$%s\n",$2,$4,sum,cost)}' cool.txt | column -t | sort -n -k4 > sorted-cool.txt

I was able to get a list of all the ECR Private Repositories, along with the Total size of the given repository, and the potential cost for it

# Rough Numbers

...
repo-d, num-of-images:  3800,  890     GB  $89.0
repo-c, num-of-images:  3100,  1000    GB  $100.0
repo-b, num-of-images:  5000,  1600    GB  $160.0
repo-a, num-of-images:  9800,  3900    GB  $390.0

Where to go from here?

I started with providing visibility, by making a post on the Slack Channel to the Team that owned those ECR Private Repositories and providing them information about what I had found. I also asked for their permission to act on cleaning out those unused resources. This eventually led to a 1 hour meeting where we identified stale images and ECR Private Repos that should definitely be deleted.

Next I gathered screenshots of all the ECR Privates Repos we would intend to purge, to get "evidence" of a before and after comparison.

After all the "evidences" were validated, sent the following message in the Slack Thread.

I intend to start this procedure by 12:45PM

[ℹ️ Turn the Ship Around! is a nice book, learned about I intend to before doing something.]

I went into the 9 ECR Private Repos and manually added the following Lifecycle Rules

1 Untagged    expire | sinceImagePushed(7 days) | untagged
2 dev-*       expire | sinceImagePushed(14 days) | tagged prefix [dev-]

This essential means:

Delete any untagged images that are 7 days old
Delete any tagged images with prefix dev- that are 14 days old

After adding it in, we waited for a couple of days for the Lifecycle Rules to execute.

Results

We saw some great results

# Before:
# Rough Numbers

...
repo-d, num-of-images:  3800,  890     GB  $89.0
repo-c, num-of-images:  3100,  1000    GB  $100.0
repo-b, num-of-images:  5000,  1600    GB  $160.0
repo-a, num-of-images:  9800,  3900    GB  $390.0

# After
# Rough Numbers

...
repo-d, num-of-images:  420,  90     GB  $9.0
repo-c, num-of-images:  360,  120    GB  $12
repo-b, num-of-images:  480,  150    GB  $15
repo-a, num-of-images:  670,  330    GB  $33

🤑 🎉 Looking at the ECR Costs in Cost Explorer, we saw that the Storage costs of the image in the ECR Private Repos (Attribute: TimedStorage-ByteHrs) dropped to around $1.48 per day ($44.4 per month) compared to the $16 per day (around $480 per month) which was ~90% savings on our AWS bills!

It was good to share these results with the Team.
One thing that I'd look forward to how to is how to implement this at scale. Since that's a lotta savings

Things that make you go `hmmmm`

Q) How did this flow under the radar?

Ans) I think that it was a mix of :

Assumptions that there were checks and balances (like the ECR Lifecycle Rules) were already setup
Changing the Docker Tags that were being used across the 4 years
No observability in terms of size of the ECR Repository
Change in Priorities and Teams might have caused this to flow under the Radar

Q) You mentioned manually adding, Why not automate it using a script?

Ans) I think that it was best to start off with manually adding in the rules to help remove most of the costs, and then acting on how this can be scaled out for all repositories.

Q) You mentioned the ECR Repos manually setup using a bash script? Why not IaC?

Unluckly, when the project initially began, there was a bash script that was provided and run manually to help setup the ECR Private Repos. There is clearly room for improvement in terms of converting it all into IaC that can be managed

References

Resolving 'provided principal was invalid' for AWS::Lambda::Permission

Prithvi Jethwa — Fri, 08 Mar 2024 19:51:59 +0000

Context

My friend puts the following Error Message on a Slack Channel that our Team monitors and provides assistance in

coolLambdaPermission | UPDATE_FAILED | Resource handler returned message: "The provided principal was invalid. Please check the principal and trying again" (Service: Lambda, Status Code: 400 ...)

He also provides some details on the Jenkins Job that showed this error message, and wanted to get some help. I was free so I jumped into resolving it.

Issue

The AWS::Lambda::Permission CloudFormation Resource would fail to update within the CloudFormation Stack, due to the Principal provided being invalid (based on the error message), even though the Principal provided within the CloudFormation Template was verified as valid.

Debugging

The first step I took was checking the AWS CloudFormation Events Tab to see the error message, and as seen the AWS::Lambda::Permission CloudFormation Resource failed to update.

Looking into the CloudFormation Template that initiated the Stack Update, I verified the the AWS::IAM::Role used within the AWS::Lambda::Permission Principal attribute existed and was valid.

I checked the Policy Permissions and Trust Relationship for the given IAM Role, and compared it with similar CloudFormation Stacks that use the same IAM Role for the AWS::Lambda::Permission resource, but things looked fine.

Quick Google-Fu searches later, I didn't really get anywhere. My Manager took a quick look and mentioned that the Lambda Permission has an IAM Role associated, which does not exist since it was deleted, which might be causing issues for CloudFormation Stack Updates 🤷

🤔 Since the CloudFormation Template changes involved the following

# serverless.yml

coolLambdaPermission:
  Type: "AWS::Lambda::Permission"
  Properties:
    FunctionName: ...
    Action: ...
-    Principal: ${self:custom.deletedRole}
+    Principal: ${self:custom.validRole}

I would have expected the new Valid Role to replace the old Deleted Role when the CloudFormation Stack is being updated, which it's exactly trying to do.

For a Sanity check, I went into the Lambda Resource, associated with the Lambda Permission, available under Lambda -> Configuration Tab -> Permissions Tab -> Resource-based policy statements and verified the Deleted Role is present in the Principal, but the actual IAM Role is deleted.

I concluded that it must be the AWS::Lambda::Permission Physical ID that might be causing the error and hence suggested my friend to create a PR that updates that resource

# serverless.yml

- coolLambdaPermission:
+ coolLambdaPermissions:
  Type: "AWS::Lambda::Permission"
  Properties:
    FunctionName: ...
    Action: ...
-    Principal: ${self:custom.deletedRole}
+    Principal: ${self:custom.validRole}

This would mean that CloudFormation would create a new AWS::Lambda::Permission Resource with a new Physical ID, and the update would go through. Nope ❌, that did not work, It failed with the same error message.

The provided principal was invalid. Please check the principal and trying again

❓ My next question was, Can I manually add in the Valid Role to the Lambda Permission? So I went into the Lambda Console, and attempted to manually add the Valid Role, and it failed with the same error. This is actually great.

Next step was to manually remove the Deleted Role Principal within the Lambda Console, and try to add the Valid Role Principal again, and that worked! 🎉

This seems like the Lambda wants all the Principals provided to be Valid, and only then would it allow for more Principals to be added in.

I suggested my friend to do the following:

Create a PR that removes the AWS::Lambda::Permission altogether

# serverless.yml

- coolLambdaPermission:
-  Type: "AWS::Lambda::Permission"
-  Properties:
-    FunctionName: ...
-    Action: ...
-    Principal: ${self:custom.deletedRole}

Create a follow up PR that adds the AWS::Lambda::Permission Resource back

# serverless.yml

+ coolLambdaPermission:
+  Type: "AWS::Lambda::Permission"
+  Properties:
+    FunctionName: ...
+    Action: ...
+    Principal: ${self:custom.validRole}

✅ That worked and the CloudFormation Stack was updated successfully

Solution

If you have the AWS::Lambda::Permission CloudFormation Resource, which contains a Principal that does not exist anymore, and if you want to update the Principal attribute with a Valid Role, you'll need to remove the AWS::Lambda::Permission CloudFormation Resource first, and then add it back in with the Valid Role
i.e.

❌ Replacing the Principal within the CloudFormation Template and deploying it would not work(atleast for me)

# serverless.yml

- coolLambdaPermission:
+ coolLambdaPermissions:
  Type: "AWS::Lambda::Permission"
  Properties:
    FunctionName: ...
    Action: ...
-    Principal: ${self:custom.deletedRole}
+    Principal: ${self:custom.validRole}

Rather,

✅ Removing the CloudFormation Resource and deploying the CloudFormation Template

# serverless.yml

- coolLambdaPermission:
-  Type: "AWS::Lambda::Permission"
-  Properties:
-    FunctionName: ...
-    Action: ...
-    Principal: ${self:custom.deletedRole}

✅ Adding back the CloudFormation Resource with the new Principal and then deploying the CloudFormation Template

# serverless.yml

+ coolLambdaPermission:
+  Type: "AWS::Lambda::Permission"
+  Properties:
+    FunctionName: ...
+    Action: ...
+    Principal: ${self:custom.validRole}

Notes

I would have assumed that changing the Principal Value within the AWS::Lambda::Permission Resource would have replaced it with a Valid Role, but the oddities of AWS are a mystery.
A question you might ponder is why was the Deleted Role, deleted? It seems like there are Resources that use the Deleted Role? It was because of a migration process that involved removing that Deleted Role since it should not be used anymore. Who started the migration process, ummm me 😬

References

https://docs.aws.amazon.com/AWSCloudFormation/latest/UserGuide/aws-resource-lambda-permission.html

DEV Community: Prithvi Jethwa

CloudFormation Template does count whitespaces in Template Size...

Context

Analysis

Next Steps

Why is this important

Reference

First contribution to Open Source - charmbracelet/huh

Context

How did I start

Why it's important to me

Next Steps

References

Parse UserParameters sent from AWS CodePipeline to AWS Lambda in Go

Context

Solution

References

Importing CloudFormation Resources to help fix deployments to Production

Context

Analysis

Results

Things that make you go hmmmm

References

Saving 90% of our AWS Cost using ECR Lifecycle Rules

Context

Analysis

Where to go from here?

Results

Things that make you go hmmmm

References

Resolving 'provided principal was invalid' for AWS::Lambda::Permission

Context

Issue

Debugging

Solution

Notes

References

Things that make you go `hmmmm`

Things that make you go `hmmmm`