Do you ever tackle a problem and know that you’ve just spent way too much time on it? But you also know that it was worth it? This post (which is a...
For further actions, you may consider blocking this person and/or reporting abuse
Excellent post! I'm looking at designing something similar myself (a completely serverless CMS), using S3, DynamoDB, and Lambda, so this was helpful for considering the design.
One recommendation I would make is that your use case seems perfect for AWS CodeBuild. I have used it a little bit for syncing a repo with S3 using webhooks, but I believe you can also set it up to run on a schedule. There is a free tier offering for 100 build minutes per month. It might save you some (minimal) cost, but considering you've fully automated the pipeline already, it's probably worth it to keep what you have.
I haven't had a chance yet to play around with Lambda and EFS, but it is definitely something that piqued my curiosity when I heard about it. Does Lambda require a NAT, or can it be routed through your IGW directly? For enterprise solutions, I imagine this is not a big deal (especially if you don't need to access the internet), but for small personal projects, obviously, this is a huge cost increase and a bit disappointing.
While writing the above, it also occurred to me that CodeCommit might be a good solution to the NAT problem, assuming your repo does not need to be hosted on GitHub. I'm not sure, but I imagine, that you could put up a VPC endpoint for your Lambda and access the CodeCommit repo without traversing the public internet to clone and do your build.
Glad the post could help a bit. That's the whole reason for sharing!
Lambdas in a VPC require a NAT to reach the internet. That was the snag as soon as I integrated EFS (which has to be in a VPN).
CodeCommit and CodeBuild are definitely possibilities here as well. @esh pointed that out to me as well. It's been a while since I used CodeBuild, so I have to circle back on it.
I would caution you in building your own CMS unless it's as a project to learn about building in the cloud. There are already a ton fo great options out there (most free and open source) that could save you a ton of time.
That said, as a learning project, it's a fun activity to take on!
Good to know about the Lambdas and NAT. A bit disappointed, but not entirely surprised. But, because it's a brand new offering, maybe this will change in the future as the service evolves. For an enterprise solution, this is probably not a deal-breaker, but for a student like me, it's definitely cost-prohibitive for portfolio projects I want to keep going on a long-term basis.
The project is definitely just a learning project! I did a few projects in the past with Lambda and DynamoDB and did not completely understand how they work. I have a much better understanding now, but definitely want to learn more. I am also considering playing around with Aurora serverless and a CMS is a fairly simple use case to integrate all of the above and get some experience with them.
Yes, for work production accounts the NAT gateway cost is usually a drop in the bucket. Though it would be nice to see some sort of slicker solution there for purely serverless setups.
Your CMS efforts sound like a lot of fun. Aurora serverless is very cool and also pretty straight forward. For the DynamoDB piece, have you read Alex Debrie's book dynamodbbook.com/?
I have not, I'll definitely check it out. Thanks for the recommendation!
Code build does not compete well on price, especially on smaller instances. The author would pay at least 10x cost for code build.
After a couple people suggested it, I checked into the latest updates around it. You're 100% spot on.
Even if I could drop build time by 50% to 2m, that's $0.01/build * 750 builds a month for a total of $7.50/mth or 10x!
Nice post! I've been thinking alot recently about the Well-Architected framework and realized it can often be accomplished without needing aws services.
For your goal
You mentioned a few points about GitHub actions
I wanted to share for other readers that it actually is possible to run a python script in GitHub actions on a schedule, for free even!
help.github.com/en/actions/referen...
Have a look. I've used this to replace main cloudwatch scheduled event lambdas for free
I wanted to clarify for other readers that you can associate a separate action for pushes. A scheduled action to run your script and a push trigger for what ever else you need, filtering on paths or branches of needed.
If this is just static content you could also just host on gh pages for free
To tie these back to the AWS Well-Architected Framework am alternative might be
Assuming this is a static website, check. A python script will run as fast as a python script can run where ever it runs ;)
Both GitHub actions and GitHub pages are free.
There's no infra to maintain. No lambda lambdas to write, no IAM credentialing to manage. You check a workflow file into your repo and GitHub manages the operations of executing it.
There's also many moving parts which can sometimes remove operational burden.
A secret store is built into GitHub actions if you need it
Same story here. GitHub scheduling system is pretty reliable. If a script fails it gets rerun on the next pass.
Sweet! Excellent tips. I'll check it out and see if how I can make that work as well.
As expected (and loved) there are always a bunch of ways to solve the problem.
This one might even come in less expensive than what I've laid out. Not because it's actually cheaper computationally but thanks to GitHub absorbing the cost!
Agreed. Love the detail and spirit in this post
Thanks for sharing this! I use the heck out of cloud-init for doing initial config of servers, but never thought to co-opt it to make an ec2 instance into something Lambda-ish. Might come in handy in the future.
Great post! I am the developer of Mailbrew and love when people hack on the product.
Thanks Francesco, I'm loving Mailbrew and signed up immediately when I saw. Super clean and amazingly useful.
Btw, I completely understand why you wouldn't support OPML in the product as it would be a nightmare on the backend. I had originally included that call out in the post but things were getting way, way too long.
Besides, this was a fun thing to solve!
If we are playing AWS cost golf, why not switch to parameter store and save 40p, 50% cost reduction. I don’t think your solution uses any feature of secret manager that isn’t part of parameter store.
I actually get that one a lot and there's a reason not to use Parameter Store and that's intention.
Yes, Secrets Manager is more expensive but if you clearly delineate between sensitive data (Secrets Manager) and configuration options (Parameter Store), you're less likely to run into accidental leaks down the road. It's a nice bit of operational security to help show you and the team what needs a high level of care.
...but yes, Parameter Store is significantly less expensive
Just getting started into AWS and this is great content for me. I like how you explained your decisions in terms of the well-architected framework. Cheers!