DEV Community: Ben Fellows

Simplifying CloudFormation Deployment

Ben Fellows — Mon, 09 May 2022 01:26:32 +0000

CloudFormation (CFN) is a great templating framework or DSL that you can use to define infrastructure in either YAML or JSON.

Over the years I've watched it develop from something that was relatively difficult to implement to now being widely accepted and easier, but still with some caveats. When I started doing enterprise cloud migrations (~2012) to AWS I was using tools like Ansible to solve the problem of automating IAM, EC2s, RDS and VPC. In addition to this we didn't have several of the tools we have now available for migrating to AWS.

I started using CFN templates to create IAM policies, EC2 templates and Roles and also VPCs. Some of the challenges back then was around the JSON formatting and injecting parameter's values into deployment. This is why I used tools like Ansible and the AWS CLI to wrap around CFN templates. This would give me control around what AMIs I could use for an EC2 for example or what AccountID I could use in a parameter for a Cross Account IAM Role.

** Conditionals

Another issue I found was that when launching a CFN template you are always going to have the issue of conditionals based on Parameter values. For example for an EC2 template if you launch it in a public subnet you might want to have a Public IP, but if you launch it in a private subnet you will only want a private IP.

** User data

User data in EC2 launches can be another problem that is not solved by templates alone. For example you can provide user data to run a command or run a script on initial launch.

Overall the biggest challenge for people who are new to using CloudFormation is learning the syntax and types of resources you need and the parameters available. The documentation provides this, but it can be challenging reading through lines and lines of documentation.

One of the ideas I had was being able to help people learn about AWS CloudFormation and supporting them in being able to create their own templates.

You can try https://app.teemops.com/ to get an idea of what I'm talking about - it allows you to use a basic UI to generate a template and launch it in your own AWS account, saving you hours of frustration learning. The source code is all available on https://github.com/teemops/tui if you want to run locally and play with.

At the moment I only support EC2 and Autoscaling groups, but the idea is to support as many AWS resources as possible. Give it a try and I look forward to feedback.

Introduction to backups in AWS

Ben Fellows — Mon, 11 Apr 2022 22:22:13 +0000

Yesterday I did a live discussion on introduction to backups in AWS. Video link is available at the end of this article.

In this video I explained how you can plan your backups and what tools you can use in AWS to configure backups.

I'm going to summarise the main points here for everyone if you can't be bothered watching the recording of the Livestream.

Planning

First you can start with a company wide SLA for the acceptable downtime for any given application or database and the acceptable time until recovery as well as point in time.

These are the areas you need to plan for an enterprise cloud backup solution:

Normal required hours of service (e.g. 7am-7pm)
Acceptable uptime as a percentage (e.g. 95%, 99%, 99.99%)
Recovery Point Objective (or RPO) as the point in time you want to recover to.
Recovery Time Objective (or RTO) as the maximum time until you recover.

Once you have these negotiated and agreed upon with your business you can design a scalable solution.

Because each enterprise and business unit may have different objectives I found it more simple to provide 2 categories of application:

Mission Critical
Normal Business Hours

Mission Critical applications are your website, phone systems, email, production databases etc.
Normal Business Hours applications might be HRIS, Payroll, Finance.

Again this depends on the type of organisation you are and if you are in several different time zones most of your applications might be mission critical.

Schedule Planning

You need to understand what needs to happen when you have different types of data to backup. For example you might have EC2 servers with files stored on EBS volumes, SQL Server Enterprise on EC2 and EBS volumes. And you may have an open source app or website running on an RDS Instance.

You can have a schedule for each task that is going to be performed. This will depend on the solution you have in place. You want to co-ordinate timing of backups with each other and what the impact is going to be on running systems.

This spreadsheet will give you a good sample schedule for backups and restore process.

https://bit.ly/awsexp-backupspreadsheet

Tools

Let’s look at the combination of an Amazon DLM solution and a SQL Server S3 based solution.

Amazon DLM is for EBS volumes and AMIs, Whereas the database solution is for backing up to S3 and having a lifecycle that sits within S3.

Amazon DLM automates the creation of snapshots and manages the retention based on policies you set. For example you can setup a policy that snapshots all your EBS volumes for your website and set a schedule that runs daily. In addition to this you can have multiple schedules per policy so you could have a policy that has daily, weekly and monthly snapshots.

DLM can automatically copy tags of existing EBS volumes and also create additional tags. I have provided a link to an AWS CloudFormation template you can use for a simple daily backup policy at the end of this article.

For a custom solution you can use S3 as a target location for files you might want to backup into S3. A common use case is backing up SQL Server Backup files to S3. S3 can be configured with Life Cycle rules for deletion or archiving of files.

Auditing

One of the things I didn’t cover in the video was auditing. It is essential that you audit your backups are running. This can be done via some simple code that checks for S3 objects by date and also check for EBS snapshots.

For example, I use some Python code that handles the auditing of backups and ensures the correct files exist. This can be run as a continuous task daily, weekly or monthly. I would recommend running this as often as possible. In addition to this, AWS Lambda can be used with a scheduled Event. (More on that in another post).

Conclusion

Plan your backups, streamline your process and audit your backups. Test your restore process regularly to make sure your backup works as intended.

Video: https://www.twitch.tv/videos/1452480056
Cloudformation for DLM Policy: https://bit.ly/teemdlmcfn

Free resources for AWS

Ben Fellows — Mon, 05 Apr 2021 01:35:41 +0000

Over the last 10 years of work I've created a lot of code, cloudformation templates and documents/diagrams that have helped customers on AWS.

Now I'm slowly open sourcing these to help anyone else become an AWS expert.

Would love your feedback and any requests for any given CloudFormation, Script, AWS CLI commands or any other thing related to AWS.

Thanks for taking a look, the 2 documentation and cloudformation repos are available at: https://github.com/awsexp

I'll be dropping more files over the next few weeks so either star the repo or follow me here on @awsexpert
Thanks!