DEV Community: Paul Marsicovetere

So you wanna learn some AWS skills huh?

Paul Marsicovetere — Tue, 13 Dec 2022 15:17:30 +0000

A common skill found on many job postings in the Site Reliability Engineering (SRE)/DevOps/Cloud Infrastructure space is knowledge and experience in Amazon Web Services (AWS). While this requirement doesn’t guarantee that AWS will be the cloud provider used forever at each company, or will even be a required skill for the long-term, Amazon the company and AWS the cloud platform have been the market leader for around 11 straight years now. There are typically a variety of questions typically asked from those just starting their cloud learning journey such as “How do you learn AWS?”, “How do you get involved with using AWS?” and “How do you get good at AWS?”. There are truly an endless number of resources to answer those questions, but today I would like to recommend my go-to areas for anyone completely unfamiliar with AWS entirely. As a word of encouragement: If I can learn this, anyone can 🙂

Create your own AWS Account

Creating your own AWS account has great benefits for learning as it can be used for testing out whatever products and services you see fit. Depending on your current employer, you may work in an organization that limits what exactly can be provisioned and where. However, with your own AWS account, you will have free range to learn all about the services without impacting any production environments or money-making applications.

💡 When creating your own brand new AWS account it is important to add the following:

A budget alarm to notify you when the monthly budget is approaching. AWS has a generous free tier for its services, however outside of the free tier limits and timeframes, certain services like Managed NAT Gateways, Elastic IPs or unattached EBS volumes can cause the monthly bill to increase. This budget alarm will help greatly in the event bad actors access and use your account for unwanted activities. You are responsible for the billing of these activities, so being alerted early can help save you from an unexpectedly large bill.

Multi-Factor Authentication (MFA) to the root AWS account and any IAM users created to further reduce the security threat of being compromised. ## Check out AWS Skill Builder

AWS Skill Builder is an online resource that provides many learning courses all for free, and you can sign up using an Amazon account. These courses are provided directly by AWS, so any bias or subjective viewpoints that you may/may not encounter from other non-AWS provided courses are not present. There are learning plans that are focused on both ramping up skills and taking AWS certifications, and for beginners, I would recommend taking a look at the AWS Ramp Up Guides, particularly focusing on the Cloud Foundations Guide. The Cloud Foundations Learning Plan is a 13-hour course that provides an overview of many of the foundational aspects of AWS. The Ramp Up Guides are also geared towards Role, Domain and Industry so there should truly be a guide and learning path for everyone at every level.

Sign up for an AWS certification and subscription courses

To certify, or not to certify: that is the question.

Certifications are sometimes a hot-button topic in our industry, and there are valid arguments for and against obtaining AWS certifications in particular. Do these AWS certifications guarantee you understand AWS completely? Absolutely not. Will an AWS certification demonstrate you have some knowledge of AWS concepts and services? I would argue yes. The truth is that there are some incredible engineers that have zero certifications and are far more talented and knowledgeable than some engineers with certifications, AWS Certifications can assist engineers in their careers and provide a framework for learning. Having AWS certifications can increase the number of job opportunities that an engineer is contacted about, and the knowledge gained can be applied to different projects and services in your professional life. Certifications are not a must-have for everyone, however, they offer some benefits and help demonstrate a basic understanding of certain AWS services.

If you are interested in obtaining an AWS certification, I would advise starting with the Cloud Practitioner and/or Solutions Architect - Associate exam(s). To prepare for these exams, there are some quality online learning courses through the AWS Training and Certification website, A Cloud Guru (my personal favourite), Cantrill or Udemy. These are just some of the many courses available that are geared toward passing the AWS certifications.

Fortunately, at Formidable, we have a generous "All-you-can-eat professional development budget" so if you can have your employer cover the costs of these learning courses, that makes it all the more enticing to sign up!

Challenge yourself by building things!

Sometimes there is no better substitute for learning than getting your hands dirty and seeing what can be created when using AWS. There are countless user instruction guides (for example the AWS Developer Center) on many different and wide-ranging services that can be built on AWS and these are easily found on the AWS website. However, it’s advisable to start with something small and iterate from there as this will provide better motivation to keep your learning progressing. Yes, AWS can seem very daunting due to the sheer size (over 200!) of services provided; but you do not need to start with learning the hardest or most complex tasks to gain an understanding of AWS.

For example; this is my personal blog initially created with S3, CloudFront, Amazon Certificate Management (ACM) and Route53. Since then, Amazon Polly text-to-speech capabilities, and a CI/CD pipeline to automate deployment to S3, have been added as well. Further, an on-demand, ephemeral OpenVPN service that runs on EC2 was also created, demonstrating the different projects that can be created with the many AWS services available. The learnings gained when building out these services, alongside my professional duties, have been immense; but you do not need to create the world’s most sophisticated service to learn AWS well. Building new projects and tools as well as using the services on AWS provides knowledge and experience that cannot be replicated simply by sitting certification exams or undertaking learning classes and courses. Learning how to successfully work with CloudFront cache invalidation to update a website’s static assets is much more valuable than simply knowing about that feature. While a certification or learning plan will explain this particular concept and why it is important, in my opinion, the knowledge becomes ingrained when you run the commands and see the results for yourself.

A good way to build something and gain knowledge in AWS I have come across is The Cloud Resume Challenge - AWS. It requires the usage of some key AWS services and is truly awesome for folks just getting started with AWS or software engineering as well.

Conclusion

If anyone looks online or asks about the best way to learn AWS, they will receive a million different opinions and answers. Truthfully, there is no right or wrong way to gain a deeper understanding of AWS, just different learning paths you can take to get there.

It is important to try and understand what your ideal learning style is and center your learning path around that for the best chance at success. Some engineers can simply read pages upon pages of technical documentation (and some who get by only reading the Linux man docs!) and that has worked very well for them. Others, like myself, enjoy video tutorials with lab-based learning, along with building out personal projects.

So whatever, your learning style is, whether it be reading the technical documentation, watching video tutorials, or drawing mind maps, try to cater and tailor your AWS learning path around that and the knowledge will follow. Good luck to everyone on their AWS learning journey!

Version 1 Terraform AWS OpenVPN Ephemeral Released!

Paul Marsicovetere — Thu, 13 Oct 2022 14:26:57 +0000

I have had a little spare time recently to revisit my beloved terraform-aws-openvpn-ephemeral repo and module and give it a much-needed cleanup. As a result, I've now published an official version 1.0.0 and I am super stoked with how it operates! Further, I even set up an openvpn-ephemeral-github-actions-template repo using GitHub Actions as another method to use CI/CD with this module, building off the previous work using GitLab and CircleCI.

Build Back Better

It's been about a year since I last made any meaningful updates to the openvpn-ephemeral terraform module as I was attempting to use the service from my machine and noticed a few different odd and interesting errors. As I reviewed the module, I had a "wow, this code is terrible" moment as I questioned the architecture choices I had made previously. In a way, I am happy because while I could have easily made these newer architectural decisions back then, the service and module were still launched and usable this time last year. Now after reviewing the architecture I have realized that it was way overcomplicated (and still might be) but as they say "perfect is the enemy of good enough". Version 0 previously released worked just fine, but now, it works even better.

So what were the changes? For starters, the previous setup required AWS SSM Manager to download the created OpenVPN file created by the EC2 instance. AWS SSM Manager is in my opinion, non-trivial to set up, especially when you are relying on your own personal machine or containers in CI/CD to use AWS SSM Manager to download the files you need from an EC2 instance. An easier solution? S3!

By swapping out downloading the OpenVPN file via AWS SSM Manager for S3, basically, Terraform creates a private S3 bucket and the EC2 instance creates the OpenVPN file and uploads the file there. Then, Terraform also downloads the OpenVPN file from S3 for you. This negates the need for a tls_private_key pem certificate and aws_key_pair resources as well! Having to use AWS SSM Manager to scp the file securely is a bit clunky and be just as securely achieved using a private S3 bucket that only your AWS Account has access to.

Terragrunt

Another thing I realized was that using local Terraform state and passing it around in CI/CD for GitLab and CircleCI was also completely unnecessary and can be efficiently changed by using Terragrunt. Terragrunt has a superior way than Terraform to set up a remote S3/DynamoDB backend for your Terraform State and Locks, and example terragrunt.hcl files are provided in the updated examples.

Using a remote S3/DynamoDB backend for the Terraform State means that you can set up a private repo in SCM and then download that repo from any machine, run your Terragrunt commands and the OpenVPN instance will be created for you. This has great advantages in case the state is ever compromised mid-run; you can simply download the state elsewhere and fix things, rather than having to manually go and clean up the resources previously.

Improved Docker Image

One of the changes I am most proud and fond of is the creation of the terragrunt-awscli Docker image. The image is created with just two lines!

FROM alpine/terragrunt
RUN apk add --no-cache aws-cli

The alpine/terragrunt image has Terragrunt and Terraform, so this new terragrunt-awscli image just adds the aws-cli on top. This is done as Terraform downloads the OpenVPN file from S3 via the AWS CLI, so it needs to be present.

Previously, the Docker image terraform-aws-ssh was used. This image was around ~272MB whereas the terragrunt-awscli image is only 90MB!

This should result in faster builds in CI/CD, saving precious minutes on the free tier in GitLab, CircleCI and ... GitHub Actions!

GitHub Actions

I have been using GitHub Actions in my professional life recently, and it is quite a different CI/CD tool compared to GitLab, CircleCI or Jenkins, to which I have previously been exposed.

However, I have been impressed with GitHub Actions simplicity. I've come to accept that to have a decent CI/CD pipeline, you need to write some decent YAML. I put together this openvpn-ephemeral-github-actions-template repo, similar to the circleci-template and gitlab-template counterparts to show how to use GitHub Actions with the OpenVPN Terraform module.

It kind of sucks that with GitHub Actions you have to have separate workflows for OpenVPN create and destroy processes, as having "manual approvals" or pauses in workflows is quite convoluted to set up in GitHub Actions. However, this is not the end of the world and honestly makes for an easy UI experience for start and stop. I would rank GitHub Actions as a solid second choice behind GitLab for using CI/CD with this workflow. GitLab slightly edges out and takes the first position because it has Manual Approval buttons for each create and destroy jobs, so you can have one workflow file for all your regions, rather than have these separated as you do for GitHub Actions. The CircleCI workflow is definitely still the least desirable of the three 😀

GitHub Actions also has a decent free tier of 2,000 CI/CD minutes per month, equating to ~500 minutes per week. The build and destroy jobs take roughly 4-5 mins each at max so the VPN could be started and stopped around 14 times per day before you would hit the free tier limits. More than enough for my personal use case!

Where to next?

Not being able to sit still, I am already thinking about how to make this service offering even better, and there are certainly a lot of obvious solutions to consider. I will probably set up things in Terraform as an Auto Scaling Group that scales to 0 when you are done with the instance. This would allow for a faster startup time (presumably) as you are not having to create the instance, role, policy and bucket from scratch each time you wish to create.

I definitely also want to use the learnings from version 1 and earlier, to set up a service that has a complete front-end for even easier use when creating OpenVPN access. A simple password-protected site where you click the region you want a VPN for and it downloads the OpenVPN profile file for you surely can't be that hard right?

Recap

Refactoring your code is awesome, and you should try and use fewer resources whenever possible
GitHub Actions is a solid CI/CD tool for starting/stopping OpenVPN EC2 instances
Releasing Version 1 of anything will release your own endorphins!

OIDC Forever, IAM Credentials Never!

Paul Marsicovetere — Thu, 08 Sep 2022 17:23:57 +0000

Let me ask you a difficult question: how often are you really rotating your IAM credentials? Secondly, if there were a security breach and someone had access to those IAM credentials, how quickly would you be able to detect it?

Security is something that nobody in tech likes to take for granted and is an area that is constantly changing and evolving as more and more threats are detected every day.

About 8 months or so ago I was thinking to myself "I really need to update and rotate the IAM Credentials I use for this blog". I was namely thinking of this because while it is good security practice to rotate your keys, unfortunately, there's no "automatic" way to do this within AWS, short of writing a Lambda and hooking this into Secrets Manager and/or SES for reminders of key rotation. I was researching the most straightforward way to automate these key renewals say every 90 days and then auto-uploading those keys to the services that use these particular keys (in my case Terraform Cloud and CircleCI). However, after spending a few hours digging in, I quickly realized that this would not be that straightforward and that it was actually way faster for me to simply log into my AWS account every 90 days, rotate the keys, and then upload these to CircleCI and Terraform.

However, I recently listened to this AWS Bites podcast episode and thought "hrmm...ok, maybe I'll just switch up my blog's CI/CD to GitHub Actions so I can use this awesome OpenID Connect (OIDC) functionality".

And then I read this great CircleCI blog post 😄

Understanding why OIDC is superior to IAM credentials

So the standard process has typically been to: create an IAM user with programmatic keys, give that user permissions of least privilege, and then upload the IAM credential keys to the service provider that will be making changes to your AWS infrastructure (again in my case Terraform Cloud and CircleCI) on your behalf.

When you really think about it, storing your IAM credentials in any service provider has its limitations. I like to think of providing IAM credentials to a service similar to giving my credit card details to a service (e.g. CircleCI). I trust CircleCI to not do anything nefarious with my details, but if my card details are stored in the service and that service for whatever reason becomes compromised, that credit card can be used by anyone (or any service) that has the details. This is no different from IAM credentials and provides us with a good reason as to why you always want to pair down the permissions.

side note: please, please don't create an IAM user with programmatic keys that has AdministratorAccess permissions to your AWS Account that you then upload those IAM credentials to these third-party services. You are basically giving anyone or any service the ability to create anything (well...within service quota limits) within your AWS account by doing so. You personally don't want that or want to deal with an unexpected large AWS bill shortly after! 😅

Continuing the credit card analogy; so instead of giving our credit card details to a service, what if there is a way to give only card authorization to that one service? Well, that sounds more like OIDC! The benefit is that you are only authorizing/allowing the service to use your credit card, meaning that there are no credit card details to be leaked, only card authorization details. If the card authorization details were somehow leaked, these are actually kind of worthless, as only that particular service is authorized to use them, not some other random third party! 🎉

I'm confident there will come a day soon when some hacker will figure out how to spoof the service that has the OIDC Identity provider authorization permission (e.g. a hacker spoofing CircleCI can then access your AWS infrastructure based on the policies attached to the OIDC provider), but for now this setup seems a whole lot more secure than storing your IAM credentials directly at the service and having to remember to rotate your access keys every so often. Further, if the IAM credentials are leaked, you really have to rotate them quickly or risk your AWS account/services being compromised.

How to setup an OIDC Identity Provider with CircleCI in AWS

Start by logging into CircleCI and then go to Organization Settings > then copy your organization ID
Log into the AWS console, and head to IAM.
Go to Identity Providers (under Access Management) and click "Add Provider"
Choose "OpenID Connect"
In Provider URL enter https://oidc.circleci.com/org/<INSERT ORGANIZATION ID HERE> (e.g. https://oidc.circleci.com/org/th1s-i5-n0t-a-0rg1d) and then click "Get Thumbprint"
In Audience, enter the same Organization ID
Click "Add Provider" and the provider should be created!
Next, go to Roles (under Access Management)
Click "Create Role" and select "Web Identity"
In Identity Provider select the CircleCI identity provider that was just created.
In Audience, select the Organization ID and click Next to add permissions
In Permissions, you will be selecting or creating policies based on what your CircleCI pipelines should have permission to do. This will vary case by case. For example, my CICD pipelines need only access to S3 and CloudFront services to publish my blog posts.
Click Create Role and when complete, copy this Role's ARN, you are good from the AWS perspective!

Using OIDC as your AWS credentials in CircleCI

Before making any changes to your CircleCI pipeline, as per the OpenID Connect Tokens doc, you will need to create a context for your job, which can be done using these steps.

Once you have a context created and added to the workflow section of your pipeline job, you can now edit your CircleCI pipeline to include the OpenID Connect token to utilize our newly created IAM role. For example, using these directions, my .circleci/config.yml file was changed from this:

version: 2.1
jobs:
  www:
    docker:
      - image: paulmarsicloud/hexo-aws:2.1
    working_directory: ~/hexo-cloudonmymindblog
    steps:
      - checkout
      - run:
          name: Generate static website
          command: hexo generate
      - run:
          name: Push to S3 bucket
          command: cd public/ && aws s3 sync . s3://www.thecloudonmymind.com
      - run:
          name: Invalidate Cloudfront
          command: aws cloudfront create-invalidation --distribution-id <REDACTED> --paths "/*" --no-cli-pager
workflows:
    version: 2
    build_and_deploy:
      jobs:
        - www:
            context:
              - cloudonmymind-web

to this:

version: 2.1
jobs:
  www:
    docker:
      - image: paulmarsicloud/hexo-aws:2.1
    environment:
      AWS_DEFAULT_REGION: <ENTER YOUR REGION HERE>
      AWS_ROLE_ARN: <ENTER YOUR IAM ROLE ARN HERE>
    working_directory: ~/hexo-cloudonmymindblog
    steps:
      - checkout
      - run:
          name: Generate static website
          command: hexo generate
      - run:
          name: authenticate-and-interact, Push to S3 bucket, Invalidate CloudFront
          command: |
            # use the OpenID Connect token to obtain AWS credentials
            read -r AWS_ACCESS_KEY_ID AWS_SECRET_ACCESS_KEY AWS_SESSION_TOKEN \<<< \
              $(aws sts assume-role-with-web-identity \
               --role-arn ${AWS_ROLE_ARN} \
               --role-session-name "CircleCI-${CIRCLE_WORKFLOW_ID}-${CIRCLE_JOB}" \
               --web-identity-token $CIRCLE_OIDC_TOKEN \
               --duration-seconds 3600 \
               --query 'Credentials.[AccessKeyId,SecretAccessKey,SessionToken]' \
               --output text)
            export AWS_ACCESS_KEY_ID AWS_SECRET_ACCESS_KEY AWS_SESSION_TOKEN
            # interact with AWS
            aws sts get-caller-identity --no-cli-pager
            cd public/ && aws s3 sync . s3://www.thecloudonmymind.com
            aws cloudfront create-invalidation --distribution-id <REDACTED> --paths "/*" --no-cli-pager
workflows:
    version: 2
    build_and_deploy:
      jobs:
        - www:
            context:
              - cloudonmymind-web

So for the eagle-eyed reader, you'll notice a few interesting pieces. Firstly, the addition of --no-cli-pager is necessary probably only for the docker image used paulmarsicloud/hexo-aws:2.1. This is a custom docker image I created that has both hexo and the aws-cli installed, as this blog/site uses the Hexo platform. By adding the --no-cli-pager option, I no longer receive a Unable to redirect output to pager. error 👍

Secondly, I had to combine the Push to S3 bucket and Invalidate Cloudfront steps into one during testing, as it seems that the read and export commands need to be run per step. I tried having just a singular authenticate-and-interact step, but subsequent steps that ran aws cli commands would fail with a Unable to locate credentials. You can configure credentials by running "aws configure" error. I will need to sit down and spend some further time testing out and potentially refactoring my CircleCI steps and pipeline as I am sure there is a way to somehow cache or pass the credentials from one step to another, rather than having to write and execute those read and export commands each time you want to run an aws-cli command. When I have the answer, ya'll will be the first to know!

In a way, it makes more sense to have these S3 and CloudFront AWS commands run in one step, even though for readability purposes, I personally like to have a "one command per step" layout for my CI/CD pipelines. It's truly a "6 in one, half a dozen in the other" kinda deal though.

Here's a link to the gist of my final updated .circleci/config.yml if you need it.

Terraform Cloud

Unfortunately as per this discussion Terraform Cloud does not support OIDC Identity. This is discouraging to see and as a result, I will be moving my Terraform state out of Terraform Cloud shortly and will set up an S3 Bucket/DynamoDB table for the state/lock. I initially set up this blog with Terraform Cloud to try it out and have automated Terraform runs, but now I will be switching to CircleCI (or GitHub Actions) so that I can utilize the awesomeness of OIDC!

Benefits of OIDC over IAM credentials

Sounder sleep knowing that my IAM credentials won't be leaked for someone or some service to use and compromise my AWS infrastructure
No need to set up anything custom or any reminders to manually rotate the keys. All authorization is now done via the OIDC Identity Provider and corresponding IAM role - it was a rad feeling deleting the IAM credentials I had stored in CircleCI and still having the pipeline runs work!
No way of accidentally forgetting to not store your IAM keys in version/source control (this happens to folks all the time, even though everyone warns about it)
OIDC providers are verified by AWS, so there is a super low risk that an OIDC Providers "Thumbprint" would be spoofed for a particular large service (like CircleCI or GitHub Actions). When your IAM credentials are created and are then used for any service, anywhere, AWS doesn't ever verify who exactly is using them. There is an implicit trust that if a user or service has the IAM credentials, they are authorized to use them. It is a totally different ball game with OIDC!

Recap

Think of IAM Credentials as your Credit Card vs an OIDC Provider as authorization to use your Credit Card 🙂
OIDC Providers are quick and simple to setup in AWS and roll into your new or existing CircleCI pipelines
You never need to rotate OIDC providers...you can't say the same for IAM Credentials!