DEV Community: Marty Henderson

The Fool's Journey (through AWS)

Marty Henderson — Thu, 20 Jun 2024 13:09:44 +0000

Tarot

Tarot cards are sometimes used as a form divination, but they are historically based on several suits or Arcana. Although you can study Tarot for quite a while and come up with different ways of interpreting a reading of Tarot, in Cartomancy Tarot the "main" suits, or Major Arcana, tell a life's journey. Numerically, the "0" or first card in the journey is called the Fool.

The Fool

The Fool, contrary to English colloquialism, doesn't represent idiocy or stupidity, but rather ignorance. We all begin as a fool in life, lacking experience or going through the other Arcana. Indeed, most journeys start with a total lack of knowledge, but not a lack of common sense or will. The Fool starts the journey.

Your AWS cloud journey will start the same way and go through the various Major Arcana as you climb to total AWS knowledge - a literally impossible task, but a goal to strive for nonetheless. In tarot, this last step is known as The World. With the World, you have all the knowledge and experience possible. Quite a grand goal in life, let alone on your AWS journey.

(It should be noted that there are different interpretations of the cards, in various languages, and that I will mostly stick to the Rider-Waite terms, but in some, the World may be known as the Universe but have similar interpretations)

Going through all 22 Major Arcana and what they mean for your journey would be a book onto itself, but I'd like to highlight some places you will go and show you that it is definitely okay.

The Magician

The first card after the Fool in is the Magician. The Magician itself can represent a few things, but the focus here is on potential. The Magician represents focused potential in the field, and this is where everyone starts.

It doesn't matter if you've never opened a bash shell before or if you're able to do subnetting in your sleep. Every cloud journey begins with potential and having it focused on AWS learning is important.

Some tips to get started into The Magician step of your journey is to commit to a goal, whether it's attaining a certification (such as the Cloud Practitioner), doing a specific task (such as setting up a blog), or simply measuring your knowledge (such as teaching a friend about what you learned). This helps your potential have a solid focus which lets your journey begin.

The Emperor

The fifth card is called the Emperor. The Emperor represents authority and structure, which is a hard thing to accept on your journey. AWS specifically makes it a bit easier, as they offer white papers and blogs on specific topics or how things are supposed to be done, but accepting the "how things are done" step is important. Eventually, you might have a good reason to deviate from authority, but at this phase of your journey, learning, knowing, and following best practices is critical. For example, AWS recommends using accounts to isolate workloads. Instead of having all your cloud functions in one shared account, this isolation protects other workloads (in what is usually referred to as "blast radius"). Violating this best practice can lead to major issues when something starts to blow up or mutate over time and creates a potentially risky or fragile structure.

Strength

Strength Arcana, the 12th one, represents something a little different and something we all want - a success. Eventually, in your journey, you will overcome a series of obstacles and eventually deploy a project to production.

Enjoy it! It's a beautiful moment when you can see your work live, whether it's a certification, a website you can click on, watching metrics as others use what you've built, or something between. Your first major success is an incredibly fulfilling moment, knowing your journey has led there.

Personally, I recall deploying an EKS Cluster on AWS (back when EKS was still in preview!) and putting my first confluent control plane on it. Years and many deployments later, I still remember the moment an application did it's first message in a queue on the Kafka setup I deployed. It's an incredible step on your journey and one you should hold tightly to on your journey - but acknowledge your journey is far from done. One success does not unlock all AWS knowledge to you!

Temperance

Temperance, the 15th Arcana, can be about many things, but primarily about moderation. At some point, you will architect beautiful, stable designs, following best practices and show them to reviewers that will complain about expense, cognitive load, engineering overhead, or straight up complexity. Even though your design is probably the ideal one for a theoretical situation and solution, it will need to be tempered by frugality, both fiscal and labor, as well as moderation of usage.

I've seen a number of web-based services use EC2 as their primary engine when ECS or even Lambdas would be a more elegant and faster solution. However, it will come to a point that the cost of an EC2 instance is worth it to an organization to reduce the engineering labor. Perhaps they have an excellent patching policy and don't yet have an understanding of how serverless truly works and worry about noisy neighbors or having their data left behind for another function to scoop up. There are also times where we might want a good service over another good one - such as leveraging Bedrock's models instead of making your own with SageMaker, but the fear of Generative AI forces usage of SageMaker instead.

On my journey, I spent a lot of time in Temperance, and honestly, probably come back to it the most for when I talk about the journey most people take. Don't expect to rush through your journey, the experience is more important than becoming the perfect architect or engineer. After all, journey before destination.

The Tower

Coming in at number 17, the Tower is an Arcana that represents upheaval or trouble. Eventually, you will own a solution and it will fail. Perhaps it simply goes down or failover fails or a malicious actor breaks it. Regardless of how it happens, your design will one day fail because you didn't consider something. This usually happens once you're more senior and can own complex architectures and designs.

This step in the journey is not one unique to your cloud journey, either. There's always the ongoing joke that you aren't a senior developer until you've taken down production environments. However, this is also the greatest learning experience on your journey - not only learning about why your implementation failed, but also learning how those around you act. What happens if your design broke something serious enough to involve a C-level officer or Legal? How the people interact with your failure is just as important, perhaps more, than your cloud service's failure.

The Tower is also one of the most stressful as it can lead to being removed from a project or being let go from a job. This is not the end of the journey! You not only can, but must learn how to recover from trouble to continue down the path you want to walk. The focus you learned during the Magician step of your journey should help you persist through the Tower, but it is never easy.

The Star

The Star, the 19th Arcana, is where I feel I am at - it represents hope and opportunity. As you walk your journey, there eventually comes a time where you've built the previous Arcana and what you've learned from them to a bright spot. Opportunities happen, whether they are careers or others, such as being an AWS Community Builder. It is here that you can truly hold great titles in Cloud and work on complex and overwhelming projects with success and deliver to expectations to others around you and also your self-held expectations.

The Star is an exciting part of your journey and one to definitely enjoy. I am not sure how long I will be on the Star step - hopefully longer than Temperance! - but I have yet to understand when you're stepping to the next part of the journey.

Beyond the Star

This is where I leave you.

There is more to learn and more things that need to happen with me to keep growing through the journey. Everyone's journey - yours, mine, your colleagues, your friends, everyone's - will be different. Perhaps you will move faster than I did. Or slower. Maybe you will have your great failure earlier than anyone else, but perhaps you will have years and years before the Tower calls for you. You don't really know until you get there.

In summary, your journey will have many steps and many things to learn at each juncture. There are upsides and downsides along the way, but they all build you to be a better engineer or architect.

Special Thanks

My knowledge of things such as tarot are limited, and I owe a great deal of gratitude to Dean of Nullsheen.com for help with understanding of such concepts. Dean is a pretty cool guy working on custom tooling for Shadowrun, discussions about things such as cyberpunk, and other super-interesting topics. His site is definitely worth checking out!

Grilled Cheese and Service Control Policies (SCPs)

Marty Henderson — Wed, 13 Mar 2024 18:19:23 +0000

Today I made, quite possibly, the worst grilled cheese of my life. The cheese was cold, I didn't wait for the pan to get warm before throwing the butter and bread in, and it all just became a too-toasted-bread-and-unmelted-cheese mess. It's quite disappointing that I did this because I am usually proficient at making grilled cheese. Perhaps I should've shown more patience but I was hungry.

Needless to say, I still ate half of it. I was hungry, after all.

But while munching on the sad substitute for a sandwich, I realized that it kind of reminds me of how service control policies often go in half baked, come out bad, and then are hated when, really, a bit more preparedness can help quite a bit. So, let's discuss how Service Control Policies work and some common tips and tricks.

Before Service Control Policies can be enabled

Service Control Policies, or SCPs, are done based on your organization, so AWS Organizations need to be set first. If you don't have an org, you can't even start on SCPs. Plus, if you don't have control of the main organization, whether through console or through automation - the better choice - you can't apply them.

I highly recommend using AWS Orgs, even on personal accounts, so you can divide your projects up and create a "blast radius" 0r places you can test and destroy things without impacting your long-term or more serious projects. My accounts are setup with an org main account, a couple of testing accounts, my "production" account, and my DeepRacer account (mostly for budgeting reasons on that one). It also helps that I control which of my friends can use certain accounts through SSO (using Identity Center and an IdP called Jumpcloud).

Alright, with all the prework done, let's jump into SCPs.

What are SCPs?

SCPs, at their core, are policies that allow or deny certain functions or entire services on a given account. For instance, you can deny any DeepRacer functions on an account with an appropriate policy, prevent removal of GuardDuty, or even enforce tagging.

They feel a lot like IAM policies, which I've written on before. They control down to granular functions if you desire, but they impact an entire account instead of a role or user. They can attach to an account or an entire Organizations Unit (OU) - something out of scope for this discussion, but still applies to this.

{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Effect": "Deny",
      "Action": [
        "cloudwatch:DeleteAlarms",
        "cloudwatch:DeleteDashboards",
        "cloudwatch:DisableAlarmActions",
        "cloudwatch:PutDashboard",
        "cloudwatch:PutMetricAlarm",
        "cloudwatch:SetAlarmState"
      ],
      "Resource": "*"
    }
  ]
}

An SCP contains an allow or deny statement - like IAM policies - but attaches to the account. One of AWS' examples is the "Prevent users from disabling CloudWatch or altering its configuration", which, as expected, prevents users from disabling CloudWatch or altering it's alarms and similar. It looks like this:

As you can see from this build, it denies, on all resources, the ability to delete or create alarms, delete or create dashboards, or snooze an alarm. Applied to an account, this impacts all non-root users - even AdministratorAccess. In fact, an SCP deny overrides IAM allow policies.

The fact it overrides an IAM policy (with denies) is important. A common issue I have seen is administrators and engineers believing they can grant the ability to delete alarms or expecting the AdministratorAccess policy to override it. This is an important consideration when applying a deny SCP as you cannot override it within the account.

You can also layer SCPs and, like everything in AWS, a deny will always override an allow. If you have another SCP like

{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Effect": "Allow",
      "Action": [
        "cloudwatch:PutDashboard",
        "cloudwatch:PutMetricAlarm",
      ],
      "Resource": "*"
    }
  ]
}

The deny policy above would still now allow anyone on the account to create new alarms or dashboards.

Let's discuss some tactics to create better SCPs to use.

Limit by User

Some policies can be applied directly to certain users from the SCP. For example, if you only wanted to let the "IAMAdmin" role to edit IAM policies, you could apply something like

{    
  "Version": "2012-10-17",
  "Statement": [
    {
      "Sid": "DenyAccessWithException",
      "Effect": "Deny",
      "Action": [
        "iam:DeleteRole",
        "iam:DeleteRolePermissionsBoundary",
        "iam:DeleteRolePolicy",
        "iam:PutRolePermissionsBoundary",
        "iam:PutRolePolicy",
        "iam:UpdateAssumeRolePolicy",
        "iam:UpdateRole",
        "iam:UpdateRoleDescription"
      ],
      "Resource": [
        "*"
      ],
      "Condition": {
        "StringNotLike": {
          "aws:PrincipalARN":"arn:aws:iam::*:role/IAMAdmin"
        }
      }
    }
  ]
}

This would prevent all changes to IAM roles (except attachments) unless you're current role is the IAMAdmin role. However, you can apply an even more narrow policy.

A good example of this is if you have a deployer role policy that you don't want people to grant more abilities to, you can tie it directly to that role, such as

{    
  "Version": "2012-10-17",
  "Statement": [
    {
      "Sid": "DenyAccessWithException",
      "Effect": "Deny",
      "Action": [
        "iam:AttachRolePolicy",
        "iam:DeleteRole",
        "iam:DeleteRolePermissionsBoundary",
        "iam:DeleteRolePolicy",
        "iam:DetachRolePolicy",
        "iam:PutRolePermissionsBoundary",
        "iam:PutRolePolicy",
        "iam:UpdateAssumeRolePolicy",
        "iam:UpdateRole",
        "iam:UpdateRoleDescription"
      ],
      "Resource": [
        "arn:aws:iam::*:role/deployer-role"
      ],
      "Condition": {
        "StringNotLike": {
          "aws:PrincipalARN":"arn:aws:iam::*:role/IAMAdmin"
        }
      }
    }
  ]
}

This is an example of a more nuanced policy that likely won't need to be removed in the future. It allows only a specific role, IAMAdmin, to edit the deployer role. This means you can prevent other account access, even AdminAccess from editing the deployer role. These kind of policies prevent what are known as "supply chain attacks" that by gaining access to one thing, they can gain access to other things.

Limit insecure practices

A common use of SCP is to limit practices that can risk security or are otherwise best practices.

A common best practice is secure objects that go into your S3 buckets. The policy recommended by AWS is

{
  "Effect": "Deny",
  "Action": "s3:PutObject",
  "Resource": "*",
  "Condition": {
    "Null": {
      "s3:x-amz-server-side-encryption": "true"
    }
  }
}

What this does is prevent anyone from putting an object in that is not encrypted by server side policies.

However, where this can be a problem is when you have existing accounts and functions that use unencrypted buckets from legacy events. I've come across applications using buckets that don't enforce server-side encryption from 4+ years ago. In this case, this SCP could break your application - yikes!

When applying SCPs, especially hard denials with no exceptions, it's best to test them in lower environments to see if they will stop your legacy applications from running.
Tagging

Another favorite SCP to have is to force tagging. However, it's important to realize what the default tagging policy requires.

The default tagging policy forces two tags - Project and CostCenter - to create EC2 instances or SecretsManager secrets.

{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Sid": "DenyCreateSecretWithNoProjectTag",
      "Effect": "Deny",
      "Action": "secretsmanager:CreateSecret",
      "Resource": "*",
      "Condition": {
        "Null": {
          "aws:RequestTag/Project": "true"
        }
      }
    },
    {
      "Sid": "DenyRunInstanceWithNoProjectTag",
      "Effect": "Deny",
      "Action": "ec2:RunInstances",
      "Resource": [
        "arn:aws:ec2:*:*:instance/*",
        "arn:aws:ec2:*:*:volume/*"
      ],
      "Condition": {
        "Null": {
          "aws:RequestTag/Project": "true"
        }
      }
    },
    {
      "Sid": "DenyCreateSecretWithNoCostCenterTag",
      "Effect": "Deny",
      "Action": "secretsmanager:CreateSecret",
      "Resource": "*",
      "Condition": {
        "Null": {
          "aws:RequestTag/CostCenter": "true"
        }
      }
    },
    {
      "Sid": "DenyRunInstanceWithNoCostCenterTag",
      "Effect": "Deny",
      "Action": "ec2:RunInstances",
      "Resource": [
        "arn:aws:ec2:*:*:instance/*",
        "arn:aws:ec2:*:*:volume/*"
      ],
      "Condition": {
        "Null": {
          "aws:RequestTag/CostCenter": "true"
        }
      }
    }
  ]
}

You can see that it's also Case Sensitive - a tag of costcenter will fail when a tag of CostCenter will work. It can also apply to an entire deploy - if you are deploying with Terraform/OpenTofu without tags, it'll deny the creation of secrets and EC2s and fail! (though using default_tags and amap of tags will help with that)

Summation

Make sure your pan is hot and ready - test your policies in lower environments
Make sure your cheese is room temperature - be patient and craft good policies, not quick policies you'll regret/roll back

Overall, SCPs are a good, powerful tool, but take time to craft them well before throwing them in front of your accounts and make sure to test them before stopping your applications!

Certifications as a Measurement

Marty Henderson — Mon, 22 Jan 2024 21:57:54 +0000

To start this off right, I renewed my AWS Security Specialist certification this week. It was due to expire in early February and, like most of my academic career, I got it done at the last minute.

The good news is that I passed! I am recertified for another 3 years - a complaint I am sure I will make come January of 2027.

However, I decided to try something I rarely do, and took the new certification exam cold - no prep work, no classes, nothing more than having some upbeat tunes as I drove through the snow to the local community college to take the exam in-person. So, let's break down why.

Why try it cold? Isn't that expensive?

Well, the good news is that AWS provides multiple ways to reduce the cost of most of the exams. Every time you pass an exam, you get 50% off the next one. If you go to some AWS conference, you get 50% off an exam, such as going to AWS Re:Invent. Some programs, including the Community Builders program, even provide a free voucher - and this is why I took the leap.

The AWS Community Builders program is focused on, as you'd expect, building a grassroots community about AWS and their products. It includes a lot of things, including good networking, some AWS credits to try and test things, an exclusive Slack instance, and a few other things - like an annual AWS exam.

If you're interested, once a year they take new applications - and they close on 28 Jan this year.

What did I learn taking it cold?

750 Points is cut-off

I got a 768 on the exam, which is enough to pass. 18 fewer points and I would've passed, 19 fewer and I would've failed. Although people should be proud of scores they get, there's nothing differentiating my score from a perfect score on Credly or in the system - a pass is a pass.

The new Security (02) exam is very practical

Without diving into the content, I felt the questions were very reasonable based on my day-to-day job. I have secured systems, entire accounts, introduced Security Hub, GuardDuty, AWS Config, and other related tooling to teams, and all of that was presented in ways that actually happened.

I remember when taking the old version of the exam that it was very theoretical and a lot of the situations were just unrealistic. This time, I think they really tightened down the questions and made sure it was less theory and more "given X situation, what should the security engineer do that's the cheapest/fastest/best practice" with actual steps. Overall, a huge improvement for the validity of the exam.

Taking it in-person helps

I live about 30 minutes from the nearest testing center. Counter to that, I live at home with my lovely partner and two pets. Trying to schedule a time where the pets could be contained, my desk cleaned, and general mayhem under control for 3 hours is a lot more difficult than anticipated. Especially in the winter, where you can't go take the dog on a hike (unless you like the single digit temps), it can be hard to make sure pets aren't trying to come into your office.

This time, I decided to deal with the 30 minute drive through snow-covered roads and take it at a local community college. The experience there was vastly superior to the remote experiences I've had recently. The proctor was very polite, allowed me to take the exam 45 minutes early, made sure I had the key to my stuff on the way out, and the room was silent - even with other test takers. After taking the test, I realized the whole time of travel and duration that I was actually testing was under the 3 hours that I would've had to plan at home.

I feel that the quality of remote proctors and/or how overwhelmed they are is a major impact on my test taking experience. I have a remote-only GitHub certification to do next week that I think will be more stressful than this one, only because I have to prepare a ton more at home, can't start early, will have to get my Drivers' License in focus on my camera, deal with a proctor maybe asking me to move my phone further away and any number of things. Overall, hopeful but not looking forward to it.

Is it a Measurement?

Back to the opening statement, was this exam a measurement? I feel it truly was. I didn't study, but I am in AWS Security in and out on a regular basis. Although there were topics that I don't normally need to know, especially about CloudFormation, it required a good understanding of the toolset and when to apply each tool. I hope that as they refresh the different certifications they take a mindset similar to how they developed this one, where it seems more realistic and less "theory so we can say use X tool for Y problems" type exams that plagued a lot of the high-stakes exams for a number of years.

I think if I took some of the other exams cold, I'd have more of an issue - such as SAP on AWS. As popular as SAP is, I've never once administered or deployed it. Trying something like this without any study would be blind guessing and a waste of mine and AWS and a proctor's time. In that case, it'd also be a measurement. However, studying for it and playing with it some on my personal accounts, I could probably get solid enough to work towards the illustrious "all the AWS certifications" achievement.

I think it's important to remember that certification measure at one particular point in time your mastery - with or without studying - of a topic. I am proud of being able to pass the measurement of Security this time without studying because I know it's truly on me. I'd also have owned a failure - it's important to own your mistakes but I felt confident in my overall AWS Security knowledge and had my AWS Community Builder benefit to make this a study worth doing.

Summary

I feel if I studied for it, I would've done significantly better - but a pass is a pass. The new Security exam is much more realistic and, in my opinion, better. Lastly, the AWS Community Builders program has really enabled me to try different things and learn how AWS works - sometimes by even trying exams without studying!

Next exam though? Definitely not a 3 hour-long specialist exam.