DEV Community: Geoffrey Wiseman

AI-powered Corporate Jargon Translation with Amazon Bedrock

Geoffrey Wiseman — Wed, 29 Jan 2025 22:53:52 +0000

After watching this Under Studio humorous video showing a translator for business jargon, it made me wonder how far off current language models were from being able to do actually do this kind of translation, particularly with the sort of dark humour that makes this work.

So I did some experimentation, crafted a prompt and started testing some models on Amazon Bedrock.

Claude 3.5 Haiku

Haiku was the first model I found that did a decent job of translating one of the lines from the video with a one-shot prompt (aka, I gave it one example, and then asked it to do a similar job):

I tweaked the prompt to request dark humour and got this:

We're royally screwed, and management is desperately trying to cover their tracks by throwing around fancy words that essentially mean "we have no clue how to fix this massive mess we've created." The "strategic processes" are basically a corporate version of putting duct tape on a sinking ship while pretending everything is totally fine. Expect a round of scapegoating and middle management sacrifices in 3... 2... 1...

Since that went well, I tried a real-world example, pulling this paragraph from a Stripe memo from their latest layoff that was leaked:

As we've been working through our plans for 2025, leaders took a close look at their organizations and team structures. It became clear that there were several team-level changes needed to make sure we have the right people in the right roles and locations to execute against our plans.

This is Haiku's translation:

We're about to unleash a bloodbath of layoffs that will make corporate Hunger Games look like a friendly team-building exercise. "Right people in the right roles" is code for "we're cutting jobs, relocating some poor souls, and making everyone who survives do three people's work for the same pay." Expect a wave of panic, LinkedIn update storms, and sudden "networking" coffee meetings. Buckle up, because corporate restructuring is just a fancy term for "we're trimming the fat" - and by "fat," they mean human beings with mortgages and families. Happy 2025, everyone! 🔪📊

Ouch!

I'm starting to think having a website that can do gallows-humour translation of corporate Jargon would be a fun side project, although I'd probably need to be careful that it didn't rack up massive costs.

Llama 3.3 70B

Meta's Llama 3.3 70B model also did a fine job. I refined the prompt to request dark humour and got this response to the video line:

Management is freaking out because we're hemorrhaging money, and they're desperately trying to duct-tape together some new "strategies" to stop the bleeding before the whole ship sinks. Think of it as rearranging the deck chairs on the Titanic, but with more PowerPoint presentations.

Funny! What about the Stripe paragraph?

Code red: the axe is coming, and HR is sharpening the blade. "Team-level changes" is just corporate speak for "we're about to fire a bunch of people and make the rest of you pick up the slack." So, update your LinkedIn profiles and start practicing your "I'm excited about this new opportunity" face, because it's about to get real.

Awesome. This is definitely a workable side-project.

Amazon Nova Premier

The translation from the video:

Management is freaking out because we're missing targets, and they're scrambling to come up with a plan that might just save their jobs. Expect some serious micromanaging and maybe a few sacrificial lambs thrown to the wolves soon.

And from Stripe:

Management's crystal ball showed them that our current team setup is a hot mess. So, brace yourselves for a round of musical chairs where some of you will end up in roles you're not suited for, and others might just find themselves without a chair at all. Welcome to the 2025 restructuring rodeo!

Maybe not quite as funny as Llama, but still pretty decent.

There's definitely something here. It would be super-funny if you could embed it as a plugin to MS Teams / Zoom as a side-channel for your next all-hands.

Top re:Invent 2024 Videos

Geoffrey Wiseman — Thu, 12 Dec 2024 17:04:21 +0000

Every year, Amazon Web Services holds a conference, "AWS re:Invent", and they release the conference sessions as videos on YouTube. It's a big conference with a lot of sessions, so there are lots of videos to pick from. So much so that it's hard to look at the list of videos and decide which ones you're interested in. If you have specific topics you want to learn about, you can search for videos on that topic, but looking at the full list is not realistic.

This year, while looking at the long list of videos, I thought it would be nice to see which are the most popular videos from re:Invent. YouTube doesn't make it very easy to get that list, so I decided to experiment with a custom-built solution. If you want to skip ahead to the working version, it's available here.

V1: The Experiment

I wrote a short Python script to call the YouTube Data API v3, identify all the videos published to the AWS Events channel in December 2024, and rank by view count.

As an experiment, I'd call it a success -- I got a long list of videos, sorted by views. I shared that list in a few places, but I knew that it wasn't complete. First of all, AWS was still uploading new videos. Secondly, as people continued to watch and share, the view counts would continue to rise, and that would likely impact the rank. In particular, the newer videos had had less time to achieve view count success than the first videos that were posted nearly a week before.

So, although the experiment was success, I wanted more.

V2: Website

If I wanted to run that script to get updated data and be able to share it, the obvious solution was to turn it into single-purpose website that I could share and that anyone could refresh whenever they wanted to get updated data.

I considered options -- I could turn the Python script into a static site generation script that made HTML and deploy that (e.g. to S3), then schedule a job to generate new content every so often.

I didn't want to run and schedule the job on my own infrastructure, and since this was an AWS-related project, I decided I'd like it to run on AWS. I didn't need the job to run super-frequently, so I didn't really want spin up an EC2 instance, so this felt like a good job for Lambda.

I packaged up the script and the google api client into a Lambda .zip file, uploaded it and began some testing. I didn't want to have to redeploy to change simple parameters, so I externalized some of the more obvious parameters into environment variables and added those to the Lambda function as well. I didn't really need a load balancer, so I added a Lambda function URL.

That worked -- but even in testing I was starting to run up against the limitations of the YouTube Data API. In order to prevent abuse (and reduce the cost to their infrastructure), YouTube's Data API has quotas. Each search call uses 100 units of the 10,000 units you're allocated by default, and getting enough information to rank by views uses at least another unit, so you're limited to less than 100 search calls. Each search call returns a maximum of 50 results, so rendering the page once was using up nearly a quarter of my quota, and if I were to share this with others, it would almost immediately stop working.

V3: Caching

Now we're back to architectural choices. If I'm going to use a cache to avoid hitting the YouTube Data API quota, how am I going to do that?

Internal Caching
Do I cache the data from YouTube and only refresh periodically -- once or twice a day? With more intelligent caching, I could even save the video information and on refresh I'd only have to identify new videos and update the video counts on existing videos, which would use far less quota. Then I'd have to store that data somewhere, and since you can't rely on a Lambda staying in memory, that meant an external cache of some kind -- a file on S3, a cache like Redis, or a datastore of some kind, maybe DynamoDB. I was briefly tempted to test out Aurora DSQL.

External Caching
I could also cache externally -- render the HTML and store that instead of the data used to generate the HTML. That also saves the work to generate the HTML, so it's a bit more efficient, although perhaps a little less flexible in some ways.

I could go back to a static site generation plan -- schedule the Lambda to run once or twice a day, take the output from the lambda function and store it on S3, where I could serve it up directly.

The simplest solution seemed to be CloudFront. In theory I could continue to use the Lambda unmodified and CloudFront could cache the results of the Lambda function, and then many page refreshes would simply be CloudFront cache hit.

This isn't quite as flexible as the internal caching, but it saves me from setting up a scheduled job for static site generation, so I decided to try this path.

V4: Resolving Quota Issues

I published the CloudFront URL to a few places, and heard back from a friend who was travelling that he got an Internal Server error. I suspected the quota had been exceeded, and I checked the logs -- sure enough, it had. This revealed two problems in V3: error handling, and CloudFront cache plurality.

I dealt with the error handling first -- I added some code to trap the exception and handle it a little better than the Internal Server Error that was showing. I didn't do anything sophisticated because the goal was to not hit the quota, rather than handle it well.

The CloudFront problem required a small architectural change -- each CloudFront location has its own cache, and if people from all over the world load the Lambda, each location will invoke the Lambda, and since I can only get away with about four invocations in a day on the quota I had, that was going to be a problem.

Fortunately, CloudFront has a solution for that -- you can add Origin Shield, which effectively means that CloudFront will also cache internally. If the first request comes in Toronto, the closest CloudFront location will check its own cache, then the internal cache, and then invoke the lambda. CloudFront will then store the result in the internal cache and within the location cache. The second request might come in from Japan, which will get a cache miss in the location-based cache, but a cache hit from the internal cache, and CloudFront will populate the location cache from the internal cache and avoid invoking the lambda a second time.

This worked well -- the quota errors went away, but then I discovered a new problem. Some of the earliest popular videos were no longer showing in the list.

V5: Resolving Dropped Videos

Was it a bug in the code? I did some testing. Turns out, the YouTube Data API pagination only gets you so many pages -- at some point YouTube doesn't give you another next page token, even though there are more videos that match your criteria.

I tested a few options -- I could break up the month into smaller date ranges and run a search for each range. I was worried this would simply hit the quota faster, but it did seem to work. I had to do a little work to make sure no duplicates showed up, but otherwise it seemed to be ok.

While I was experimenting with that, I discovered that the YouTube Data API had a feature I missed on the first pass - I could order the results by view count. This meant that I no longer needed to iterate through every video in the month, I could simply use YouTube's own knowledge of the most popular videos that matched the search criteria that I'd established.

This also meant I could significantly reduce my API traffic -- I could issue just enough search requests to fill my list and stop. No architectural change, just a code change.

I tested it, and that worked well.

Here's the current version of the Python code. There are still improvements that could be made, certainly, but I'm willing to share imperfect code:

import datetime
import html
import os
import re
from typing import NamedTuple

from googleapiclient.discovery import build
from googleapiclient.errors import HttpError


class TopVideoConfig(NamedTuple):
    """
    Configuration for fetching top videos from YouTube.

    Attributes:
        api_key (str): The API key for accessing the YouTube Data API.
        channel_id (str): The ID of the YouTube channel to fetch videos from.
        published_after (str): The start date for fetching videos (ISO 8601 format).
        published_before (str): The end date for fetching videos (ISO 8601 format).
        max_requests (int): The maximum number of API requests to make.
        max_results (int): The maximum number of video results to return.
    """
    api_key: str
    channel_id: str
    published_after: str
    published_before: str
    max_requests: int
    max_results: int


class VideoResult(NamedTuple):
    """
    One record of top videos from YouTube.
    """
    video_id: str
    title: str
    published_at: str
    view_count: int


class Retriever(object):
    def __init__(self, config: TopVideoConfig):
        self.config = config
        self.youtube = build('youtube', 'v3', developerKey=config.api_key)
        self.retrieved_ids = set()
        self.videos = []
        self.search_requests = 0
        self.title_pattern = r'(^AWS re:Invent 2024\s*-\s*|)'
        self.next_token = None

    def fetch(self) -> list[VideoResult]:
        start_date = self.config.published_before
        while True:
            results = self.fetch_search_page()
            print(f"Videos: {results}")
            self.videos.extend(results)
            if len(results) == 0:
                print("No more results found.")
                break
            if self.search_requests >= self.config.max_requests:
                print(f"Reached max requests limit of {self.config.max_requests}")
                break
            if not self.next_token:
                print("No more pages found.")
                break
        print(f"Retrieved {len(self.videos)} videos using {self.search_requests} requests")
        self.videos.sort(key=lambda v: v.view_count, reverse=True)
        return self.videos

    def fetch_search_page(self) -> list[VideoResult]:
        video_ids = self.fetch_video_ids()
        if len(video_ids) == 0:
            return []
        self.retrieved_ids.update(video_ids)

        video_request = self.youtube.videos().list(
            part='snippet,statistics',
            id=','.join(video_ids)
        )
        response = video_request.execute()
        return [self.transform_video(video) for video in response['items']]

    def fetch_video_ids(self) -> set[str]:
        request = self.youtube.search().list(
            part='id',
            channelId=self.config.channel_id,
            maxResults=50,
            order='viewCount',
            type='video',
            publishedAfter=self.config.published_after,
            publishedBefore=self.config.published_before,
            pageToken=self.next_token
        )
        response = request.execute()
        video_ids = {item['id']['videoId'] for item in response['items']}
        print(f"Retrieved {len(video_ids)} video ids with page token {self.next_token}")
        self.search_requests += 1
        self.next_token = response.get('nextPageToken')
        return video_ids.difference(self.retrieved_ids)

    def transform_video(self, video: dict[str, any]) -> VideoResult:
        return VideoResult(
            video_id=video['id'],
            title=self.transform_title(video['snippet']['title']),
            published_at=video['snippet']['publishedAt'],
            view_count=int(video['statistics']['viewCount'])
        )

    def transform_title(self, title: str) -> str:
        return html.escape(re.sub(self.title_pattern, '', title))


# Load configuration from environment variables
def get_config() -> TopVideoConfig:
    # API Key
    api_key = os.getenv('YOUTUBE_API_KEY')

    # AWS Events channel ID
    channel_id = os.getenv('YOUTUBE_CHANNEL_ID')

    # Dates
    published_after = os.getenv('YOUTUBE_PUBLISHED_AFTER')
    published_before = os.getenv('YOUTUBE_PUBLISHED_BEFORE')
    max_requests = int(os.getenv('MAX_REQUESTS', '1'))
    max_results = int(os.getenv('MAX_RESULTS', '50'))

    return TopVideoConfig(api_key, channel_id, published_after, published_before, max_requests, max_results)


def render_videos(config: TopVideoConfig, videos: list[VideoResult]) -> str:
    response = "<html><head>"
    response += "<title>Top re:Invent 2024 Videos</title>"
    response += "<style>body { font-family: sans-serif; } li { padding-bottom: 10px; }</style>"
    response += "</head><body>"
    response += "<h1>Top re:Invent 2024 Videos</h1>\n"
    response += f"<h3>(as of {datetime.datetime.now().strftime('%Y-%m-%d %H:%M:%S %Z')})</h3>\n"
    response += "<ol>"
    for video in videos[:config.max_results]:
        response += f"<li><a href='https://youtube.com/watch?v={video.video_id}'>{video.title}</a>"
        response += f" ({video.view_count:,} views)</li>\n"
    response += "</ol></body></html>"
    return response


## Bootstrap for Lambda
def lambda_handler(event, context):
    config = get_config()
    try:
        videos = Retriever(config).fetch()
        response = render_videos(config, videos)
        return {
            'statusCode': 200,
            'body': response,
            'headers': {
                'Content-Type': 'text/html; charset=utf-8'
            }
        }
    except HttpError as e:
        return {
            'statusCode': e.resp.status,
            'body': e.content,
            'headers': {
                'Content-Type': 'text/plain; charset=utf-8'
            }
        }


## Bootstrap for Direct Testing
if __name__ == "__main__":
    print(lambda_handler(None, None))

If I were making this maintainable, I'd probably split it up a bit more, move the title transformation into the rendering code, add some more error handling improvements, etc. The HTML could be templated, but I wanted to keep the dependencies low.

Done?

It seems to be working and stable. There are still things I could do to improve it, but it's working well enough that I might move on to the next experiment. It's configurable if I decide to run the same experiment for the next re:Invent or another conference. I can't run too many of these in parallel without requesting a YouTube Quota increase, but I could certainly run a few.

I wouldn't mind automating the deployment of it, but there aren't any really interesting parts to that for me, because I've done lots of AWS automation and deployments, so it would mostly be to save myself time, and unless I need to deploy it frequently, it's likely to end up costing me time instead. For an experiment, I'm not sure the automation is warranted.

I'm also curious to see what the total cost of running this will end up being -- I'm expecting it to be fairly cheap, but we'll see.

My AWS pre:Invent Highlights: Nov 2024

Geoffrey Wiseman — Mon, 02 Dec 2024 02:03:13 +0000

The weeks leading up to AWS re:Invent are filled with announcements that Amazon has decided not to hold for either a keynote or a major session. There are so many that even before re:Invent starts, it’s easy to miss many interesting things. Of course, I’m sure I’ve missed things, and the things I care about are not necessarily the things that you care about, but even so I’ll share a list of some of the things that I’d recommend you read.

Root Sessions

Managing AWS root accounts has often been a bit painful. They’re powerful, so you want to keep them locked down, but there are just some places where nothing else works, and you need access to them. This has often meant that security standards recommended Hardware MFA, but as your organization grows, even this gets a bit complicated to manage.

With Centrally managing root access for customers using AWS Organizations, it seems like organizations can get rid the liability of having security root accounts while retaining the ability for sufficiently authorized users to open a root session when needs require it.

This feels a bit like sudo, in a good way.

ECS Rebalancing and EC2 Zonal Shift

If you’re an AWS customer, you’ve likely experienced the fear that comes from hearing of any AWS outage — when AWS has an incident, it’s not uncommon for some or many of their customers to be having an incident as well. If that doesn’t happen to you frequently, it will likely take you some time to recover because you’re not experienced at diagnosing and recovering, perhaps you haven’t practice and prepared.

When that happens, anything AWS can do to simplify that process for you, allow certain things to happen automatically, the better. ECS AZ rebalancing and Zonal Shift for EC2 are both features that can make things a little easier for you when ad things are already happening.

EKS Auto Mode

ECS is often my starting point for a containerized service on AWS. Kubernetes is very much the industry standard, but if your team isn’t already up to speed on setting up, managing and using Kubernetes, ECS definitely feels like a lower barrier to entry, a simpler model.

Having said that, if you feel like you want to go with Kubernetes, anything AWS can do to simplify the job of making and running Kubernetes infrastructure sounds like a win. EKS auto mode sounds like it simplifies that experience somewhat, so it’s probably a good starting point for anyone making their first attempt at Kubernetes on AWS.

S3 Browsing, Integrity, Conditional Writes

If you’ve built integration with S3 into an application you manage, you’ve likely needed from time to time to access S3 directly, or to offer S3 access to a privilege group of users in your organization. When those users aren’t developers, asking them to use the AWS Console is a high barrier to entry. There are third-party applications, but setting up and maintaining those, credentials and so on is also real work.

By adding Transfer Family web apps (including Identity Centre / SSO) and a component that you can more easily integrate into your own application, AWS is offering ways that might simplify access to S3 when you need it.

And behind the scenes, integrity features using hashes and conditional writes with both seem very useful.

Aurora Serverless v2: Scaling to Zero

When Aurora Serverless v2 was announced, I know some people were disappointed that it didn’t seem to have the scale-to-zero capability, because one of the things that people value about the “serverless” model is the usage based billing — if you aren’t using it, it doesn’t cost you anything.

Sounds like that’s back as a feature, and your Serverless databases can now scale down to 0 ACUs.

And More …

Honourable mentions:

S3 Conditional Writes are valuable when there might be multiple processes acting in parallel
Allowed AMIs and lineage information for AMIs make for good governance guard-rails.
Amazon Q Java 17 and library upgrades are good to help with currency / modernization and go hand in hand with the preview of a Java upgrade transformation CLI.
EBS time-based copy seems suited for DR planning and audits.
Amazon Connect Email will probably appeal most directly to people already using Amazon Connect telephony, leading to more of an omni-channel solution.

For the full details, read What’s New with AWS.

AWS AllowList "Update"

Geoffrey Wiseman — Sat, 16 Nov 2024 16:36:22 +0000

I've just published v1.2.0 of AWS AllowList to pypi, an open-source python CLI tool for small AWS accounts to allow users to maintain a security group for SSH access into a VPC.

Before I get into the changes, let me remind you that while this can be a helpful tool, there are many alternatives that may be better suited for most accounts.

Alternatives

For an AWS environment of any significant size, there are alternatives that you ought to consider first. This is a simple solution for simple environments, but it's definitely not what I'd recommend as the best solution.

What's New

Update Commands

In v1.1.0, I added support to describe and automatically describe additions, but I found that as I went from place to place, it was easy to add new rules with awswl as I needed them, but that cleaning up past entries was a chore that wasn't directly needed to get work done, so easy to defer -- but also that each old entry was a potential security hole.

So I decided I ought to have another model in v1.2.0 -- a way to update existing rules (by description) so that the new entry would replace the old one, thus handling the cleanup as part of the process of adding a new entry.

To this end, I've added two update commands:

update
- Replaces an existing security group rule as identified by description with a new rule (same description, newly specified CIDR)
update-current
- Same as update, but instead of specifying the CIDR block, the CIDR block is automatically generated from your current external IP address.

Subcommands

In v1.1.0, I used CLI options to allow multiple commands to be run on one single invocation of the CLI. That was starting to get increasingly complex to understand and reason about, and I was finding it difficult to add new features without tripping over weird combinations of options.

So I've replaced the options with subcommands (e.g. git style). So what would have been awswl --add is now awswl add.

Dependencies / Security

There were some security vulnerabilities present on some of the dependencies, so all dependencies have been reviewed and/or updated where possible to ensure that awswl is modernized to cover any issues that have come up since the last release.

AWS Prefix Lists help simplify Networking

Geoffrey Wiseman — Wed, 03 Jul 2024 14:15:56 +0000

I’ve been working with a client that acquired another company, and has multiple office sites, data centres, and a fair number of private networks. As part of the acquisition, they’ve been working on integrating the systems of the parent and the acquired company, and one element of that has been simplifying the IP address management by removing overlaps between private networks, including AWS VPCs.

As a result, I’ve been working on setting up some new VPCs, and connecting them into the corporate network. While setting this up I discovered that customer-managed prefix lists on AWS can really simplify some use cases.

By way of example, imagine a networking situation like this:

With multiple customer networks and multiple VPCs, you can quickly end up needing to add a lot of CIDR blocks to a lot of places, particularly if you only want to target specific ranges, rather than a broad block (like 10.0.0.0/8). Using specific examples, you might need to:

add several CIDR ranges to each VPC’s route table to route the traffic to the transit gateway
add several CIDR ranges to one or more ports in one or more security groups in each VPC in order to allow access from customer networks

If the organization adds another site or another CIDR block to an existing site, you might have to go back and find all the places you added those CIDR blocks to, and add it again. This can quickly get tedious.

This is particularly important if you’re making these changes by hand in the AWS console, either because you don’t have infrastructure setup automated or perhaps because you’re making an exploratory change ahead before changing your infrastructure automation, but even if your infrastructure is fully automated with Terraform, CloudFormation, CDK or Pulumi, you might find that a prefix list makes a security group easier to read.

For instance, this security group that allows ICMP, HTTP and HTTPS from a prefix list:

And this one allows it from the CIDR ranges in the diagram above:

I know which of these I find easier to visually inspect.

So if you’re managing complicated networking on AWS and you haven’t taken a look at prefix lists, I hope I’ve convinced you that it’s time to take a look.

MySQL v5.7, AWS and Extended Support

Geoffrey Wiseman — Mon, 01 Jan 2024 02:42:04 +0000

If you are using MySQL v5.7.x still for a software project, you likely know that you’re already skating on thin ice. MySQL v5.7 was released in 2015, and Oracle has just moved the product into its ‘Sustaining Support’ category. However, if you’re using MySQL in AWS RDS, there are some additional factors to consider.

Starting 2024-Feb-29, AWS will automatically enroll any MySQL v5.7 databases (RDS or Aurora) into an Extended Support plan, which comes at an additional cost per vCPU-hour.

Automatic Enrolment

The original announcement for extended support positioned it as an opt-in support feature. Amazon says that they changed their mind due to customer feedback:

However, we heard lots of feedback from customers that these automatic upgrades may cause their applications to experience breaking changes and other unpredictable behavior between major versions of community DB engines. For example, an unplanned major version upgrade could introduce compatibility issues or downtime if applications are not ready for MySQL 8.0 or PostgreSQL 15.

Anyone who is aware of this impending change would be reasonably likely to understand that if they cannot upgrade, they ought to enrol into Extended Support, so my guess is that the customers who provided feedback weren’t the ones who were most likely to suffer. But there’s definitely a tradeoff to be made here. Assuming some percentage of customers are not going to get the message no matter how many emails and announcements you publish, do you want those customers to:

Have a new unexpected cost added to the bill?
Possibly have application downtime after their database gets an upgrade and their software can no longer connect?

Neither experience is great, but I suspect AWS did the math and decided that option #2 is possibly better for customers and potentially better for revenue as well. Some customers will disagree of course.

Avoiding Extended Support

There’s a way to avoid extended support. Upgrade. Move your existing v5.7.x MySQL Databases to MySQL v8.x. It sounds simple, and for some of you it might be fairly simple. Anyone currently running MySQL v5.7 on AWS RDS or Aurora ought to at least evaluate that option, and see if that’s feasible. You will, however, need to do a proof-of-concept and some testing to make sure that your application doesn’t have any side-effects of a major version upgrade.

If you’ve set up your RDS instances manually, this might be something you can test by doing an upgrade in the console. I wouldn’t recommend testing this in production, but I hope most of you have at least one development AWS environment you can use to test this, and if you don’t, this is a good time to consider setting that up.

If you’re using automation (e.g. GitHub Actions, Terraform, etc), you might need a little bit more time to implement and test the change, but fundamentally it’s the same sort of thing.

Why use Extended Support

The main reasons I can see for continuing on MySQL v5.7 past the end of February are these:

Upgrade Failed
- You’ve tried to upgrade to MySQL v8 and you’ve encountered a problem.
- You may be able to fix that before the end of February or you may not.
- If you haven’t resolved all your problems with the upgrade by the end of February you’ll probably know whether or not those problems are worse than paying for extended support.
Opportunity Cost
- Upgrading MySQL will take some time. Depending on your organization it might take time from:
  - your AWS or DevOps people (doing the upgrade, adjusting automation, etc)
  - testers (QA, business users)
  - production staff (during deployment, support)
- Those people might already be working on business-critical tasks that you don’t want to take them off. That time comes from somewhere, and you might put a higher value on that than on the Extended Support cost.
Risk
- This might be organization-specific; the work involved and the perceived risk of doing this before the end of February can vary a lot.
- If you’re a small team, with a small project and you aren’t currently under the gun, two months might seem like way more time than anyone needs to implement and test this change.
- On the other hand, in larger companies with more complexity and that move a little more slowly it might take most of those two months to come up with a good plan for what systems need to be upgraded, which teams will do the work to upgrade, test and roll it all out.

Cost

The cost varies by region, by the number of vCPUs (instances * vCPUs/instance) but the current costs are $0.10 - $0.21 per vCPU-hour for the first two years, and roughly double that for the third year. It will depend a lot on how many MySQL instances you’re already using, which instance types, and what region you’re running them in.

I’m working in Canada, so in ca-central-1, your additional cost for a given instance type would be $0.108/vCPU-hour. That would work out to $72.58 - $80.35 in additional monthly costs per vCPU. If you were running an admittedly massive db.m6i.32xlarge, that would be upward of $10k for a single instance in extended support costs for a 31-day month.

Even a tiny instance is non-trivial — a single db.t4g.micro might cost $0.018 per instance-hour in Canada, but it has two vCPUs, so would cost $0.216 per instance-hour for the extended support. Extended support in this case costs more than 10x the instance cost. About 7.5% of the cost of that instance after March 1st would be for the instance and the remaining 92.5% would be support costs. On larger instances, typically this ratio isn't quite so skewed. Looking at a few more examples, again using Canadian pricing:

Instance Class	Instance Cost (HA)	Extended Support (/hr)	% of Total Cost for Support
db.t4g.micro	$0.018	$0.216	92%
db.t4g.small	$0.035	$0.216	86%
db.t4g.large	$0.007	$0.216	97%
db.t4g.xlarge	$0.141	$0.432	75%
db.t4g.2xlarge	$0.563	$0.864	61%
db.m6g.large	$0.168	$0.216	56%
db.m6g.xlarge	$0.336	$0.432	56%
db.m6g.2xlarge	$0.672	$0.864	56%
db.m6g.4xlarge	$1.344	$1.728	56%
db.m6g.8xlarge	$2.688	$3.456	56%
db.m6g.12xlarge	$4.032	$5.184	56%
db.m6g.16xlarge	$5.376	$6.912	56%
db.r6g.large	$0.278	$0.216	44%
db.r6g.xlarge	$0.556	$0.432	44%
db.r6g.2xlarge	$1.113	$0.864	44%
db.r6g.4xlarge	$2.226	$1.728	44%
db.r6g.8xlarge	$4.451	$3.456	44%
db.r6g.12xlarge	$6.677	$5.184	44%
db.r6g.16xlarge	$8.903	$6.912	44%

It's worth noting that this isn't considering the cost of storage, backups, snapshots, and also isn't factoring in reserved instances or savings plans. I've also left out the high-availability options for now. There are so many variables in AWS cost projections that it's very difficult to be comprehensive.

I’ll do a case study below with specific costs for a particular scenario, to get a sense of what it might look like with multiple database instances of different classes.

It’s going to add up, so if you can upgrade, you probably need to start looking at your options to do that.

End of Extended Support

Extended Support doesn’t last forever. It will help bridge the gap if moving off of MySQL v5.7 is hard for you, but at the moment RDS Extended Support for MySQL v5.7 is expected to be available until 2027-Feb-28. If upgrading from (or switching away from) MySQL v5.7 is a challenge, you’re still going to want to put it in your roadmap for sometime in the next few years, otherwise you’ll be facing an even harder challenge in 2027.

What should I do?

First, do the math. How many MySQL v5.7 instances do you have? How many vCPUs do those have? What’s the additional cost per month that you’ll have to pay under Extended Support?

Next, set some time aside. Attempt an upgrade in a local environment or in a development environment. Is it easy? Does it seem to work? A quick proof-of-concept isn’t necessarily enough to move forward, but it should give you some sense of how much work you have ahead.

Then evaluate the tradeoff. Can you afford the time required to make the changes, test them and deploy them? How much time will that take in your organization? What else is your team working on and can you afford that time? What’s the opportunity cost of doing the upgrade when compared to the support cost of not doing the upgrade?

And if you can’t do the upgrade by the end of February, when can you do it? How much will that support cost add up to in three months, or six months?

Case Study

I have a client in this situation. They had planned to move off MySQL v5.7 in October, but those plans were derailed, and further testing on a proof-of-concept has revealed problems that don’t have an identified fix — it’s a weird problem, and it only seems to be exposed in the cloud environments, but it’s going to take some work to diagnose and fix.

In addition, this client has a lot of work already in flight on the project, and the opportunity cost of delaying that work may be larger than the cost per month they’ll currently have to pay in extended support.

So if they were running:

5 t3.micro instances
- $0.037 per hour each
- 2 vCPUs, so $0.216 per hour each
8 t2.micro instances
- $0.037 per hour each
- 1 vCPU, so $0.108 per hour each
1 m5.2xlarge instance
- $1.512 per hour
- 8 vCPU, so $0.864 per hour

If I’m doing the math correctly, I could project their bill for the next six months as:

Month	Cost
Jan	$1,483.79
Feb	$1,406.69
Mar	$2,366.66
Apr	$2,290.32
May	$2,366.66
Jun	$2,290.32

Effectively, starting March, the database cost will go up roughly 65% until they can find the time to resolve the issues and upgrade.

What about you?

Were you able to migrate in time? Are you already working towards it? Or have you decided that you're going to be using Extended Support already?

AWS ECS Features for Restart Loops

Geoffrey Wiseman — Wed, 05 Oct 2022 21:50:46 +0000

I had a client get an unexpectedly high AWS bill whose fundamental cause was an AWS ECS service stuck in a restart loop. Although I’ve encountered these before, I decided to dig a little deeper and write up a series of blog posts on the subject.

On the previous post, I described in more detail what an ECS Restart Loop is, and what it looks like, for people who haven’t encountered one, or haven’t spent much time looking into them.

Now I’m going to talk about AWS ECS features that can help deal with restart loops: service throttling and circuit breakers.

Deployment Circuit Breakers

Deployment circuit breakers were introduced to ECS in late 2020. These work reasonably well if your infinite restarts are caused by a broken deployment, and can even roll back your service to the previous working state.

This is what the ECS events look like with a failed deployment:

It's worth pointing out that the failure threshold (10 failures minimum) can take a while to reach, particularly on small services. This depends in part on how long it takes for a single health check to fail, but in extreme cases particularly for services with a very small number of tasks, it might take up to an hour for the circuit breaker to kick in.

Deployment circuit breakers solve the most common case and are easy to turn on, so this is a great starting point. If you're using ECS and you haven't looked into circuit breakers, I recommend you do so now.

There are limitations:

Not all infinite restart problems happen during deployment
It's still possible to create an ECS service with deployment circuit breakers disabled
The failure threshold is not configurable

Throttling

ECS Service Throttle Logic can help in the specific scenario that an ECS service does not reach the RUNNING state, so that when ECS tries to start a service and it cannot be started, it will throttle the rate at which it re-attempts to start the container.

The limitations here are important:

Only employed if a task goes from PENDING to STOPPED
More common if a task does not have the necessary resources, or the docker image can't be pulled

This can be helpful outside of a deployment context, but in most scenarios where I've encountered ECS service restart loops, it would not have helped.

Recommendations

Firstly, if you're not already using Circuit Breakers, you probably should be. Unless you've already evaluated circuit breakers and decided they aren't right for your environment, I'd recommend enabling these by default. If you have organizational defaults in a tool like Terraform modules, CDK or Pulumi, make enabling circuit breakers your default.

Secondly, the place where I've seen this the most is during deployment. If you're automating deployments, you might want to consider making sure that ECS considers the deployment successful and complete before your automation finishes. This might mean the automation needs to wait for the service deployment to be complete and the service to be healthy, which can take time, but it’s better than doing a deployment and assuming everything’s ok until you discover later than it’s not.

Is this enough?

So if ECS has features to deal with restart loops, is that enough? Is the problem solved? No.

Circuit breakers and deployment automation will help with failed deployments, but that's not the only reason an ECS service might become unhealthy.

They're also the something that you have to put in place, a precaution you have to deliberately take. In lots of organizations, you might have many different workloads built by different teams. If a team spins up their first ECS service and doesn’t turn on circuit breakers, will you catch that? Do you have the right policies in place to make sure that can’t happen? Or do you have monitoring to catch it when it does?

That’s the topic of the next post: Monitoring AWS for ECS Restart Loops.

AWS ECS Restart Loops

Geoffrey Wiseman — Thu, 11 Aug 2022 15:37:00 +0000

When you use Amazon Web Services (AWS) Elastic Container Service (ECS), you may encounter a situation where ECS tries to deploy one of your ECS services, but your tasks are unhealthy, so it tries again.

And again. And again. And again. And again.

ECS is very determined to start healthy tasks for you, and if you're not careful it may continue to try relentlessly, for days, weeks, months.

In the process, it may be creating log streams, allocating elastic network interfaces, downloading container images from the internet, incurring NAT gateway charges, making AWS config recordings, logging CloudTrail events and EventBridge events.

In an extreme case, this could trigger a pretty extreme spike in billing that might take you hours to track down.

I'm going to write up a few ways to avoid ECS restart loops and to detect them so that you can intervene in the next couple of posts, but in case this is unfamiliar territory, I'll start with some definitions in and a demonstration.

Definitions

Amazon Web Services (AWS) has a number of approaches to deploy and run software applications in a containerized model. Elastic Container Service (ECS) is one of the oldest and simplest AWS services to run software in a container, and it's usually what I'd recommend for someone getting into containers on AWS for the first time.

When you want ECS to run containers, you define a ECS Task: essentially a deployable unit with metadata about the container(s) to run, how they should be networked, what resources they need.

At that point you can ask ECS to create a task using that definition on an ECS/EC2 instance you control or on AWS Fargate, but typically you're more likely to wrap that ECS task in an ECS Service, which will make it easy to run a number of tasks in parallel, often in different availability zones, load balance them, restart them if they fail health checks, and so on.

The AWS ECS documentation has a Amazon ECS Components section that describes the elements of ECS if you're new to all of this.

One of the capabilities that ECS services add is resilience. What happens if the EC2 instance your task running on becomes unresponsive or is terminated? ECS can detect that and replace that task with another one running on another instance. What if your task has a memory leak and crashes? ECS can detect that your task is unhealthy and shut it down and start another. Many of the ways that ECS can intervene to keep your service healthy and robust revolve around shutting down and starting up the containerized tasks you've defined.

But ECS doesn't know the internals of your application. It doesn't know why your task is shutting down or failing health checks. So if it starts up a task and that task also fails, all it can do is keep trying. If it tries repeatedly and the tasks fail repeatedly, that's a restart loop.

Restart Loop

Why might a task fail and when? There are lots of scenarios:

The cluster might not have the necessary resources to start the task, causing the task to fail during startup, never reaching the RUNNING state.
The container might contain a crashing bug, causing the process to exit on startup.
The container might contain a crashing bug that requires time or a particular scenario to occur, causing the process to exit after being healthy and functional for some time.
The container might be healthy, but contain a faulty health check, causing ECS to believe the task is not healthy. This would allow the task to start, but ECS will terminate the task after running the initial health checks.
The container might be healthy and health check might be correctly defined, but the security group might prevent the health check from reaching the container, causing ECS to terminate the task after health checks.
The container's health check might depend on an external resource that isn't available. This could happen at any time, causing ECS to terminate the task. (Incidentally, this is a good reason to isolate health checks from the environment.)
Something about the environment (VPC, networking, security groups, etc.) might change in a way that causes the container or the health check to fail. This could happen anytime.

And what should ECS do when a task fails? If that task is part of a service, ECS is supposed to maintain a certain number of healthy tasks (as configured in the desiredTasks parameter). If a task terminates, it should be replaced by starting a new task. If a task fails health checks, it should be terminated and replaced.

But what if a task is replaced and the replacement also fails immediately? Should ECS continue to try? For how long? There's no one right answer here:

If there's a problem with the container or health check, the task will never succeed, and ECS is wasting cloud resources trying to "fix" a service that is fundamentally broken.
If there's a temporary problem with the environment or the dependency, restarting the task may not be helpful, but the service may return to health when the temporary problem ends.
If the task is flaky (occasionally crashes, memory leak, etc), restarting the task each time it fails is probably the best choice.

But should ECS detect that replacing tasks isn't working and change its behaviour? Perhaps. Until recently, ECS would just keep trying by terminating and replacing tasks indefinitely.

There was always a way to manually intervene -- even if you don't know what's causing the restart loop, you might decide that the restarts aren't helping. If you want it to stop, you can set desiredTasks to 0. This means that when a task fails, ECS wouldn't need to replace that task, ending the restart cycle.

Demonstration

What does it look like when an ECS service goes into a restart loop? If you look at an ECS service in a restart loop, you'll see a series of events to:

start / stop tasks
register / degregister targets from load balancer

If the failure happened during a deployment, you might also see a deployment in progress.

What's next?

So if you're having a problem with ECS services going into a restart loop or you're simply wondering how to protect yourself from it, what should you do?

In the second post, I'm going to go over some the new features added to ECS in the last few years that can help deal with service restarts, including:

service throttling
circuit breakers

And in the third post, I'm going to discuss some of the options for monitoring ECS to detect service restarts.

Tracking down a spike in NAT Gateway charges

Geoffrey Wiseman — Wed, 27 Jul 2022 15:11:00 +0000

If you spend enough time architecting and building solutions for the cloud, you're likely to have either experienced or at the very least, heard tell of an exciting cloud billing story (📈).

This is my tale of investigating an unexpected spike in NAT Gateway charges in a cloud migration project.

The Project

One of my clients has recently migrated a complex on-premise application to AWS, a lift-and-shift project that uses GitHub Actions, Terraform and Ansible to automate the creation of environment-specific VPCs containing several web applications, several microservices, and supporting infrastructure.

After the new environments were launched, the project team took some time to step back and look for problems. We found and documented points of friction in the automation, we tightened the security, and we looked at the month by month bills from AWS.

Much of the AWS costs were as expected, but there was one category that we didn't have an explanation for: NAT.

What's NAT?

NAT stands for "Network Address Translation". You typically need NAT when you're trying to communicate between a resource that does not have a public IP address and some outside resource. Because the internal machine does not have a public IP address, there's no way to communicate directly, so you can have another resource (a server, or in this case, an AWS-managed NAT Gateway) translate between an external address and port to an internal address and port.

Unexplained NAT Costs

The VPCs created by the project's automation have private subnets and those subnets use NAT Gateways to communicate with the outside world when necessary.

However, the NAT costs were high and rising:

The NAT traffic was surprisingly high -- we were approaching nearly 40TB of monthly NAT, way more than the team would have expected. Where was the traffic coming from, and where was it going to, and why?

There's a VPC per environment, and some of the environments share an account, so let's see if each VPC is seeing similar amounts of NAT traffic by using CloudWatch Metrics for the managed NAT gateways.

CloudWatch Metrics

Let's look at the metrics for NAT Gateways in CloudWatch. "NATGateway > NAT Gateway Metrics" has a bunch of metrics, but since we're looking at data volume, let's turn on all the ones that refer to "Bytes":

Ok, it's just one NAT Gateway that seems to be responsible for the majority of the traffic, both BytesInFromDestination and BytesOutToSource. Both of those suggest that the traffic is coming from an outside source -- that resources inside the VPC are downloading large amounts of data:

VPC Flow Logs / CloudWatch Insights

Now that we've narrowed it down to one VPC, how do we figure out where the traffic is coming from and going to? A good next step would be VPC Flow Logs, which log traffic in your VPC. VPC flow logs were already turned on, and there's a convenient knowledge-base article that gives tips on how to use CloudWatch Log Insights to analyze contributors to traffic through a NAT Gateway.

Looking at traffic going in and out of the NAT gateway, it looks like there's a lot of different internal IP Addresses involved:

So what's the source of this data?

A quick scan suggests that these are AWS addresses, likely S3. So a variety of internal IP addresses in a single VPC are downloading lots of data from AWS S3 instances. Why?

Tracking Down IP Addresses

At this point, I checked a couple of things first -- in particular, I was starting to be suspicious that there might be an unhealthy ECS service that wasn't being well-monitored, but I didn't see one on a very quick scan, so I decided to continue with the systematic search. What are these IP Addresses associated with?

So, now we look in EC2 > Network Interfaces (ENIs):

Nothing. I searched for several of the ip addresses listed in the flow logs, and none of them are currently active. Suspicious. Something is using an IP address, doing some NAT traffic and then ... going away before I'm searching for it.

Can we find the IP address on CloudTrail?

Yes we can:



❯ aws cloudtrail lookup-events --max-results 20 --lookup-attributes AttributeKey=EventName,AttributeValue=CreateNetworkInterface --end-time "2022-06-10" | jq '.Events[].CloudTrailEvent | fromjson' | jq .userAgent| sort | uniq -c
  20 "ecs.amazonaws.com"

Conveniently, the security group which appears in the event also matches the ECS service name. Sounds like my theory was correct, I just hadn't found the right service yet.

It's immediately clear from the service events that the service isn't healthy and has been restarting since some event in March without anyone noticing:



{
    "id": "715c6493-9a08-4de2-82d1-afcaf54c54ef",
    "createdAt": "2022-07-13T15:24:18.650000-04:00",
    "message": "(service qa-XXX-service) deregistered 1 targets in (target-group arn:aws:elasticloadbalancing:ca-central-1:004110273183:targetgroup/qa-XXX-service/a7f603824fb1e497)"
},
{
    "id": "0fea699a-3628-4f9d-abe0-dd89a930aa45",
    "createdAt": "2022-07-13T15:24:01.939000-04:00",
    "message": "(service qa-XXX-service) has begun draining connections on 1 tasks."
},
{
    "id": "68e666d1-f489-40a3-9dda-c5b739a7e6ad",
    "createdAt": "2022-07-13T15:24:01.926000-04:00",
    "message": "(service qa-XXX-service) deregistered 1 targets in (target-group arn:aws:elasticloadbalancing:ca-central-1:004110273183:targetgroup/qa-XXX-service/a7f603824fb1e497)"
},
{
    "id": "3cca6545-d38d-4b88-a8b9-1e936f737526",
    "createdAt": "2022-07-13T15:23:30.948000-04:00",
    "message": "(service qa-XXX-service) registered 1 targets in (target-group arn:aws:elasticloadbalancing:ca-central-1:004110273183:targetgroup/qa-XXX-service/a7f603824fb1e497)"
},
{
    "id": "36b50859-babe-47eb-8dcb-ca2c5c661491",
    "createdAt": "2022-07-13T15:23:21.103000-04:00",
    "message": "(service qa-XXX-service) has started 1 tasks: (task d1a70a3629d64febbc91695b4b848d71)."
},
{
    "id": "32f354a3-21ec-4a26-aac0-ceb02805dd16",
    "createdAt": "2022-07-13T15:23:16.401000-04:00",
    "message": "(service qa-XXX-service) has begun draining connections on 1 tasks."
},

Each time it restarts, it's pulling down a new container image from ECS, and incurring NAT bandwidth costs to do it.

What Now?

First, stop the bleeding:

Set the number of desired tasks to 0.
This will stop ECS from trying to create new tasks that will never be healthy.
This is a good way to stop ECS from trying to fix a broken service without having to significantly alter the task definition or service definition, allowing you to diagnose later.

At this point, we can monitor NAT Gateway metrics to make sure the usage has dropped, and AWS billing costs over the next few days to make sure the NAT Gateway costs have dropped.

Next?

We've solved the immediate problem, but it has identified lots of areas for further investigation:

Technical Debt
- This is an old service that has been replaced by the new environments but hasn't been cleaned up.
- Those cleanup tasks that had been deferred now have a very real cost.
Better Monitoring of ECS Services
- I've seen an ECS service go into this infinite unhealthy loop before.
- There are lots of ways to detect and diagnose failing ECS services, but most of them will require some setup on your part.
- These NAT costs make it clear that the project team will need to invest more time into detecting unhealthy ECS services.
NAT Gateway Monitoring
- One NAT gateway was doing significantly more traffic than the others
- Could add an alarm for when NAT gateway usage passes a reasonable limit.
- Could also consider CloudWatch anomaly detection.
Cost Monitoring
- A cost of this magnitude should probably not have gone unnoticed for months.
- While the other items still have value, this is just one way out of a seemingly infinite number of ways your AWS bill could be surprising. If we improve our monitoring of ECS services and NAT gateways, that won't prevent something else from going off the rails in a month or two.
- There are lots of options here:
  - Manually reviewing AWS bills when they come in
  - Billing threshold alarms
  - Cost anomaly detection
  - Third party solutions to manage and optimize AWS costs.
PrivateLink
- Without PrivateLink, traffic between ECS tasks running on Fargate and AWS services like ECS and S3 are travelling through the NAT gateway.
- By configuring PrivateLink for the VPC, we can route that traffic through AWS internal infrastructure, reducing NAT gateway traffic and costs.
Replace Managed NAT Gateways
- It is possible to set up your own NAT gateways on EC2 instances instead of using the AWS-managed ones.
- This would be more work to setup and maintain, and I'm not sure it's worth it, but there's an argument to be made.
- I will admit to having some sympathy for Corey Quinn's "Unpleasant and Not Recommended" tagline.

None of these are immediately critical, but if we don't address them, these and similar problems will recur and bring with them new and exciting AWS bill problems.

Intro to EC2 Global View

Geoffrey Wiseman — Wed, 06 Oct 2021 14:51:27 +0000

If you've spent more than a few days using the AWS console, you've probably had a moment where you've found yourself staring at an AWS console wondering where to find a resource, or why a resource you're looking for doesn't appear.

And if you have ever used resources in more than one region it's reasonably likely that some of those times were resolved when you suddenly noticed up in the corner that the region you were on isn't the one you expected, or possibly the region the resource you were looking for is in a region you didn't expect.

Even if most of your resources are in one region there are reasons why you might need to have some resources in another region. For instance, if you want to use an AWS Certificate Manager (ACM) certificate with CloudFront, it has to be in us-east-1.

When it really gets a little annoying is when you're not sure which region to look in -- now you're clicking from one region to the next, looking in each region for the resource.

Enter EC2 Global View

AWS announced EC2 Global View, a new interface for viewing your EC2 resources across all regions.

Think of EC2 Global View as an alternate interface for viewing some of your EC2 resources. It's not part of the EC2 console, and it's not a new entry in the region menu, it's a whole separate interface.

When you launch the interface, you'll see the Region Explorer:

I've numbered a few useful elements that we'll talk about.

The Region Explorer (1)

The region explorer will show you an overview of all the supported resources that you have in all regions, and then a breakdown of those resources by region. Most of the elements you click on will take you to the Global Search with specific filters applied.

This is essentially a dashboard with roll-up information and quick links to areas of interest. You can also use the search box in the region explorer to filter the regions that you're exploring.

Global Search (2)

The Global Search shows you specific resources. You can switch to this view yourself and enter your own searches by resource id, tags or region:

This is particularly useful if you have a good tagging strategy that might let you filter down to a very specific set of resources.

It's also the view you'll come to when clicking links on the dashboard, which will populate the search with filters that match what you clicked on.

Search by Resource Type (3)

If in the resource summary at the top of the region explorer, you click on any one of the resource types, you'll be shown a search view filtered by resources of that type.

Here's a search for security groups:

Search by Region (4)

If you just want to see all the supported resources in a region, simply click on the region in the first column of the region table, and you'll be taken to the Global Search with a region filter:

Alternately, go to the search view and start typing a region name.

Search by Region and Resource (5)

If you want to search by both region and resource, simply click the cell at the intersection of the row corresponding to the region and the column corresponding to the resource. For instance, if I wanted to see volumes in ca-central-1:

Limitations

It's still early days. It's called "EC2 Global View", not "AWS Global View", so it's still limited to Instances, VPCs, Subnets, Security Groups, Volumes. I can't find my CloudFront ACM Certificate using Global View.

It's also not super-discoverable. If you don't know that EC2 Global View exists as a separate console interface, then clicking in the region menu isn't going to give you any hints.

Fortunately, you can share this article with everyone you know who uses AWS and they can discover it that way.

Old AMIs on your AWS Account

Geoffrey Wiseman — Tue, 05 Oct 2021 14:26:18 +0000

If you're using Amazon Web Services (AWS), there's a pretty reasonable chance that you're also explicitly using an Amazon Machine Image (AMI) -- the format that AWS uses to make a virtual machine image. And if you've been using AWS for more than a few months, you might have some resources that were built with an old AMI. Maybe it was current when you created the instance or launch configuration, but has been aging since then.

If you're using an old AMI and you haven't been patching it, or if you have been patching it but the release is no longer supported, your resources might be vulnerable to attack. Patch management is crucial to keeping a long-running server secure. Of course, just because an AMI is old doesn't mean it hasn't been patched. Amazon Linux and Ubuntu both have an option for a rolling release model that can keep an old AMI relatively secure. If you've invested a lot into devops, you might also have a continuous deployment pipeline that rebuilds phoenix servers on a regular basis, which might include updating the AMI.

You might also be using AMIs, but not explicitly. You might be implicitly using AMIs through Fargate or Lambda, for instance. Getting out of the business of configuring and patching server instances is part of the value proposition of the 'serverless' model. There are still servers and patches, but you, the AWS user, don't need to care about them.

But if you're using AMIs and you're not sure that you've been keeping on top of the support timelines and the patch management, then it's probably also a good idea to consider how old those AMIs might be.

Finding Old AMIs

If you are using AMIs, how can you find out if the AMIs you're using are current? You can look at the AMI metadata:

CreationDate
- This tells you when the AMI was created.
- A creation date that isn't recent doesn't mean that image is unsupported or cannot be well-patched, but if you're still using an image created years ago, this might at least suggest that you need to take a closer look.
- If this image is still well-supported, you might still want to look at devops practices that allow you to rebuild old servers using more current images on a regular basis.
DeprecationTime
- AWS recently added a new piece of metadata that allows the image creator / maintainer to indicate the date/time on which the image is deprecated.
- This is more explicit than the creation date, but because it's new, there may well be many old images that are borderline abandoned and that don't have this date set.

So, if you gather a list of all the AMIs that you're using, you can look at these two fields to get hints about which images need a closer examination. An image created a year or more ago, or marked as deprecated are both worth examining to see if they're well-patched and well-supported.

A Simple Tool

If looking up all the AMIs in use on your account and checking the metadata sounds like a lot of work, then you're in luck. After reading about the new DeprecationTime data point, I made a small JavaScript tool, oldamis (github, npm) that looks up AMIs and checks both of those data points for you. You can run it with npx if you have NPM installed:

❯ npx @codiform/oldamis
   ___    _       _        _      __  __   ___
  / _ \  | |   __| |      / \    |  \/  | |_ _|  ___
 | | | | | |  / _` |     / _ \   | |\/| |  | |  / __|
 | |_| | | | | (_| |    / ___ \  | |  | |  | |  \__ \
  \___/  |_|  \__,_|   /_/   \_\ |_|  |_| |___| |___/

ami ami-730ebd17 is old (created 2016-08-22T19:58:21.000-04:00), sources:
  - instance i-13e13eeb963a78ab9
ami ami-0cde1f5ee149df291 is ok, sources:
  - instance i-a3c31bb5ebbd4790d
  - instance i-11aff774c13d785ef
  - instance i-486d7a5e0171e6749
ami ami-0f1c5116668d961c3 is ok, sources:
  - instance i-8f434ca2c2c36dfb5
  - instance i-4b344522536719e4f
  - launch config demo-launch-config-2340234

This isn't a sophisticated tool -- it's a proof of concept. Organizations that want to monitor this sort of thing should probably be looking at monitoring or policy-as-code tools that can be configured to look at this and many, many other things that could go wrong with your AWS account. That said, if you're not sure how old the AMIs you're using are right now, and this tool makes your life easier, I'm happy to hear it.

Just to be clear, oldamis doesn't record information about your account, intercept data, or use your credentials in sneaky ways. The tool respects your privacy and its open-source, so you're welcome to dig into the code before running it, just to make sure.

How it's made

I made oldamis with the AWS JavaScript SDK v3. I've done a bunch of AWS automation using Python and Boto3 and wanted to try a different language and SDK for a change, and refresh my knowledge of publishing an NPM module. There were a few hiccups, but all in all it worked well, and there's a good library for mocking API calls, which I've found to be somewhat important when you're writing a tool that is a thin layer over AWS API calls.

As an example, once oldamis has figured out which AMIs you might be using, it uses the DescribeImages api call to get the DeprecationTime and CreationDate:

const getAmiDates = async (amis) => {
    const command = new DescribeImagesCommand({
        "ImageIds": amis,
        "IncludeDeprecated": true
    });
    const response = await ec2Client.send(command);
    return response.Images.reduce((o, img) => {
        const {ImageId, DeprecationTime, CreationDate} = img;
        o[ImageId] = {DeprecationTime, CreationDate};
        return o;
    }, {})
}

After getting the response it transforms the results into a smaller data structure for consumption by the CLI that looks like this:

{
    "ami-730ebd17": {
        "CreationDate": "2016-08-22T19:58:21.000-04:00",
        "DeprecationTime": null
    }
}

There's more to it, but if you want to see more code samples, then I invite you to check out the GitHub repository. Contributions and feedback are welcome.

The oldami tool uses chalk to colorize the output and figlet for the banner.