DEV Community: Neil Clark

Event-Driven Architecture - Image Analysis

Neil Clark — Fri, 07 Feb 2025 14:53:54 +0000

Welcome Back!!

We are spoiled with the available services across AWS and most of them we can play with for free or for a limited cost if you are willing to invest some money. I have been guilty of shying away from trying to cover more complex subjects because i struggle where there is any coding involved.

However with Amazon Q Developer this has really helped me with this element where I struggle. Now don't get me wrong Q Developer is not a silver bullet to all your problems and during the creation of this blog I came very close a few times to finding something else to write about because it wouldn't give me working solutions from the off.

A lesson I have learnt creating this blog is to break down what you are trying to achieve into its simplest forms to start with and then iterate as you go to get more advanced elements. It is easier to iterate on top of a simple working example to get to where you want to be.

This blog is going to explore a simple event driven architecture to analyse images uploaded to an S3 bucket using Amazon Rekogniton. Upon an image being uploaded to S3, EventBridge will trigger a Lambda function that calls the Rekognition API to analyse the Image and provide Labels that are at least 95% confidence rating, and then add these to the S3 Object as meta data.

The final architecture will look like this

Now as I mentioned earlier, I was a bit naive when I started this blog, I thought to myself this should be easy I just need an S3 Bucket, an EventBridge Trigger and a Lambda Function (plus some IAM and Bucket polices). So I configured an S3 Bucket, an EventBridge trigger, then I asked Q Developer to write me a lambda function that took the input from an EventBridge Event (Bucket Name, Object Name), started Amazon Recognition and analysed the image and passed labels back to Lambda to apply to the S3 Object as meta data.

Wow watch Q Developer go.... spitting out Python code and instructions... I was thinking i'll be done in no time at all... LOL!!! yeah thats not how this went, I implemented the lambda code and instructions from Q Developer and it failed, I spent a lot of time troubleshooting and got to the point where the Lambda would run but nothing really happened it just ran and ended... Too much time later I scrapped it all and went and did something else to let my brain reset.

Start from a working point!

So after a break I decided to start again, but this time I looked for an AWS tutorial that was similar to what I was trying to achieve and I found this one

https://docs.aws.amazon.com/rekognition/latest/dg/lambda-s3-tutorial-python.html

This tutorial takes you through setting up and S3 Bucket, a Lambda function to call the Rekognition detect Labels function and a Client Python code to provide the S3 bucket and object name and invoke the Lambda function and receive back all the labels Rekognition detected.

After the initial implementation, it didnt run, and the only reason was due to me not updating the name of the lambda function in my client code and I hadnt created an aws cli profile to allow the python code to run and login to my AWS account.. Once I corrected this BAM!!! it worked..

I used this picture of my Dog Cooper

and the process returned this extract of labels

Analyzing image in S3 bucket:
{'body': '{"Labels": [{"Name": "Soil", "Confidence": 99.9998779296875, '
         '"Instances": [], "Parents": [], "Aliases": [], "Categories": '        
         '[{"Name": "Nature and Outdoors"}]}, {"Name": "Animal", "Confidence": '
         '99.24742889404297, "Instances": [], "Parents": [], "Aliases": [], '   
         '"Categories": [{"Name": "Animals and Pets"}]}, {"Name": "Canine", '   
         '"Confidence": 99.24742889404297, "Instances": [], "Parents": '        
         '[{"Name": "Animal"}, {"Name": "Mammal"}], "Aliases": [], '
         '"Categories": [{"Name": "Animals and Pets"}]}, {"Name": "Dog", '      
         '"Confidence": 99.24742889404297, "Instances": [{"BoundingBox": '      
         '{"Width": 0.23906654119491577, "Height": 0.12388644367456436, '       
         '"Left": 0.45508497953414917, "Top": 0.3805483877658844}, '
         '"Confidence": 99.24742889404297}], "Parents": [{"Name": "Animal"}, '
         '{"Name": "Canine"}, {"Name": "Mammal"}, {"Name": "Pet"}], "Aliases": '
         '[], "Categories": [{"Name": "Animals and Pets"}]}, {"Name": '
         '"Mammal", "Confidence": 99.24742889404297, "Instances": [], '
         '"Parents": [{"Name": "Animal"}], "Aliases": [], "Categories": '
         '[{"Name": "Animals and Pets"}]}, {"Name": "Pet", "Confidence": '
         '99.24742889404297, "Instances": [], "Parents": [{"Name": "Animal"}], '
         '"Aliases": [], "Categories": [{"Name": "Animals and Pets"}]}, '
         '{"Name": "Puppy", "Confidence": 99.22718048095703, "Instances": [], '
         '"Parents": [{"Name": "Animal"}, {"Name": "Canine"}, {"Name": "Dog"}, '
         '{"Name": "Mammal"}, {"Name": "Pet"}], "Aliases": [], "Categories": '
         '[{"Name": "Animals and Pets"}]}, {"Name": "Nature", "Confidence": '
         '98.79219055175781, "Instances": [], "Parents": [{"Name": '
         '"Outdoors"}], "Aliases": [], "Categories": [{"Name": "Nature and '
         'Outdoors"}]}, {"Name": "Outdoors", "Confidence": 98.79219055175781, '
         '"Instances": [], "Parents": [], "Aliases": [], "Categories": '
         '[{"Name": "Nature and Outdoors"}]}, {"Name": "Sky", "Confidence": '
         '98.79219055175781, "Instances": [], "Parents": [{"Name": "Nature"}, '
         '{"Name": "Outdoors"}], "Aliases": [], "Categories": [{"Name": '
         '"Nature and Outdoors"}]}, {"Name": "Rock", "Confidence": '
         '98.52386474609375, "Instances": [], "Parents": [], "Aliases": [], '
         '"Categories": [{"Name": "Nature and Outdoors"}]}, {"Name": "Field", '
         '"Confidence": 98.21697998046875, "Instances": [], "Parents": [], '
         '"Aliases": [], "Categories": [{"Name": "Nature and Outdoors"}]}, '
         '{"Name": "Grassland", "Confidence": 98.21697998046875, "Instances": '
         '[], "Parents": [{"Name": "Field"}, {"Name": "Nature"}, {"Name": '
         '"Outdoors"}], "Aliases": [], "Categories": [{"Name": "Nature and '
         'Outdoors"}]}, {"Name": "Land", "Confidence": 98.10195922851562, '
         '"Instances": [], "Parents": [{"Name": "Nature"}, {"Name": '
         '"Outdoors"}], "Aliases": [], "Categories": [{"Name": "Nature and '
         'Outdoors"}]}, {"Name": "Person", "Confidence": 97.86608123779297, '
         '"Instances": [{"BoundingBox": {"Width": 0.02859390340745449, '
         '"Height": 0.04032939299941063, "Left": 0.22646482288837433, "Top": '
         '0.15954364836215973}, "Confidence": 97.86608123779297}], "Parents": '
         '[], "Aliases": [{"Name": "Human"}], "Categories": [{"Name": "Person '
         'Description"}]}, {"Name": "Plant", "Confidence": 95.07865142822266, '
         '"Instances": [], "Parents": [], "Aliases": [], "Categories": '
         '[{"Name": "Plants and Flowers"}]}, {"Name": "Vegetation", '
         '"Confidence": 95.07865142822266, "Instances": [], "Parents": '
         '[{"Name": "Plant"}], "Aliases": [], "Categories": [{"Name": "Nature '
         'and Outdoors"}]}, {"Name": "Grass", "Confidence": 94.17696380615234, '
         '"Instances": [], "Parents": [{"Name": "Plant"}], "Aliases": [], '
         '"Categories": [{"Name": "Plants and Flowers"}]}, {"Name": "Hound", '
         '"Confidence": 92.6964111328125, "Instances": [], "Parents": '
         '[{"Name": "Animal"}, {"Name": "Canine"}, {"Name": "Dog"}, {"Name": '
         '"Mammal"}, {"Name": "Pet"}], "Aliases": [], "Categories": [{"Name": '
         '"Animals and Pets"}]}, {"Name": "Ball", "Confidence": '
         '83.73759460449219, "Instances": [], "Parents": [], "Aliases": [], '
         '"Categories": [{"Name": "Sports"}]}, {"Name": "Sport", "Confidence": '
         '83.73759460449219, "Instances": [], "Parents": [], "Aliases": '
         '[{"Name": "Sports"}], "Categories": [{"Name": "Sports"}]}, {"Name": '
         '"Tennis", "Confidence": 83.73759460449219, "Instances": [], '
         '"Parents": [{"Name": "Sport"}], "Aliases": [], "Categories": '
         '[{"Name": "Sports"}]}, {"Name": "Tennis Ball", "Confidence": '
         '83.73759460449219, "Instances": [], "Parents": [{"Name": "Ball"}, '
         '{"Name": "Sport"}, {"Name": "Tennis"}], "Aliases": [], "Categories": '
         '[{"Name": "Sports"}]}, {"Name": "Road", "Confidence": '
         '79.90119934082031, "Instances": [], "Parents": [], "Aliases": [], '
         '"Categories": [{"Name": "Transport and Logistics"}]}, {"Name": '
         '"Tree", "Confidence": 79.13050079345703, "Instances": [], "Parents": '
         '[{"Name": "Plant"}], "Aliases": [], "Categories": [{"Name": "Nature '
         'and Outdoors"}]}, {"Name": "Wilderness", "Confidence": '
         '61.0156364440918, "Instances": [], "Parents": [{"Name": "Nature"}, '
         '{"Name": "Outdoors"}], "Aliases": [], "Categories": [{"Name": '
         '"Nature and Outdoors"}]}, {"Name": "Landscape", "Confidence": '
         '57.71144104003906, "Instances": [], "Parents": [{"Name": "Nature"}, '
         '{"Name": "Outdoors"}], "Aliases": [], "Categories": [{"Name": '
         '"Nature and Outdoors"}]}, {"Name": "Ground", "Confidence": '
         '57.36166763305664, "Instances": [], "Parents": [], "Aliases": [], '
         '"Categories": [{"Name": "Nature and Outdoors"}]}, {"Name": "Ice", '
         '"Confidence": 56.176082611083984, "Instances": [], "Parents": [], '
         '"Aliases": [], "Categories": [{"Name": "Nature and Outdoors"}]}, '
         '{"Name": "Cloud", "Confidence": 55.94695281982422, "Instances": [], '
         '"Parents": [{"Name": "Nature"}, {"Name": "Outdoors"}, {"Name": '
         '"Sky"}], "Aliases": [], "Categories": [{"Name": "Nature and '
         'Outdoors"}]}, {"Name": "Gravel", "Confidence": 55.84575653076172, '
         '"Instances": [], "Parents": [{"Name": "Road"}], "Aliases": [{"Name": '
         '"Dirt Road"}], "Categories": [{"Name": "Materials"}]}, {"Name": '
         '"Water", "Confidence": 55.7204475402832, "Instances": [], "Parents": '
         '[], "Aliases": [], "Categories": [{"Name": "Nature and Outdoors"}]}, '
         '{"Name": "Pebble", "Confidence": 55.63995361328125, "Instances": [], '
         '"Parents": [], "Aliases": [], "Categories": [{"Name": "Nature and '
         'Outdoors"}]}, {"Name": "Face", "Confidence": 55.62419128417969, '
         '"Instances": [], "Parents": [{"Name": "Head"}, {"Name": "Person"}], '
         '"Aliases": [], "Categories": [{"Name": "Person Description"}]}, '
         '{"Name": "Head", "Confidence": 55.62419128417969, "Instances": [], '
         '"Parents": [{"Name": "Person"}], "Aliases": [], "Categories": '
         '[{"Name": "Person Description"}]}, {"Name": "Photography", '
         '"Confidence": 55.62419128417969, "Instances": [], "Parents": [], '
         '"Aliases": [{"Name": "Photo"}], "Categories": [{"Name": "Hobbies and '
         'Interests"}]}, {"Name": "Portrait", "Confidence": 55.62419128417969, '
         '"Instances": [], "Parents": [{"Name": "Face"}, {"Name": "Head"}, '
         '{"Name": "Person"}, {"Name": "Photography"}], "Aliases": [], '
         '"Categories": [{"Name": "Hobbies and Interests"}]}, {"Name": "Snow", '
         '"Confidence": 55.35792541503906, "Instances": [], "Parents": '
         '[{"Name": "Nature"}, {"Name": "Outdoors"}], "Aliases": [], '
         '"Categories": [{"Name": "Nature and Outdoors"}]}, {"Name": '
         '"Weather", "Confidence": 55.241214752197266, "Instances": [], '
         '"Parents": [{"Name": "Nature"}, {"Name": "Outdoors"}], "Aliases": '
         '[], "Categories": [{"Name": "Nature and Outdoors"}]}, {"Name": '
         '"Grove", "Confidence": 55.16259002685547, "Instances": [], '
         '"Parents": [{"Name": "Land"}, {"Name": "Nature"}, {"Name": '
         '"Outdoors"}, {"Name": "Plant"}, {"Name": "Tree"}, {"Name": '
         '"Vegetation"}, {"Name": "Woodland"}], "Aliases": [], "Categories": '
         '[{"Name": "Plants and Flowers"}]}, {"Name": "Woodland", '
         '"Confidence": 55.16259002685547, "Instances": [], "Parents": '
         '[{"Name": "Land"}, {"Name": "Nature"}, {"Name": "Outdoors"}, '
         '{"Name": "Plant"}, {"Name": "Tree"}, {"Name": "Vegetation"}], '
         '"Aliases": [{"Name": "Forest"}], "Categories": [{"Name": "Nature and '
         'Outdoors"}]}], "LabelModelVersion": "3.0", "ResponseMetadata": '
         '{"RequestId": "0e2ee3f5-113d-489f-b922-f8568c023dc8", '
         '"HTTPStatusCode": 200, "HTTPHeaders": {"x-amzn-requestid": '
         '"0e2ee3f5-113d-489f-b922-f8568c023dc8", "content-type": '
         '"application/x-amz-json-1.1", "content-length": "7061", "date": '
         '"Thu, 06 Feb 2025 17:14:20 GMT"}, "RetryAttempts": 0}}',
         '"Confidence": 55.16259002685547, "Instances": [], "Parents": '
         '[{"Name": "Land"}, {"Name": "Nature"}, {"Name": "Outdoors"}, '
         '{"Name": "Plant"}, {"Name": "Tree"}, {"Name": "Vegetation"}], '
         '"Aliases": [{"Name": "Forest"}], "Categories": [{"Name": "Nature and '
         'Outdoors"}]}], "LabelModelVersion": "3.0", "ResponseMetadata": '
         '{"RequestId": "0e2ee3f5-113d-489f-b922-f8568c023dc8", '
         '"Confidence": 55.16259002685547, "Instances": [], "Parents": '
         '"Confidence": 55.16259002685547, "Instances": [], "Parents": '
         '[{"Name": "Land"}, {"Name": "Nature"}, {"Name": "Outdoors"}, '
         '{"Name": "Plant"}, {"Name": "Tree"}, {"Name": "Vegetation"}], '
         '"Aliases": [{"Name": "Forest"}], "Categories": [{"Name": "Nature and '
         'Outdoors"}]}], "LabelModelVersion": "3.0", "ResponseMetadata": '
         '{"RequestId": "0e2ee3f5-113d-489f-b922-f8568c023dc8", '
         '"HTTPStatusCode": 200, "HTTPHeaders": {"x-amzn-requestid": '
         '"0e2ee3f5-113d-489f-b922-f8568c023dc8", "content-type": '
         '"application/x-amz-json-1.1", "content-length": "7061", "date": '
         '"Thu, 06 Feb 2025 17:14:20 GMT"}, "RetryAttempts": 0}}',
 'statusCode': 200}
         '"Confidence": 55.16259002685547, "Instances": [], "Parents": '
         '[{"Name": "Land"}, {"Name": "Nature"}, {"Name": "Outdoors"}, '
         '{"Name": "Plant"}, {"Name": "Tree"}, {"Name": "Vegetation"}], '
         '"Aliases": [{"Name": "Forest"}], "Categories": [{"Name": "Nature and '
         'Outdoors"}]}], "LabelModelVersion": "3.0", "ResponseMetadata": '
         '{"RequestId": "0e2ee3f5-113d-489f-b922-f8568c023dc8", '
         '"HTTPStatusCode": 200, "HTTPHeaders": {"x-amzn-requestid": '
         '"0e2ee3f5-113d-489f-b922-f8568c023dc8", "content-type": '
         '"Confidence": 55.16259002685547, "Instances": [], "Parents": '
         '[{"Name": "Land"}, {"Name": "Nature"}, {"Name": "Outdoors"}, '
         '{"Name": "Plant"}, {"Name": "Tree"}, {"Name": "Vegetation"}], '
         '"Aliases": [{"Name": "Forest"}], "Categories": [{"Name": "Nature and '
         'Outdoors"}]}], "LabelModelVersion": "3.0", "ResponseMetadata": '
         '{"RequestId": "0e2ee3f5-113d-489f-b922-f8568c023dc8", '
         '"Confidence": 55.16259002685547, "Instances": [], "Parents": '
         '[{"Name": "Land"}, {"Name": "Nature"}, {"Name": "Outdoors"}, '
         '{"Name": "Plant"}, {"Name": "Tree"}, {"Name": "Vegetation"}], '
         '"Confidence": 55.16259002685547, "Instances": [], "Parents": '
         '"Confidence": 55.16259002685547, "Instances": [], "Parents": '
         '[{"Name": "Land"}, {"Name": "Nature"}, {"Name": "Outdoors"}, '
         '{"Name": "Plant"}, {"Name": "Tree"}, {"Name": "Vegetation"}], '
         '{"Name": "Plant"}, {"Name": "Tree"}, {"Name": "Vegetation"}], '
         '"Aliases": [{"Name": "Forest"}], "Categories": [{"Name": "Nature and '
         'Outdoors"}]}], "LabelModelVersion": "3.0", "ResponseMetadata": '
         '{"RequestId": "0e2ee3f5-113d-489f-b922-f8568c023dc8", '
         '"HTTPStatusCode": 200, "HTTPHeaders": {"x-amzn-requestid": '
         '"0e2ee3f5-113d-489f-b922-f8568c023dc8", "content-type": '
         '"application/x-amz-json-1.1", "content-length": "7061", "date": '
         '"Thu, 06 Feb 2025 17:14:20 GMT"}, "RetryAttempts": 0}}',
 'statusCode': 200}

Great we have now got something working!!

Let's Iterate...

So with the Lambda function working, I copied it back into VScode and asked Q Developer if it could modify the function so that it could take the Bucket and Object name from an EventBridge Event... so off it went and spit out modified function code.

So next I added the EventBridge Rule to track when object creation occurred in my S3 bucket and to trigger the Lambda function.

So with this in place I uploaded a new image to my S3 bucket this time i used a different image, this was of a lovely Mercedes AMG from when I recently attend a training event at Mercedes Benz World.

EventBridge triggered Lambda but it threw some errors... Oh no here we go again..

I copied the errors from CloudWatch and told Q Developer that these errors were occurring with the Lambda code, it immediately came back with

The error occurs because the event structure you're receiving contains nested dictionaries for the bucket and object information, but Rekognition expects simple strings.

It proceeded to spit out a new modified version of the Lambda function to fix these errors, so I dropped the new code into the Lambda function and deployed it, deleted the image object I had previously uploaded and uploaded it again.

IT WORKED!!!

As you can see from the above image, Rekognition Labels have been returned... Woooo we are part way there.

Let's Iterate again...

So as you saw earlier Rekognition returns a lot of labels with varying degrees of confidence. With the ultimate end goal to add these labels to the S3 Object as metadata we probably don't want all of the labels, so I asked Q developer again, could the Lambda function be modified to only return labels that had a confidence rating above 95, so off it goes and within 30 secs it has created modified function code.

So I then updated the Lambda function and deployed it, and ran through the process of removing the image object and uploading again, this time it ran perfectly with no issues (Q Developer is getting better as we go) and as you can see from the below it is returning labels as requested.

Now we are cooking...

Let's Iterate one more time...

So now we have this sorted we are going to change the code one more time, so back to Q Developer, so this time, I asked Q Developer to modify the Lambda function to take the returned labels from Rekognition and update the S3 Objects metadata with the labels that have been found. So like a good assistant Q Developer spits out updated function code with the new requirements in. So I update the Lambda function once more and also modify the Lambda execution role to allow Put functions, once this is complete I then went through the process of removing the image object and uploading again and again everything ran perfectly with no issues and the object metadata was updated as expected.

So there you have it

So we got there in the end with the heavy lifting being done by Q Developer, but I hope this shows you two things, 1) You don't have to be great at everything to be able to have a go at using some of the other more advanced services especially with Q Developer getting better and better overtime (when I first used Q Developer is wasn't great..) and 2) Breaking down solutions in to manageable bites and getting things working and linking them together as you go is a great way to solve issues and break down the barriers that might stop you from giving it a go.

I hope this inspires you to go and have a play with AWS and services you may not touch normally.

I have dropped the working Lambda Python codes into a Github repo for you to grab and try if you wish, please just keep in mind you may need to update names in the code if you have called functions different names etc, but use them, try them, use Q Developer to help you if they dont work...

Until next time enjoy playing with AWS!

Neil works for Telefonica Tech UK helping guide their customers who are considering or using AWS for their production work loads, He also works across Telefonica Tech UK's business units helping develop offerings and solutions.

What does securing your AWS Environment look like in 2025?

Neil Clark — Fri, 17 Jan 2025 21:50:59 +0000

When you look across todays digital landscape it is more critical than ever to ensure that our Cloud environments are secured against bad actors both externally and internally

Wait you say... Internal bad actors, surely this is not a thing?

Well most companies in the present day spend time and money vetting their staff (especially in the UK, I’m not sure on the rest of the world), via supplied references, BPSS checks, potentially even as far as UK Government Security vetting if the systems they are working on require it. This is about as much due diligence you can do, however we have all made questionable decisions in our lives and unfortunately, when a person is under duress, disgruntled etc there is no telling what they may do, so we need to ensure that we protect internally as much as we can.

The Basics

1 - Develop a comprehensive security plan

Before diving into specific practices it is essential to have a well-defined security plan. Your plan should outline things like your security goals, identified potential risks and establish protocols for incident response. Once you have this in place you can dive into specific security policies and configurations standards for AWS Infrastructure, AWS Services and your applications.

2 - Identity & Access Management

Identity & Access Management (IAM) is the backbone of AWS Security so implementing strong IAM policies and configurations will mean you are 90% there.

So when we talk about IAM what areas do we want to look at?

Secure your Root accounts - Enable MFA, remove programmatic access, use strong passwords and be ultra strict with who has access and what access they have.
Principle of least privilege - This should be front and center of every permissions policy, that we only grant users or apps/services the permissions they need to perform their tasks.
MFA..MFA...MFA - Ensure through policies that all users need to have MFA enabled to be able to do anything in the account.
IAM Roles and Policies - IAM Roles should be used for applications and services so no hard coded credentials are used, combined with this, IAM policies with permissions defined using the principle of least privilege. We can also use permissions boundary policies to set the limits of what levels of privilege cane be added to IAM roles (especially useful for developer accounts). Reviewing Roles and Policies regularly is recommended to ensure Roles are still needed and policies are still protecting as needed.
Regular Audits - This is often over looked, I know I have mentioned it previously but it is hugely important that you regularly review configured users and IAM roles and assigned IAM policies to ensure that they are still in use and that policies grant the needed permissions only. IAM policies need to be regularly reviewed to ensure that they are assigned to users or roles and if they aren’t carry out some housekeeping and remove them. Maybe this is something you can automate using the AWS CLI and Python or maybe Lambda, maybe someone has already done this... if you have comment a link to help others out.

3 - Data Encryption at Rest and In Transit

Data at Rest - There are very few AWS Storage services that are not encrypted by default these days, up until very recently... I think (my memory is not as good as it used to be) S3 and EBS were the only services that weren't encrypted by default, however AWS updated S3 in 2023 to ensure all newly created buckets were encrypted by default, which leaves EBS as the only service that is not encrypted by default and requires the user to enable it. So AWS have got you covered there, for all your Data at Rest, you just need to remember to turn it on... and you can always use AWS Systems Manager and AWS Config to check services they need manual enablement of encryption and alert you and take action if needed.
Encryption in Transit - Making sure all your communications between Applications and AWS Services seems like common sense, however it can be tempting especially when your AWS elements are all in the same VPC and subnets to neglect encryption in transit, however within AWS although things may appear they are all together it is very unlikely and your traffic will be using an internal network to talk, so making sure traffic is encrypted is essentially.
AWS Key Management Service - Utilising AWS Key Management Service (KMS) can be key to a successful implementation of encrypting data at rest, it allows for you to create and manage keys via AWS, you can bring your own cryptographic material, or if you need something really special you can even setup a Cloud HSM if you have specific requirements.

4 - Monitoring and responding to threats

Visibility is key - If you are blind to events that are going on in your environment then you cannot act. At the very least you need to have AWS Cloud Trail enabled and CloudWatch setup, CloudTrail ensures you have visibility of actions going on within your AWS Accounts when AWS API's are being called. CloudWatch gives the ability to see what is going on with your workloads and can ingest application logs if the CloudWatch Logs Agent is deployed.
Detection of Threats - So we have observability but there is a huge amount of information go through, in fact I would wage that it is more information than a human could process effectively. So what do you do, well this is where we would use Guard Duty and AWS Security Hub. With Guard Duty we can utilise powerful machine learning for intelligent threat detection for your AWS environment. Guard Duty can be combined with AWS Security Hub to provide a centralised portal for security alerts and automated compliance checks.

5 - Let's Automate

Configuration Tracking - so once you have got you environment configured as it should be to best practices etc, how do you know if something changes? Well we have CloudTrail but this only tells you what commands have been executed.. This is where AWS Config comes in, AWS Config tracks your configurations and has several functions that you can utilise, the first is configuration tracking and alerting, so if a configuration changes you can get alerted to the changes so you can act, however the second function you can pair it with is AWS Systems Manager (SSM) Automation, so that when a particular configuration is modified SSM can be triggered and take action to correct configurations.
Lambda is not just for serverless apps - Lambda is very versatile and can be used in an event driven way to automate responses to security threats. A great use case for Lambda is isolating compromised instances or revoking access keys.
SSM can do the hard work - keeping workloads up to date with the latest security patches can be tiresome however SSM Patch manager can lighten the load and ensure that your workloads are protected from the latest vulnerabilities, and with automation these can be scheduled at a time that is best for your business.

By implementing the above you can significantly enhance the security of your AWS environment. Remember security is an ongoing process the requires continuous monitoring and improvement.

If you have any other gems you think others would benefit from drop them in the comments.

In my next blog we'll have a look at some environment enhancements that go beyond the basics to provide further security enhancements.

Neil works for Telefonica Tech UK helping guide their customers who are considering or using AWS for their production work loads, He also works across Telefonica Tech UK's business units helping develop offerings and solutions.

Building Scalable Applications in AWS

Neil Clark — Tue, 17 Dec 2024 19:54:37 +0000

Overview of Scalability in AWS

In simple terms scalability refers to a system’s ability to handle increased loads by adding resources, this is crucial as is ensures that applications can maintain performance levels despite growing user demand. So what does this mean in terms of AWS services?

Scalability is broken into two variants:-

Vertical Scaling
Horizontal Scaling

Vertical Scaling - This is where we are adding more horsepower to an existing instance, More CPU, More RAM, this is achieved by changing the instance size or type.

Horizontal Scaling - This is where we are adding more of the same type of machine for world domination.. I mean increased capacity.

AWS Core Services for Scalability

Amazon EC2

Probably AWS's most used service EC2 provides on-demand compute capacity for servers deployed in the cloud. EC2 instances come in a huge array of types and sizes to fill just about any need, from burstable T3 instances to the recently announced u7inh Family with 1920 vCPUs and 32TB of RAM truly huge computing capacity in current times.

So the instance families and the varying degrees of sizes within those families covers the vertical scaling element, and we can horizontally scale those by adding additional instances...

Well hold on are you saying add them manually??

Yes I am...

Ok... well is there a way to do this automatically

Well of course there is... this is called EC2 Auto Scaling

Auto Scaling allows you to create auto scaling groups and as part of these groups you set parameters that control your minimum and maximum number of instances with a desired number for normal operations. From this you can then set scaling events to determine when your scaling group grows or shrinks. For example I have a group that has a minimum number of instances set for 2 and a maximum of 8, my desired number is 3. My scaling event is set to monitor EC2 instance CPU utilisation and when it breaches 60% a new instance is added, and when the utilisation resolves itself instances numbers gradually reduce back to the desired number.

Scaling Types are broken down into the following

Dynamic scaling: Adapts capacity to actual loads to optimize resource utilization
Predictive scaling: Uses workload forecasting to plan future capacity
Scheduled scaling: Allows you to scale based on a schedule
Fixed number of instances: Allows you to maintain a fixed number of instances
Proactive scaling: Allows you to scale proactively

Elastic Load balancing

Now here is a service that goes hand in hand with EC2 and auto scaling now while it is great to have the ability to automatically scale the number of instances needed to meet demand generally you would have to manually add them into you application or run scripts to get the most from them. Elastic load balancers and auto scaling groups overcomes this issue. When combining ELB's and auto scaling the ELB acts as the front end to your servers and when autoscaling is used this allows for new servers that are started to be registered with the ELB and they automatically become part of your application landscape until they are no longer required and scaling back starts where they are removed from the ELB and shutdown.

Amazon RDS

Amazon RDS this is Amazon's Managed Database Service supporting a wide variety of popular database engines. Scaling RDS is really quite simple, in a traditional setup your database node would handle both write and read operations from your applications, in an application where you have heavy read and write operations, relying on a single node is not always the best option however it might be the only option due to the application. In this instance RDS allows you to scale your RDS instance vertically both for the instance size and storage options, this means that there would be disruption during the scaling event which is not ideal and would mean planning this in during the least impactful time which for some apps could be never...

The other option that RDS offers is read replicas, this allows you to scale out the read operations for the database to multiple read replicas to remove the load from the primary database and freeing up capacity and adding durability to your application. For situations where you need to keep scaling up as an option RDS should be deployed in a Multiple AZ setup, when RDS is setup this way, when you need to scale your primary nodes up, the standby node is scaled up first with a failover then taking place from the primary and then the primary being scaled up also, this would provide minimum downtime for your database.

AWS Serverless Services

AWS Lambda

AWS Lambda is a serverless compute service from AWS, it allows you to run code without provisioning any compute resources, it is triggered by various supported AWS services and scales automatically based on the number of incoming requests from services. A service that scales by default with no intervention required by the infrastructure team (unless you hit the concurrency limits)

There are some best practices to help with scaling though..

These are :-

Optimize Function Code: Ensure your function code is efficient and minimizes execution time. This helps reduce the overall load and cost.
Use Asynchronous Invocations: For non-blocking tasks, use asynchronous invocations to allow Lambda to handle more requests concurrently.

Amazon API Gateway

API Gateway enables you to create, publish, maintain and monitor API's at any scale, this again is another Serverless managed service that requires no management of infrastructure.

As with AWS Lambda, API Gateway scales automatically with demand and again there are some recommendations to help it scale effectively..

API Gateway Caching: Enable caching to store responses and reduce the load on your backend services. This can significantly improve performance and reduce latency for frequently accessed data
Request Throttling: Set up throttling to limit the number of requests per second for each API key. This helps protect your backend services from being overwhelmed by too many requests1.
Usage Plans: Create usage plans to enforce throttling and quota limits on individual API keys, ensuring fair usage and preventing abuse

AWS Fargate

AWS Fargate is a serverless compute engine for containers that works with Amazon ECS and EKS. It allows you to run containers without managing the underlying infrastructure. This means you can focus on building and running applications without worrying about the servers.

Fargate scales automatically based on the resource requirements of your containers. When you define your task, you specify the CPU and memory requirements, and Fargate takes care of provisioning the right amount of resources. This ensures that your applications can handle varying loads without manual intervention.

Fargate can auto scale using the below policies

Auto Scaling Policies: Use Amazon ECS Service Auto Scaling to automatically adjust the number of running tasks based on demand.
Target Tracking Policies: Adjust the task count to maintain a specified metric (e.g., CPU utilization).
Step Scaling Policies: Add or remove tasks based on specific thresholds.

Amazon DynamoDB

Amazon DynamoDB is a fully managed NoSQL database service that provides fast and predictable performance with seamless scalability. It is designed to handle high-traffic applications and can scale horizontally to accommodate growing workloads.

DynamoDB automatically adjusts throughput capacity based on traffic patterns, ensuring that your application can handle sudden spikes in demand. It also offers features like Global Tables for multi-region replication and on-demand capacity mode for flexible scaling.

DynamoDB can scale using the below policies

Auto Scaling Policies: Use DynamoDB auto scaling to automatically adjust the provisioned throughput capacity based on actual traffic patterns. This ensures that your tables and global secondary indexes can handle sudden increases in traffic without throttling.
Target Utilization: Set a target utilization percentage for your table's read and write capacity. Auto scaling will adjust the provisioned throughput to maintain this target, ensuring efficient use of resources.

Amazon S3

Amazon Simple Storage Service (S3) provides scalable object storage for a wide range of use cases. S3 is designed to store and retrieve any amount of data from anywhere on the web, making it ideal for backup, archiving, big data analytics, and more.

S3 is a great service that can scale infinitely... ok well not infinitely it depends on AWS storage hardware having the capacity but let’s just say i don’t think AWS will be running out when people need it.

This is just a whistle stop look at some AWS services and how they scale to allow your applications to scale for your users, there are many more AWS services you can use to integrate with your applications and the theme is the same there too.

Neil works for Telefonica Tech UK helping guide their customers who are considering or using AWS for their production work loads, He also works across Telefonica Tech UK's business units helping develop offerings and solutions.

AWS Cloud Network - How does an environment evolve over time

Neil Clark — Tue, 12 Nov 2024 18:12:29 +0000

Expectations vs Reality

Picture the scene: you're a budding Cloud Architect. You've completed a few AWS certifications and recently passed your Advanced Networking Specialty. At this point, you know AWS Cloud Networking and the best practices that come with it. You're then assigned to assist a customer with their AWS environment. In the excitement leading up to starting work, you envision large-scale Transit Gateway networks, funky BGP routing, multiple Direct Connect connections with site-to-site VPNs as a backup.

The reality, when you get onboard, couldn't be further from the truth. In fact, it is often a "WTF" moment. This is not uncommon, and a customer's AWS Cloud Networking design/deployment will more often than not be dictated by how far along they are in their journey with AWS. It's sometimes easy to forget that Cloud deployments are not like traditional on-prem deployments. There likely wasn't a huge program of work to design and deploy an enterprise-scale environment in the Cloud. There are exceptions, of course, but it might surprise you that a lot of companies' foray into AWS was setting up a single account and using a company credit card to pay the bill.

In this blog, I want to explore how a customer's hybrid networking environment may have evolved over time as they became more mature in their deployment in AWS. This is by no means what always happens but how things could evolve, at least in my head.

Yay lets use AWS..

So let's take the first step and look at what an environment might look like when we are starting out.

Above is about as simple as things get in terms of hosting an app in AWS: a single account, a single VPC with an Internet Gateway, some subnets to host instances, and a VPN Gateway to give connectivity back to on-prem to allow the app to talk back to other systems. As it stands, to host a single app, this architecture and hybrid connectivity would suffice despite what some might try to tell you.

Okay, so this is our base. Now let's evolve the solution.

WooHoo we love AWS lets expand..

After successfully using AWS for a period, our fictitious organization has decided they want to expand their AWS environment. They want a few more Prod VPCs and have also decided to aid with putting apps in the Cloud by adding staging and testing VPCs.

Now we have added the additional Prod VPCs and a Staging and Testing VPC. On the face of it, this is not too bad—still manageable but not the most elegant of solutions. It would still work, but as a former Network Architect, I would be raising alarms right about now about the future manageability of this environment.

Okay, so let's push the network admins to breaking point. These apps are now deemed critical and need resilient connectivity and resiliency across regions...

Weerrrrpp Weeeerrrppp WARNING WARNING!!!

The AWS deployment has been deemed so successful that the apps that run it are now critical to the organization. We need to evolve this to provide resilient hybrid connectivity and region resiliency... God help us!

And here it is in all its glory... Honestly, it looks like Peter Parker designed this. I think we can all agree this is now beyond manageable as a network service. So how do we solve this? Well, in this instance, we would now need to look at transit architectures. In all honesty, once you get to AWS holding critical apps, it's time to move to Direct Connect also. So let's morph this into a manageable network service again.

Ah... Thats better

Wow, what a masterpiece! We have introduced Direct Connect to give us a reliable, stable connection with bandwidth speeds based on requirements. This is linked into a Direct Connect Gateway, which makes the Direct Connect connections available across regions. We have then deployed a Network VPC with a Transit Gateway for each region, as they are only regional resources. Finally, each VPC has a Transit Gateway Attachment to allow it to have access to the Direct Connects and back to the on-prem systems (subject to the correct routing, of course).

This architecture would allow the organization to grow steadily with a low overhead to manage as it progresses.

However, there are some further additions and modifications that could be considered to further improve things. Let's look at some of those, as I am sure there are some veterans shouting at their screen that there are better options... lol

What else could be changed?

The first thing I would consider is centralizing Ingress and Egress Internet. While there is no charge for Internet Gateways (IGWs), there is a per-hour charge for NAT Gateways (NAT GWs). If we take our setup with 10 VPCs and likely at least 2 NAT GWs per VPC, that is going to get expensive very quickly. Moving Ingress and Egress Internet centrally will cut down on this cost and also give you more scope for having security solutions to protect your environment, such as proxy servers, etc. Because let's be honest, if you have gotten to this point in your AWS journey, it is more than likely that someone from your security team has clocked your rapid success and now wants to slam on the anchors to make sure things are secure :-)
At this point in the journey, it would probably be time to start having conversations about adopting and deploying a Multi-Account Landing Zone setup through the use of Control Tower or maybe even a customized Enterprise Scale Landing Zone through a partner. This is where we would have multiple accounts set up, all with different responsibilities. For instance, in a simple standard Landing Zone, you would have a Management Account that is your bill-paying account, the root of your AWS Org, and central IAM control. Then we have Logging and Archive to centralize logs from all accounts, Security and Audit account where we would enable security tools to protect your AWS environment and workloads, and then you would have your workload accounts. In our setup, for each VPC (and its DR equivalent), we would look to migrate that into its own account and then link it up with the relevant services. Unfortunately, this subject is too much to cover in a single paragraph, so if you are interested in Landing Zones, have a read-up separately. This setup really brings better governance to your environment and ensures that we have clear separation of responsibilities in the environment.

I hope this has been an enjoyable read, and I hope to do some further blogs on other AWS Networking subjects in the near future.

Neil works for Telefonica Tech UK helping guide their customers who are considering or using AWS for their production work loads, He also works across Telefonica Tech UK's business units helping develop offerings and solutions.

AWS Centralised Logging with Amazon OpenSearch

Neil Clark — Fri, 09 Aug 2024 15:31:28 +0000

When looking at any AWS Perscriptive guidance for deploying Multi Account Landing Zones, centralised logging is always a central pillar in an Well Architected envrionment.

A centralised Logging Account with all other accounts streaming logs to CloudWatch are the standard way of doing things.

While there is nothing wrong with doing this CloudWatch does have some short comings. Another architectural pattern that can be used to provide enhanced capabilities is to centralise logs into Amazon Opensearch.

There are some key benefits of using Opensearch:-

Fine-grained Data Access Control

Amazon OpenSearch Service allows you to limit access to data down to the field level and anonymize sensitive data based on user permissions. This is useful if you want to enable troubleshooting access without exposing sensitive information.

Aggregating Logs Across Accounts and Regions

By streaming logs from multiple AWS accounts and Regions into a centralized Amazon OpenSearch Service cluster, you can analyze trends, issues, and perform analytics across your entire infrastructure in a single location.

Flexible Querying and Analytics

Amazon OpenSearch Service provides a powerful search and analytics engine that goes beyond the capabilities of CloudWatch Logs. You can use advanced querying, aggregation, and visualization features to gain deeper insights from your log data.

Scalability and Performance

Amazon OpenSearch Service is a fully managed service that can automatically scale to handle large volumes of log data and provide low-latency search and analytics capabilities.

Integration with Other AWS Services

Amazon OpenSearch Service integrates with a wide range of AWS services, such as Amazon S3, Amazon DynamoDB, and Amazon Kinesis Data Firehose, allowing you to consolidate data from multiple sources into a unified platform.

How do I build an OpenSearch Setup?

I setup a very basic setup in my AWS account, three VPC's, Two subnets per VPC, One Instance deployed into each subnet, and VPC Flow Logs created two three log groups. I then created a Amazon OpenSearch Domain.

Below is a diagram of my simple setup

Lets look at the setup of the OpenSearch Domain

Create it with the minimum required

I set mine to public access, feel free to play with VPC Access

This is important, make sure if you are doing public access you add a source IP address for your public home broadband address so you can access the OpenSearch Kibana Dashboard. Also make sure that you add the IAM ARN of the Lambda function that is being used to transport logs from CLoudWatch to Opensearch, and also make sure it has all the relevant permissions for CloudWatch and Opensearch.

Once you can got to this point you can go ahead and create your domain. (this will take a while)

Once your domain is active we need to create the subscription filters in CloudWatch.

Select your log group and then select 'Actions' > 'Subscription Filters' > 'Create Amazon OpenSearch Service subscription filter'

note yours may look slightly different and ask you to select a Lambda Role you may have to create one.

Once you have done this you need to create soe traffic. I found the easiest way to do this was to just select all my instances and Stop them and then Start them again.

I would wait 10-15 mins to allow logs to get to the OpenSeach Cluster

The easiest way to check the logs are getting to OpenSearch is to click on your domain in OpenSearch and Click the indexes tab, if you see an index with the prefix cwl... they are getting there succesfully.

To Access the dashboad click on the Kibana URL link and this will take you in.

Once in you will be presented with the below screen, you need to click 'Connect to your Elasticsearch index'

you will then be asked to create an index pattern, cwl-* is the best one to use to see all the CloudWatch data.

Once you have created the index pattern you will be able to see your data

From here you can start querying your logs using the filter options, You can build dashboards to visualise the data.

Unfortunately I have not had much chance to play with Kibana but would encourage you to have a look at the differnt features it has.

If you run into issues with getting your data into OpenSeach I would check your Lambda logs to see if there are any permissions issues with the execution role and also check the security policy on the domain settings for OpenSearch to ensure that the Lambda IAM ARN is listed to be allowed to carry out actions against the OpenSearch domain.

Hope you find this whistle stop tour of setting up Amazon OpenSearch for Centralised Logging.

Neil works for Telefonica Tech UK helping guide their customers who are considering or using AWS for their production work loads, He also works across Telefonica Tech UK's business units helping develop offerings and solutions.

How to — AWS Auto Stop/Start of EC2 Instances using Tags

Neil Clark — Thu, 18 Apr 2024 10:30:21 +0000

In a previous blog post I talked about AWS Cost Control and some simple methods I used to cut cost on a customer environment by more than 50%.

I thought I would show you how this was achieved in a how to guide, so please join me below to see how I did it.

Pre-requisites

Regardless of which AMI you decide to use to run on your instances, ensure that SSM Agent is installed, this is present in some AMI’s but not all so it always worth checking.

This should be a standard install on all instances really in my view, Systems Manager has such an array of features it is silly not to have it.

How will we Identify the instances?

This is the first thing we need to decide… AWS Systems Manager has a couple of different ways to do this through the Maintenance Windows feature, you can manually select all instances in the Registered Targets but this is not ideal as it requires manual intervention when new servers are added, and the chances of someone remembering to do this, is slim.

The alternative is to use Instance Tags, I have two thought processes on this, you can either use an existing tag you already use on all instances, for example “environment:non-prod” on the face of it this seems like a great idea and for some it will, however I personally think using existing tags makes things a little less flexible, for instance if you wish to exclude some servers from being identified you would have to either remove the tag from those instances or change it, neither is really a great solution since tags are great for identifying instances. The other option is to have a specific tag for Auto Start/Stop, this is a tag that can be applied to all instances through IAC and then you only need to modify this tag to exclude instances… But you just said changing tags isn't a great idea, yes I did however this tag is a specific tag it means all my other identify tags are intact.

So in my solution I went for the tag as below:-

“auto_start_stop:yes”

So when “yes” is present in the key value pair the instance is included in the auto start/stop solution and when it is set to “no” it is not.

Two Instances

So in my Dev lab I have launched 2 instances Test_Server_1 and Test_Server_2. Test_Server_1 has the tag “auto_start_stop” set to “yes” and Test_Server_2 has it set to “no”.

At this point it is important to point out that the EC2 Instances should have an IAM Instance Profile attached with the sufficient permissions to allow Systems Manager Automation. I have attached the AWS managed policy “AmazonSSMAutomationRole” (see below) the others are used to allow the SSM Session Connection Manager to connect to EC2 instances that don't have a public IP.

Always remember AWS IAM Permissions make the world go round :-)

Resource Groups

The next thing we need to do is create a Resource Group in “Resource Groups & Tag Editor” under “Management & Governance” we will call this group later on in the setup.

So when you reach the correct page, create a new resource group.

Group Type = Tag based

Resources Types = AWS::EC2::Instance

Tags = auto_start_stop:yes

At this point if you Click the “Preview group resources” button you will notice the table below will populate with instances that match the tag combination.

Just a note that if you just add the Tag “auto_start_stop” with a blank entry when you click the preview it will pick up every instances with the tag and encompass all instances including instances you may not want to be included, so make sure you use the “yes” identifier as well.

Do you want to create a Maintenance Window?

Maintenance Windows are a feature of AWS Systems Manager if you hadn't gathered already. So browse to Systems Manager in the AWS Console, and on the left hand side find “Maintenance Windows” under the “Change Management” section.

Click “Create Maintenance Window”

Fill out the information needed, see my entries below

Name — auto_stop_2100 (for me it makes sense to have the time in to easily identify the window)

Description — will stop EC2 Instances at 2100

Unregistered Targets — Untick this

Schedule — Select CRON/Rate expression

CRON/Rate expression — cron(00 21 ? * MON-FRI *)

Duration — 3

Stop initiating tasks — 1

Schedule timezone — set this to your time zone ( this is especially important if you are looking to turn of resources when no one is using them overnight otherwise it will default to the AWS System Time Zone for the region.

Leave others options blank as they are optional.

This is obviously set for my specific time zone and requirements you will need to adjust your CRON expression and time zone to what you want.

Registering Targets

Now we have created our Maintenance Window we need to register some targets for it to run against.

So click on your maintenance window and click the “Targets” tab

Now click “Register Target”

Give your target a name, the select the radio button “Choose a resource group”.

In the drop down select the resource group we created earlier.

Under “Resource types” select “AWS::EC2::Instance” this is optional though I added it for completeness.

Then click “Register target”

Creating an Automation Task

The final thing we need to do is create an Automation task that will stop the instances.

So click the “Tasks” tab

The select the “Register tasks” drop down and select “Register Automation task”

Give your task a name

In the Automation Document section this is where you select the Automation you wish to run, here is where you see just how powerful Systems Manager is, the document we want is AWS-StopEC2Instance, and Document version we want to set to “Latest version at runtime”

Under the targets section we select the targets we want this automation to run against.

Select the tick box next to the target group we created earlier

The next section is the Input Parameters for the Automation, you will notice that Instance id is a required field, now you might think that you need to put in all the instance ID’s of your instances that you want the automation to run again, and if you like huge admin overhead then this would be the way to go, and I am not ashamed to admit in my first few plays with this that's exactly what I did… however AWS have a great way to get round this, it is called a pseudo parameter , so instead of putting all the instance id’s in we put the parameter

{{RESOURCE_ID}}

in this box. What this does is read the instance ID from the targets and passes it to the SSM Automation process.

The next section is rate control, this is how much you want done at once..

This is personal preference really I have set mine to Percentage and set Concurrency to 80% and an Error Threshold to 40%.

The final thing we need to set is the IAM Service Role, this should be a Role that has the permissions to run Systems Manager Automation and read the Resource Groups, for this demo I am using the same Role I assigned to my EC2 Instances however this wouldn't be best practice outside of a Dev Environment it would be a different role (I am being lazy here).

and that's it, click “Register Automation task”

We now have a Maintenance Window set to run Monday to Friday at 21:00 every night that will stop EC2 Instances with a Tag “auto_start_stop:yes”

Over to you

So as Morpheus said

I’m trying to free your mind, Neo. But I can only show you the door. You’re the one that has to walk through it.

I have shown you how to create a Maintenance Window that Auto Stops EC2 Instances, now its your turn to create the opposite and create one to Auto Start the EC2 Instances…

Good Luck and I Hope you found this useful. This is just one way of doing this there are many others, but rememeber your customer may not grant you full access to all services and features in AWS so you need to be flexible.

*UPDATE — should you wish to skip playing yourself, the link here will take you to my repo where there is terraform code to deploy what you need to run this solution.

Neil works for Telefonica Tech UK helping guide their customers who are considering or using AWS for their production work loads, He also works across Telefonica Tech UK's business units helping develop offerings and solutions.