DEV Community: ndewijer

From Docker to Lambda: An AWS Admin's Journey into Python Applications

ndewijer — Mon, 20 Jan 2025 14:33:07 +0000

You know how it goes, you start with a simple Python script to automate some AWS tasks, then another one to parse some CloudWatch logs, and before you know it, you're asking Claude to explain what a metaclass is because the documentation might as well be written in Ancient Greek. That was me, Three months ago.

How I Got Here

After years of writing Python scripts for AWS automation (and yes, I'm guilty of having one of those "does everything" scripts), I decided to build something proper. Between my own historic scripts, Stack Overflow, and Claude helping me understand Python's more esoteric features, I started to feel like I was finally getting this whole "software development" thing.

Screenshot of the app. This is Seed data, not my real investments. (I wish)

You see, I got tired of managing multiple Excel spreadsheets tracking various investment fund portfolios. Every time I bought or sold shares, I had to manually update prices, calculate returns, and track dividends. As an AWS professional, the idea of manual updates made me cringe. We automate infrastructure, why not this?

So I built my first real Python application: An Investment Portfolio Manager. Thanks to some help from our AI friends that did the heavy lifting on the front-end, via trial and error I learned about proper project structure, SQLAlchemy relationships and even some unit testing!

The application handled everything: portfolio management, transaction tracking, dividend processing, and even automated price updates. I had it running beautifully in Docker on my home server, with separate containers for the Flask back-end, React front-end (yes, I learned a bit of JavaScript too, thank you Mr. LLM), and a SQLite database.

The "Your hobby is also your job" Dilemma

Being an AWS professional, running this on my home server felt wrong. I mean, I live and breath AWS at my day job, why am I managing containers on my own hardware? Plus, every time my home internet hiccuped, my wife would complain that she couldn't check her investment returns. (Not really. She's very happy with the app and understanding of "the process" <3)

The easy way: ECS. I already had the docker-compose file:

services:
  backend:
    build: ./backend
    container_name: investment-portfolio-backend
    environment:
      - DB_DIR=/data/db
      - LOG_DIR=/data/logs
      - DOMAIN=${DOMAIN:-localhost}
    volumes:
      - /path/to/your/data:/data
    networks:
      - app-network

  frontend:
    build:
      context: ./frontend
      args:
        - DOMAIN=${DOMAIN:-localhost}
        - USE_HTTPS=${USE_HTTPS:-false}
    container_name: investment-portfolio-frontend
    environment:
      - DOMAIN=${DOMAIN:-localhost}
      - USE_HTTPS=${USE_HTTPS:-false}
    ports:
      - "80:80"
    depends_on:
      - backend
    networks:
      - app-network

But then I started thinking like an AWS architect (and looking at the AWS pricing calculator):

The price update function only needs to run once a day.
Same goes for our accessing it. We check it a few times a week. so why run (and pay for) 24/7 containers?
The frontend is just static files, sounds like an S3 website to me
API Gateway and Lambda to handle the back-end API calls
Aurora Serverless for the relational data (portfolios, transactions)
DynamoDB could store the price history (better than my SQLite price table) Spoiler: I never got to this step.

So, that's when I fell down the serverless rabbit hole.

I've gone in the shallows of this specific diving pool before. I've built a small serverless app, a temperature tracking project that was a collaboration with my wife. It pulls temperature data from KNMI (Dutch meteorological institute) and generates a color-coded table that my wife then used to create a temperature rug. Literally turning weather data into a crafting project!

| Date       | Min.Temp | Min.Kleur   | Max.Temp | Max.Kleur   |
----------------------------------------------------------------
| 2023-03-01 |   -4.1°C | darkblue   |    7.1°C | lightblue  |
| 2023-03-02 |    1.3°C | blue       |    6.8°C | lightblue  |
...

The app could run either locally or through Lambda via API Gateway, taking parameters for start date, end date, and date ordering. It was a perfect starter project, combining AWS services with a practical (and creative!) real-world use case.

Going from that simple Lambda function to a full Flask application with SQLAlchemy models, background jobs, and complex relationships is like trying to explain IAM policies to a developer; Make time in your agenda because it won't be quick and there's going to be some confusion along the way.

The Serverless Temptation

So, bringing it back, I couldn't help but look at my beautiful containerized application humming along and think "this could be more... cloudy." I mean, we have all these amazing serverless services, and here I am, managing containers like it's 2015.

The application was working great, don't get me wrong. But every time I was working in the AWS Console, those Lambda and API Gateway services were just sitting there, judging me. "Why aren't you using us?" they seemed to ask. "We could scale automatically, you know. No more container management..."

So I did what any reasonable AWS admin would do. Completely re-architect my application into that of a serverless architecture. Because why make small, sensible changes when you can do a complete architectural overhaul? The original project only took 2 months, I'm sure this will be a breeze.

The Database Dilemma

First up was the database. I had the backend use SQLite, which worked great locally, but then reality hit: SQLite and Lambda aren't exactly best friends.

Sure, you could use SQLite in Lambda (and part of me really wanted to try, Only way I'm ever getting into Corey Quinn's blog.), but let's be reasonable here. Plus, I'd then need to maintain two code bases. One for the docker-based one and another for the Lambda one.

Pass.

I needed something that would play nice with all the SQLAlchemy knowledge I'd just painfully acquired and after much contemplation (and several cups of coffee) later, I realized I could attempt to actually use both! SQLAlchemy speaks PostgreSQL Aurora Serverless seems like a good fit here. So, we make a dual-handler.

@contextmanager
def db_session():
    """
    Environment-aware database session manager.

    Why this approach:
    1. Flask App: Uses Flask-SQLAlchemy session management
    2. Lambda: Uses direct AWS database connections
    3. Connection pooling optimization
    4. Automatic cleanup of resources
    """
    if is_flask_context():  # Check if running in Flask
        # Flask benefits:
        # - Integrated with Flask-SQLAlchemy
        # - Handles application context
        # - Uses Flask configuration
        from ..models import db
        try:
            yield db.session
            db.session.commit()
        except:
            db.session.rollback()
            raise
        finally:
            db.session.close()
    else:  # Running in Lambda
        # Lambda benefits:
        # - Direct database connections
        # - Optimized for serverless
        # - No Flask dependency overhead
        session = create_aws_db_session()
        try:
            yield session
            session.commit()
        except:
            session.rollback()
            raise
        finally:
            session.close()

def is_flask_context():
    """Check if code is running in a Flask context"""
    try:
        from flask import has_app_context
        return has_app_context()
    except ImportError:
        return False

The Lambda Learning Curve

Next up was converting my Flask application to Lambda functions. How hard could it be, right? I mean, I've written plenty of Lambda functions before:

def lambda_handler(event, context):
    return {
        'statusCode': 200,
        'body': json.dumps({'message': 'Hello World!'})
    }

But converting a full Flask application? That was a different story. My first attempt looked something like this:

# My first attempt at converting a Flask route
@app.route("/portfolios", methods=["GET"])
def get_portfolios():
    portfolios = Portfolio.query.all()
    return jsonify(portfolios)

# So, I'm not even trying to keep the codebase singular and I'm already at this monstrosity
def lambda_handler(event, context):
    try:
        with db_session() as session:
            portfolios = session.query(Portfolio).all()
            return {
                'statusCode': 200,
                'headers': {
                    'Content-Type': 'application/json',
                    'Access-Control-Allow-Origin': '*'  # CORS... *sigh*
                },
                'body': json.dumps([{
                    'id': str(p.id),  # UUID needs string conversion
                    'name': p.name,
                    'description': p.description,
                    'is_archived': p.is_archived
                } for p in portfolios])
            }
    except Exception as e:
        print(f"Error: {str(e)}")  # CloudWatch, my old friend
        return {
            'statusCode': 500,
            'headers': {'Content-Type': 'application/json'},
            'body': json.dumps({'error': str(e)})
        }

So, this is terrible. And not maintainable. So, not wanting to repeat myself (The DRY principle see, I'm learning developer stuff!), I wrote a decorator:

def lambda_response(func):
    @wraps(func)
    def wrapper(event, context):
        try:
            result = func(event, context)
            return {
                'statusCode': 200,
                'headers': {
                    'Content-Type': 'application/json',
                    'Access-Control-Allow-Origin': '*'
                },
                'body': json.dumps(result)
            }
        except Exception as e:
            # Log to CloudWatch because that's where I live now
            print(f"Error in {func.__name__}: {str(e)}")
            return {
                'statusCode': 500,
                'headers': {'Content-Type': 'application/json'},
                'body': json.dumps({'error': str(e)})
            }
    return wrapper

Now my Lambda functions started looking much cleaner:

@lambda_response
def get_portfolios(event, context):
    with db_session() as session:
        portfolios = session.query(Portfolio).all()
        return [p.to_dict() for p in portfolios]

But that will only work for lambda. Now the original flask routes are broken. So, I wrote a decorator that would allow me to use the same routes for both Flask and Lambda:

def dual_handler(route_path, methods=None):
    """
    Decorator that supports both Flask routes and Lambda handlers.

    Benefits:
    1. Single source of truth for route definitions
    2. No conditional code in business logic
    3. Transparent handling of environment differences
    4. Easy testing of both modes
    """
    def decorator(f):
        # Register Flask route if in Flask context
        if current_app:
            f = current_app.route(route_path, methods=methods)(f)

        @functools.wraps(f)
        def wrapper(*args, **kwargs):
            # Detect environment and adapt accordingly
            if len(args) == 2 and isinstance(args[0], dict):
                event, context = args
                with current_app.test_request_context():
                    flask_request = create_flask_request(event)
                    flask_response = f(*args, **kwargs)
                    return create_lambda_response(flask_response)
            return f(*args, **kwargs)
        return wrapper
    return decorator

with the following combined functions to return the correct response, no matter what the environment:

def create_lambda_response(flask_response):
    """
    Convert Flask response to Lambda response format.

    Why this matters:
    1. Maintains API Gateway integration
    2. Preserves HTTP semantics
    3. Handles binary responses correctly
    """
    return {
        'statusCode': flask_response.status_code,
        'headers': dict(flask_response.headers),
        'body': flask_response.get_data(as_text=True)
    }

def create_flask_request(event):
    """
    Convert Lambda event to Flask request.

    Why this matters:
    1. Allows reuse of Flask route handling code
    2. Maintains compatibility with Flask extensions
    3. Enables gradual migration of functionality
    """
    http_method = event['requestContext']['http']['method']
    path = event['requestContext']['http']['path']
    headers = event.get('headers', {})
    query_string = event.get('queryStringParameters', {})
    body = event.get('body', '')

    # Convert query parameters to string format
    query_string = '&'.join([f"{k}={v}" for k, v in query_string.items()]) if query_string else ''

    builder = EnvironBuilder(
        path=path,
        method=http_method,
        headers=headers,
        query_string=query_string,
        json=json.loads(body) if body else None
    )

    return Request(builder.get_environ())

making it possible to use the same routes for both Flask and Lambda:

@dual_handler("/portfolios", methods=["GET"])
def get_portfolios(event, context):
    with db_session() as session:
        portfolios = session.query(Portfolio).all()
        return [p.to_dict() for p in portfolios]

The Static Simplicity

The front-end was actually the easiest part. All dynamic components are in the backend. S3 static website hosting and CloudFront? That's bread and butter.

A simple script like this can upload your frontend to S3 and invalidate the CloudFront cache to force a refresh of the content you just uploaded:

# This part I understand!
aws s3 sync build/ s3://${BUCKET_NAME} \
    --delete \
    --cache-control 'public, max-age=31536000' \
    --exclude 'index.html'

aws cloudfront create-invalidation \
    --distribution-id ${CF_ID} \
    --paths "/*"

The Results

After weeks of learning, coding, and occasionally questioning my life choices, I had successfully transformed my containerized application into a fully serverless architecture. I don't think I'll keep it online as I don't want to build the security around it or suffer a Denial of Wallet attack if the URL ever comes out, but I learned a lot and I'm proud of what I've built.

Here's what I learned:

Python isn't just for scripts anymore (but my shell scripts still come in handy. Come see: https://github.com/ndewijer/CodeSnippets
The AWS Free Tier is your friend during development
CloudWatch Logs are still the best debugging tool
Sometimes the "proper" way isn't the AWS way (and that's okay)

Would I do it again? Oh god no. But I'm glad I did it. My investment portfolio manager runs perfectly fine and secure on my private network but I loved the journey. I learned a ton about both Python and dual stack development.

Geo-restricting AWS Workspaces: Finding out how to cook parsnip soup

ndewijer — Mon, 22 Jan 2024 13:13:20 +0000

Once I spun up a virtual machine in Sydney to get a recipe from the MasterChef Australia website due to the site being restricted to just Australia. I could do this because of cloud computing. It made it possible to do work from anywhere on Earth as long as both ends have an internet connection.

Because while connectivity around the globe is becoming better, faster, and more prevalent, data sovereignty is becoming a significant issue for organisations. Be it the intellectual property of MasterChef recipes or the personal (health) data of residents in a country staying within certain geographic boundaries. And this is what I’ll elaborate on.

I worked for a British customer connected closely with the United Kingdom National Health Service (NHS), the central government-funded medical and health care services all things health in the United Kingdom. This organisation has access to very sensitive data, which falls under Tier 3 data as defined by the Alan Turing Data Safe haven tiers (1).

These tiers, together with agreements and requirements from the NHS, the United Kingdom’s Government and third-party data providers can be distilled into a set-in-stone rule that any and all data within the customer’s Tier 3 is not allowed to leave the United Kingdom (UK).

In. Any. Way.

The environment has all its resources placed in the AWS EU-West-2 region so that all data, instances and all ingress and egress into this environment are located within the United Kingdom. Data in S3 buckets, Data Analysis on EC2 instances, and ingress/egress of Data via secured upload methods, all authenticated and authorized via an AWS Managed Active Directory and again, all safely within the EU-West-2 region.

For User Ingress, Amazon WorkSpaces is going to be used. Amazon WorkSpaces is a fully managed desktop virtualization service for Windows and Linux. The Workspaces will give access to a secured Windows environment using a remote desktop session via PC over IP (PCoIP) that meets all the requirements set by the customer and regulators. But, where all infrastructure required for this to work, from domain controllers to the WorkSpaces themselves are on British soil, there is no way to have checks in place to see if the users themselves are in the UK.

By default, any users that can fully authenticate against the environment can connect to this environment from anywhere in the world. Long live cloud computing! Except when you’re not allowed to have data leave the country. Because, when we say that data cannot leave in any way, this also means the data, as pixels, on a screen of an RDP session. This means that an authorized user that works on this data daily is not allowed to do so when they are at a conference in Norway, for example.

Let’s delve into how to solve this. How can we prevent users from logging into their Workspace when not in the UK? How do we create geo-restriction for a solution that does not have any?

First, we start by looking at the documentation and see how this solution works. (2)

Figure 1 – AWS Managed WorkSpaces using AWS Managed Active Directory

We see that the solution initiates the connection with authentication from the client, via the AWS-managed Auth/Session gateway to the Directory Service that is managing authentication. After that completes successfully, the workspace streaming is a setup that users can actively work through.

The entire flow is completely managed by Amazon Web Services (AWS). Users (or administrators, for that matter) cannot manipulate the stream of data at all. When the author’s streaming data is attempted to be redirected or streamed via third-party solutions, it breaks. This is a great design. We don’t want any man-in-the-middle attacks succeeding.

Only when the traffic has come through the AWS-managed parts and arrives at the customer’s VPC do we gain control of this traffic and can steer it using the standard methods like security groups, routing, and NACLs. But, at that point, any data we could use for geolocation purposes, like the user’s IP address, have been lost.

So, what can we do?

The Amazon WorkSpaces client has a Proxy Server option, but this only applies to the auth traffic. The moment the WorkSpaces client tries to set up the streaming connection, only a direct connection to AWS is accepted. So, setting up a proxy server behind some form of geo-restriction is not going to work.

Let’s look at the backend options within WorkSpaces. The two client-side options are:

Trusted devices via certificates
IP access control groups

The trusted devices only control what devices can log into the Workspace but with IP Access control we can control where a user can log in from.

Trying to find out how many IP addresses / IP ranges are in use within the UK (4), we come to the lofty sum of 85.000 CIDR ranges. That is a bit more than what Amazon WorkSpaces supports. Amazon WorkSpaces documentation describes (5) that only 250 IP rules can be added to an Amazon WorkSpaces instance.

This inspired me to develop a solution. As I love cloud computing and it allows me to, if something doesn’t exist, build it myself. So, the solution I designed looks as follows:

Figure 2 – High-level infrastructure design WorkSpaces geo-restriction

While Amazon WorkSpaces does not have geo-restriction, AWS Web Application Firewall (WAF) does! WAF cannot be directly connected to workspaces, but we can use the IP Access Control List.

So, the design is:

An AWS WAF with a geographic match rule for the United Kingdom.
An Application Load Balancer with the WAF attached.
A Lambda that will validate the user and then set their home IP in WorkSpaces.
An identity provider that has credentials for the user, in this case, Cognito.

Figure 3 – UML Workflow user experience

Users going to the URL of the Application Load Balancer will have their IP checked by the WAF. If this fails, the user will not even have the opportunity to log in. But if the user is within the UK, the user can then sign in with their credentials. When that succeeds, the AWS Lambda will add the user’s IP to the access control list and the user can then log into their workspace. This entire workflow is shown in figure 3.

Finally, returning to my initial paragraph: this solution is not foolproof. By bringing up a virtual machine in the right location or using a VPN, working around the geo-restriction is not incredibly difficult. But like all things security, you work in layers. The geo-restriction works together with the authentication mechanism to not only have a user in the right place but also know the correct information to reach the secured environment.

In conclusion, a thank you to Network 10 for planting the seed of my geo-restriction design and thank you for not having the recipe behind a login or else I would never have been able to eat this wonderful parsnip soup (6) my wife made.

Links:

This blog was originally published on https://helecloud.com/blog/geographic-restrictions-workspaces-finding-out-how-to-cook-parsnip-soup/ but has since then gone down due to Helecloud's acquisition.

Scaling Identity Access Management: From Startups to Enterprises with AWS Solutions - Part 2

ndewijer — Mon, 24 Apr 2023 10:57:11 +0000

In Part 1, we discussed the benefits of AWS Organizations, such as centralized billing, organisational units (OUs), tagging policies, and service control policies (SCP), which provide enhanced security and standardization across AWS accounts. We also touched on AWS IAM Identity Center's capabilities, including integration with other Identity Providers (IdP) like Azure or Okta and the introduction of Attribute-based Access Control (ABAC). In Part 2, we'll explore how to leverage AWS Organizations, IAM Identity Center, and other AWS services together for easy, secure, and flexible access to your AWS environment

The meat and potatoes

Managing resource access with Identity Center, Tagging and SCP
The corner stone of this solution is the following IAM Policy condition:

"ec2:ResourceTag/Department": "${aws:PrincipalTag/Department}"

This policy line is comparing a dynamic resource tag to a dynamic principal tag. In short, this policy is Incredibly flexible and this section will explain why.

Lets start with a high-level diagram.

This diagram describes a user called John Smith within the Azure AD of “Company”. This organisation has setup AWS IAM Identity Center with SAML and SCIM and John is being synced into Identity Center. Further, John is a member of the Security group Developers within AzureAD.
The admins of the organisation have setup a permission set granting Administrator rights and have provisioned this permission set into their workload account for the Developer group to use.
In short, John now has admin rights to the AWS Workload account with his AzureAD user credentials.

The workload account that John has logged into with his administrator rights has an EC2 instance that belongs to the Finance department. Looking at the rules “Deny unless Allow unless Deny” we can infer that John can do anything and everything with this EC2 instance.

Now, let’s bring in the condition "ec2:ResourceTag/Department": "${aws:PrincipalTag/Department}"

The EC2 instance has a tag that defines the Department as Finance. By modifying the policy with a condition to only allow admin access if the user’s department matches that of the Instance’s department tag, as such:

{
  "Version": "2012-10-17",
  "Statement": [
      {
          "Effect": "Allow",
          "Action": "*",
          "Resource": "*",
          "Condition": {
              "StringEquals": {
                  "ec2:ResourceTag/Department": "${aws:PrincipalTag/Department}"
              }
          }
      }
  ]
}

We can define that they are an administrator but within a certain scope. Do be aware that this policy removes an allow so the rights for this now exists as an implicit deny which means that another policy could still grant access. To block any possible allows, the policy could be rewritten as

{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Effect": "Allow",
      "Action": "*",
      "Resource": "*"
    },
    {
      "Effect": "Deny",
      "Action": "*",
      "Resource": "*",
      "Condition": {
        "StringNotEqualsIfExists": {
          "ec2:ResourceTag/Department": "${aws:PrincipalTag/Department}"
        }
      }
    }
  ]
}

This places the policy in the explicit deny instead of implicit.

Do be aware that the above policy might break many things as not all resources can be tagged. Especially list and describe actions will be affected. A better way of specifying the policy for our EC2 instance example would be to declare the actions like this:

{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Effect": "Allow",
      "Action": "*",
      "Resource": "*"
    },
    {
      "Effect": "Deny",
      "Action": [
        "ec2:delete*",
        "ec2:modify*",
        "ec2:stop*",
        "ec2:start*",
        "ec2:create*",
        "ec2:associate*",
        "ec2:terminate*"
      ],
      "Resource": "*",
      "Condition": {
        "StringNotEqualsIfExists": {
          "ec2:ResourceTag/Department": "${aws:PrincipalTag/Department}"
        }
      }
    }
  ]
}

Web federated user

The core of AWS IAM Identity Center works via the federated user architecture specifically, web identity federation. This login method has existed for a long time and is also used by Amazon Cognito to authenticate. Keeping this in mind will make working with federated identities that are passed to the AWS accounts by Identity Center easier.

When the user ‘arrives’ in the workload AWS account, only a few bits of ‘identifying information’ are left that can be read and worked with.

❯ aws sts get-caller-identity
{
    "UserId": "AROA4XGUTKGXSNEJDUOXZ:john@company.com",
    "Account": "456789012346",
    "Arn": "arn:aws:sts::456789012346:assumed-role/AWSReservedSSO_AdministratorAccess_bff86352c1cc681d/john@company.com"
}

UserId, which replaces UserName when a user is federated. This contains the RoleID of the assumed role which will start with “AROA” and the username.
FriendlyName, which is the last part of the UserID, in this case “john@company.com”
Account, which is the AWS account ID the user is logged into.
Arn, which is the full Amazon Resource name of the principal user within the account.

UserId, FriendlyName and the PrincipalArn called in an IAM policy by using the
AWS:userID, AWS:friendlyName or AWS:PrincipalArn variables respectively.

These four items are the main identifying variables for a federated user. Identity Center however, also passes along the attributes of the federated user defined in Attribute-based Access Control as principal tags and this is what makes it possible to compare the principal tags against resource tags!

At the time of writing, there is no API call that can present a list of the full AWS session tags. So checking if a federated user has received the right tags into their account is a bit convoluted. A work around to this is looking up the “AssumeRoleWithSAML” event in CloudTrail of the account being logged into.

Such an event looks as follows and is split up into three main sections.

{
    “eventVersion”: “1.08”,
    “userIdentity”: {
        “type”: “SAMLUser”,
        “principalId”: “lUqu3Gchksa6MnzH4DmnCtbi8nA=:john@company.com”,
        “ sername”: “john@company.com”,
        “identityProvider”: “lUqu3Gchksa6MnzH4DmnCtbi8nA=”
    },
    “eventTime”: “2022-10-21T14:25:20Z”,
    “eventSource”: “sts.amazonaws.com”,
    “eventName”: “AssumeRoleWithSAML”,
    “awsRegion”: “eu-west-1”,
    “sourceIPAddress”: “xx.xx.xx.xx”,
    “userAgent”: “aws-internal/3 aws-sdk-java/1.12.291 Linux/4.14.287-148.504.amzn1.x86_64 OpenJDK_64-Bit_Server_VM/11.0.16+9-LTS java/11.0.16 vendor/Amazon.com_Inc. cfg/retry-mode/standard”,
The first section is the event information. Who is making the request, against what service, which region, when, etc.

“requestParameters”: {
        “sAMLAssertionID”: “x”,
        “roleSessionName”: “john@company.com”,
        “principalTags”: {
            “Department”: “Development”
        },
        “roleArn”: “arn:aws:iam::456789012346:role/aws-reserved/sso.amazonaws.com/eu-west-1/AWSReservedSSO_AdministratorAccess_bff86352c1cc681d”,
        “principalArn”: “arn:aws:iam::456789012346:saml-provider/AWSSSO_6a25cda37e6197d5_DO_NOT_DELETE”,
        “durationSeconds”: 28800
    }

Next comes the request parameters these contain variables presented to the service. For the request itself, the sAMLAssertionID, roleSessionName, roleArn and principalArn are the most important components but within this we can also see the principal tags being passed through:

“principalTags”: {
   “Department”: “Development”
}

Lastly, the service responds with what has been issued to the user.

“responseElements”: {
        “credentials”: {
            “accessKeyId”: “ASIA4xxx”,
            “sessionToken”: “xxx”,
            “expiration”: “Oct 21, 2022, 10:25:19 PM”
        },
        “assumedRoleUser”: {
            “assumedRoleId”: “AROA4XGUTKGXSNEJDUOXZ:john@company.com”,
            “arn”: “arn:aws:sts::456789012346:assumed-role/AWSReservedSSO_AdministratorAccess_bff86352c1cc681d/john@company.com”
        },
        “packedPolicySize”: 1,
        “subject”: “john@company.com”,
        “subjectType”: “persistent”,
        “issuer”: “https://portal.sso.eu-west-1.amazonaws.com/saml/assertion/ODc0NDxxxx”,
        “audience”: “https://signin.aws.amazon.com/saml”,
        “nameQualifier”: “lUqu3Gchksa6MnzH4DmnCtbi8nA=”
    }
}

Especially the “assumedRoleId” is important here as that is the main “userId” that the user identifies itself with inside the AWS account.

Adding oomph to IAM

In the last chapter we setup IAM policies with IAM Identity Center to only allow users to work with resources that they ‘own’, so to speak. We’ve done this by defining a policy that only allows actions on a resource if the principal making the request has the same department tag value as the resource.

Now let’s add an extra layer of enforcement to make sure resources actually have this tag!

All credits go to Arun Chandapillai and Shak Kathir who have written a blog about this exact subject. A link to the blog can be found in the sources.

Within the AWS Organizations, three types of policies can be defined that enforce certain behaviour and/or restrictions on its member accounts. Backup policies, Tagging policies and Service control policies (SCPs).

By creating a tagging policy that looks as follows:

{
  “tags”: {
    “costcenter”: {
      “tag_key”: {
        “@@assign”: “department”
      },
      “tag_value”: {
        “@@assign”: [
          “Development”,
          “Finance”,
          “Sales”,
          “Legal”
        ]
      },
      “enforced_for”: {
        “@@assign”: [
          “ec2:instance”,
          “ec2:volume”
        ]
      }
    }
  }
}

And attaching it to either an account to just effect that account or an OU to affect all accounts in that OU and its children, it will enforce that when the tag department is used on an EC2 instance or volume, it must comply with having a specific value.
The challenge here is that this policy does not prevent the tag from not being created in the first place and this is where an SCP comes in.

{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Effect": "Deny",
      "Action": [
        "ec2:RunInstances",
        "ec2:CreateTags",
        "ec2:DeleteTags"
      ],
      "Resource": [
        "arn:aws:ec2:*:*:instance/*",
        "arn:aws:ec2:*:*:volume/*"
      ],
      "Condition": {
        "StringNotEquals": {
            "ec2:ResourceTag/department": "${aws:PrincipalTag/department}"
        },
        "Null": {
            "ec2:ResourceTag/department": false
        }
      }
    }
  ]
}

By creating the following SCP and attaching it to the same accounts, EC2 instances or EC2 volumes cannot be created without having the department tag attached to the resources. The tags can also not be (re)created by someone that’s not from the same department nor can it be removed at all.

The combination of the tagging policy forcing a specific value for resource tag, enforcing newly created resources to have this tag and then blocking access to modify the resource tag will ensure that resources in scope will always meet the tagging requirement.

Then using the IAM policies to only allow access to resources if they match their department tag with the principal’s department tag will guarantee only the right access is provided.

Managing organisation access with Identity Center and SCP

AWS advises that, wherever possible, as many separate AWS accounts should be used within your AWS Organization (AWS Org) as possible and workable. “An AWS account provides natural security, access and billing boundaries for your AWS resources, and enables you to achieve resource independence and isolation.”

Following this advice, we can break accounts into out different ways.
Several examples are a workload-based design:

Or a prod/non prod design:

Or a department style setup:

The options here are numerous and each organisation should take care to make an OU design that fits for their purpose.

By splitting the OUs in the way that works for the organisation, creating specialised policies becomes easier.

We’ve spoken about how AWS IAM Identity center provides access to users by linking them directly or via a group to an AWS account via a permission set.

What is also possible within Identity center is to delegate access to another AWS account. For instance, a dedicated security account.

By setting up access permissions in such a way, users that cannot access the master billing account can access Identity center from the Delegated Administrator account.
These two components, a delegated Identity center and SCPs can combine to give an extra layer of security to who can and cannot access the AWS accounts within the AWS Org.

Within this example, the following SCP is assigned to the Development OU:

{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Effect": "Deny",
      "Action": "*",
      "Resource": "*",
      "Condition": {
        "ForAllValue:StringNotEquals": {
          "aws:PrincipalTag/Department": [
            "Development",
            "Audit"
          ]
        }
      }
    }
  ]
}

This policy will add a second layer of ‘defence’ that will prevent any user that is not part of the Development or Audit department to perform any action within the accounts under the Development OU.

Even if, for an example, a user/group from the Operations department is assigned to a development account, the SCP will block usage of this account as the department is defined in the organisations identity provider.

And due to the SCP and Identity Center running in different accounts, these two vectors of security can be kept separate.

Outro

Combining AWS Organizations with AWS IAM Identity Center enables organisations to securely grant their teams access to AWS accounts and resources while utilising their existing Identity Access Management solution.

This native toolset is ideal for organisations as they begin working with AWS, providing essential guardrails and tools. Additional services like AWS GuardDuty, Inspector, and AWS Config help address security and compliance needs, while AWS Billing offers insights into costs across the AWS estate. These services effortlessly scale with your organisation in AWS, regardless of its size.

Nick de Wijer – Principal AWS platform architect.

Scaling Identity Access Management: From Startups to Enterprises with AWS Solutions - Part 1

ndewijer — Mon, 24 Apr 2023 10:57:07 +0000

Identity access management (IAM) becomes increasingly complex as organizations grow from startups to large enterprises. Initially, with a small team, managing access and identity is simple, but as the company expands, so does the need for enhanced IAM systems. New hires, department restructuring, and the introduction of cloud platforms like AWS further complicate IAM. Learn how AWS IAM Identity Center and Service Control Policies via AWS Organizations can help streamline user and group management, while maintaining robust security.

Before going deep in hooking up all kinds of different services together, let’s begin with a bit of context. To put all the different services and acronyms in their right place.
If you’re familiar with the relevant AWS services, you can skip to part 2 of the blog.

IAM Policies 101

I often summarise access control in AWS with this statement.

Denied unless Allowed unless Denied.

What this means is that, by default, an entity within AWS has no rights to do anything unless these rights have been given AND, and this is the important part, they have not been explicitly denied.

So, written out fully, it’s Implicitly Denied until Explicitly Allowed until Explicitly Denied.

Within AWS what is allowed or denied to an entity is defined within Identity Access Management (IAM) policies. These policies are JSON formatted documents that define what a “principal” is allowed to do or, more importantly, not allowed to do within a certain context.
This principal is the initiator of the action and can be an IAM user or an application because within AWS, nothing and no-one is allowed to do anything without an IAM policy allowing it to do so.

Policies do not live in a vacuum. Policies are attached to things. These things differ depending on the policy.
If the policy is identity-based, they are assigned to roles or groups which in turn are assumed by (roles) or assigned to (groups) users or other AWS resources.
If its resource based, the policy is attached directly to an AWS resource that will then be used to determine what rights the interacting principal has on this resource.

The components of these policies are as follows:
• Version: The policy language version so that AWS can correctly parse the data defined in the policy. Two exist, the older 2008-10-17 and the current 2012-10-17.
• Statement: Contains one or multiple policy statements.

Within the statement, policies are defined with the following.

Principal

Defines the principal (initiator of the action) that this policy statement applies to.

A principal, when defined must map against an IAM or STS user, group or role or an AWS Service. These can be as broad as an entire AWS account or as narrow as a single IAM user.

Examples:
AWS Account:

"Principal": {"AWS": "arn:aws:iam::045676890123:root"}

AWS Service:

"Principal": {"Service": "delivery.logs.amazonaws.com"}

STS Assumed Role:

"Principal": {"AWS": "arn:aws:sts::045676890123:assumed-role/{rolename}"}

AWS Role:

"Principal": {"AWS": "arn:aws:iam::045676890123:role/{rolename}"}

If the policy is used in an identity-manner, it is not required to be defined because it is inferred by the principal the policy is attached too.
If the attached is used as a resource-based object however, it is required to define what principals are allowed to interact with the resource.

Effect

Does the policy either “Allow” or “Deny” the action defined in the policy.

Action

Define an action within AWS.
Actions are defined as “{service}:{action}” where part or the entirety can be replaced with a wildcard *. Examples are:

"Action": "s3:PutObject"

This action allows for the writing of an object into S3.

"Action": "kms:*"

This action allows every action within the Key Management Service (KMS)

"Action": "*"

This action allows everything.

If not secured with either a Resource or a Condition statement, it will allow full access to every part of an AWS account or resource.

Actions can be defined as “Action” or “NotAction” creating a white or blacklist respectively. Combined with Effect:Deny, double-negative style allows can be created. We will zoom in on this later.

Resource

Define the AWS resource that the Action is limited too. This resource must be defined with an AWS ARN and one or more wildcards (*) can be used to broaden the scope of the resource(s) under the policy.

Examples are:

"Resource": "arn:aws:s3:::accesslogs/*"

Which defines all objects within an S3 bucket called “accesslogs”

"Resource": "arn:aws:kms:eu-west-1:045676890123:key/87132241-9a03-4c89-a723-af0b43aea2bc7"

Which defines a specific KMS key within account 045676890123 in AWS Region eu-west-1
"Resource": "*"
Which defines all resources

Condition

Conditions is an optional statement that can contain one or more sub-statements where further allowances or limitations can be made on top of Principal/Action and Resource.

Within conditions, logic checks can be performed. For example, comparing values from the calling principal to the targeted resource. Another often used condition is to check (and then block) for unencrypted traffic.

Full Policies

Bringing all these together will create policies as these two examples:

{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Effect": "Allow",
            "Action": "*",
            "Resource": "*"
        }
    ]
}

This identity-based policy allows administrator rights to the entire AWS Account.

{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Principal": {
                "AWS": "arn:aws:iam::045676890123:root"
            },
            "Effect": "Allow",
            "Action": "S3:PutObject",
            "Resource": "arn:aws:s3:::accesslogs/*",
            "Condition": {
                "Bool": {
                    "aws:SecureTransport": "true"
                }
            }
        }
    ]
}

This resource-based policy allows all principals from the account 045676890123 to write into the bucket “access logs” as long as the connection is encrypted.

Bringing it home

When a request is made within AWS all the policies attached to the identity requesting and the resources requested from are taken and combined into a single policy. Then, if at least one “Allow” to the resource and no “Deny” is found, access is given.

If, however, a “Deny” is found, the request will be denied, even though an “Allow” for that same request exists.

AWS Organizations (Service name, further abbreviated to AWS Org) is designed to be used by organisations (EN-UK spelling) to centrally manage AWS accounts under their ownership more effectively.

It allows for grouping AWS accounts in a hierarchical method using one or multiple levels of organizational units (OU) based on the requirements of the organisation.

These OUs will house one or more AWS accounts and can have several types of policies attached to either the OU or accounts directly to allow management on a high or granular level. The available policy types are.

Backup policy – Ensure consistency in how AWS Backup plans are implemented
Tag policies – Standardise tags on all supported resources
Service control Policies (SCP) – Control IAM permissions within the AWS account
AI services opt-out policies – Control what AI Services can store and what content within an account it can use to do its job

When a policy is applied to an OU, the policy will affect all child OUs and AWS accounts within those OUs.

Service control policies (SCP)

Service control policies (SCP) introduce an extra level of securing AWS accounts by granting or denying access to resources.

We’ve spoken about IAM policies. How they are constructed and how identity and resource-based policies grant or deny access for a request.

What is important to know at this stage is that that an extra step is introduced when AWS Org is enabled.

Where before, only Resource and Identity based policies (and permission boundaries but I am leaving these out right now for the sake of simplicity) were at work determining if a request for access was to be allowed or denied. With the introduction of SCPs, they will also weigh in if a request will be allowed or denied.

Tagging policies

The last component I want to highlight within AWS Organizations is tagging policies. These policies can enforce the way resources can be tagged.
*They however do not enforce a specific tag being used. *

{
    "tags": {
        "deparment": {
            "tag_key": {
                "@@assign": "Department"
            },
            "tag_value": {
                "@@assign": [
                    "finance",
                    "sales",
                    "security"
                ]
            },
            "enforced_for": {
                "@@assign": [
                    "ec2:instance",
                    "kms:*"
                ]
            }
        }
    }
}

As an example, this policy defines that if the tag “Department” is used, it must have one of the three values defined in “tag_value”. Furthermore, this enforcement is then only done on ec2:instance and all kms resources.

It’s important to fully understand the goal of tagging policies. They are designed to be used for compliance reasons. Using the Management console or the AWS CLI, you can generate a report of non-compliant resources to allow for correction later.

Also, not all resources can be tagged. “ec2:*”for instance is not supported and specific resources must be defined. A complete list can be found here

Enabling AWS Org will also enable other services and enable centralised management functionally of existing services. A sample of services which are important to AWS Org or relevant to this document are the following:

Centralised billing - Free

Enabling AWS Org instantly turns the account that created it into the “master billing account” (MBA), also known as the Root account. What this means is any account in the AWS Org, including itself, reports its AWS service usage to the MBA from where a single billing dashboard will be available, and a single bill will be generated and paid out to AWS. The consolidated billing dashboard is free to use and a great way to analyse costs across the accounts in the AWS Org by service, account or any tagging enabled for billing purposes.

AWS IAM Identity Center (successor to AWS SSO) – Free

This service used has been recently renamed from AWS Single Sign-On (SSO) which is why the name right now is a mouth full as both are being used in this transition phase.

Identity Center enables the organization to centrally manage and control the AWS Management Console and API access for Users and Groups across all AWS Accounts within the Organization. This is done by using either the built-in Users and Groups functionality of IAM-IC or by connecting to a third-party Identity provider (IdP) like Okta or AzureAD.

We will go in further detail in the next chapter.

Enterprise support – Paid

Standard AWS accounts have three tiers of support.

Basic, which allows only for account, billing and business support cases.
Developer, which opens up the capability of logging technical support cases.
Business, which adds P1 and P2 urgency to support cases and give quicker support.

With an Organisation, a fourth tier opens; Enterprise support. (Which itself split up in two tiers. Enterprise On-Ramp and Enterprise.)

The first three tiers of support were set on an account level. Enterprise support is set at the organizational level and allows for every single account to have the highest grade of support available.

AWS IAM Identity Center

“AWS IAM Identity Center (successor to AWS SSO)” is the single sign-on solution for AWS accounts joined together within an AWS Org. Identity Center gives a centralised location to manage users and group, their access to AWS accounts within the organization and what rights they have within those accounts using Permission sets. By default, the root account of the AWS Org is the administrator of Identity Center but this can be delegated to, for instance, a dedicated IAM or Security account. This will prevent to many users accessing the, arguably, most important account within the AWS Org.

A permission set is a structure that contains one or more of the following IAM policies:

AWS Managed Policies
Customer managed policies
Inline Policy, so defined directly inside the permission set
Optionally, a permissions boundary.

When a Permission set is provisioned in a member account, an IAM role is created with policies that correspond to the specific rights defined within the permission set. E.g., an attached policy or an inline policy. These IAM roles (and policies) are protected entities that cannot be modified from the member account.

When a user logs in via Identity Center, they can then assume the IAM role in the member account to work within this account within the boundaries defined in the IAM role.

The power of AWS IAM Identity Center lies in its capability of integrating with Security Assertion Markup Language (SAML) 2.0 compliant Identity providers (IdP). Most organisations already use an identity provider to manage their users with providers like Okta, Active Directory Federation Services (ADFS) or Ping among others.

Going one step further, by using System for Cross-Domain Identity Management (SCIM), users and groups can be synchronised automatically from the SAML IdP into IAM-IC.

The result of this allows organisations to setup a single sign-on environment where users can access AWS with the same credentials they use for their workstation, email, or other work environments. Often without having to even fill in their credentials again.

Attribute based access control

Identity Center also introduces a feature that is not enabled by default and slightly hidden away but allows organisations to manage user access in an entirely new way by basing access on enterprise specific parameters, which is called Attribute-based access control (ABAC).

In a simple Azure AD SAML & SCIM setup, only these user attributes are synchronised:

https://docs.aws.amazon.com/singlesignon/latest/userguide/attributemappingsconcept.html

These attributes are the ones minimally required to perform successful logins into AWS.
What ABAC allows for is to sync more attributes from AAD into Identity Center.

Examples of these are department and cost centre. Or office location and function title.

By passing these attributes into AWS and linking them in ABAC, direct references to these can be made by Identity Provider.

These attributes, as the name of the service implies, can then be used to help dictate access for users within AWS.

Now, let’s bring all these components together and use them in tandem to provide easy, secure, and flexible access into AWS.

Read part 2 of this blog.

An AWS Community Builders holiday story

ndewijer — Tue, 20 Dec 2022 11:00:00 +0000

Hi all!

I love the initiative that the 'HotChowSprings' crew thought up. There has been a lot of love been given to the community from the community leaders which immediately moves into the first question!

What surprises you most about the community builders program?

I've been surprised by the level of interaction between the community leaders and the community builders. But also, and this activity is just a public example, of the community builders giving back to the program.

I love the Slack community. The help we give each other from technical and mindset, to education and training. It's spectacular.

What’s your background and your experience with AWS?

I've been working with AWS for years. I started together with my then company going from datacenter to AWS.

I started as a Windows engineer when the cloud project started. But by stepping up and proposing, designing the method of migrating Splunk into AWS using IaC. The company gave me the trust to implementing it and I succeeded without downtime and under time/budget.

Over the years, I've designed an implemented more and more workloads on AWS. Moved companies and now I do architecture for a living!

What’s the biggest benefit you see from the program?

The biggest benefit I feel is the connection with like-minded people all over the world. You need to apply to get into the Community Builders. And everyone that made it in has put in a good amount effort to bring AWS to the world.

By speaking with other Community Builders, I get inspiration in my own work. And I've helped people out with technical and architectural advice.

What’s the next swag item that you would like to get?

I am such a hoodie-hoarder. I wear them every day. My 2020 hoodie has been worn down that my wife is complaining.

This year, at Re:Invent I was able to bribe my colleague for his hoodie so I could bring back two and could switch them.

So, in case it wasn't obvious, I'd like a lanyard... No, a hoodie! CB Hoodie! GIEV! ;)

What are you eating for dinner today? Share the recipe!

Tonight I'm making Chinese Mushroom Noodle Soup!

The base recipe is here:
https://www.errenskitchen.com/chinese-mushroom-noodle-soup/

I'm adding a bit of spring onion and pak choi to it for some more veggies!

Is there anything else you would like to say about the community builders program in 2022?

Thank you for existing. Your public and private presentations and interactions are amazing, the events are great and the community is awesome. I love being part of it. Giving us this platform on dev.to is great, even though I haven't used it much. (a blog is coming though!) If I were to give any comment, especially for our use of dev.to, it would be to inspire for quality over quantity.