Rerouting legacy server IP address

Petri Kallberg — Tue, 02 Nov 2021 18:38:58 +0000

This is second repost in series that covers various details of migrating services from traditional data center to AWS. You can find this and other post from my personal blog https://carriagereturn.nl/

Problem

I’ve been working on lift-and-shift -migrations from on-premises data center to AWS. In these projects you often find workloads that have been running for a fairly long time with minimal changes and not all “modern IT” practices can not be taken for granted.

One such practice is using DNS to find server IP address. It is not that uncommon to find an application with clients connecting to server with IP address. And when moving the server from data center to AWS VPC, IP address is going to change because typically it is not possible to move the whole network in one big bang and reuse data center CIDRs for AWS VPC.

So how could clients keep connecting to the old IP address, until they have been reconfigured?

Solution

Sometimes an old idea can be repurposed for a new use-case. I was thinking the problem of rerouting the legacy server IP to EC2, when I remembered the post of Thinking outside of the VPC where I described a pattern of using IP address outside of VPC CIDR. The Goal at the time was to implement a virtual IP that could be “assigned” to EC2 instances across multiple subnets and AZs. Could the same technic be used for routing data center IPs to EC2s? It turns out it could!

Here is how to reroute a single IP address from data center network to an EC2 instance in VPC.

In the on-prem router, route the legacy server IP 1.2.3.4/32 to AWS Direct Connect or VPN.
Next is the Transit Gateway attached to target VPC. In the diagram it is drawn on the same account and region as the target VPC but it would work also across AWS accounts and regions. In Transit Gateway route table add a static route for 1.2.3.4/32 to attachment of target VPC.
In VPC Subnet route table add a static route for 1.2.3.4/32 to the new server instance ID.
On EC2 instance OS level, define the secondary IP address 1.2.3.4/32 for the network interface.
Disable EC2 instance Source/Destination Check to so packets to 1.2.3.4/32 can reach it.

Limitations and side effects

Nice side effect of above is that the EC2 instance will answer on both old and new IP address. This means the clients can be reconfigured to new IP address (or maybe even start using DNS) one by one. When all of them are using the new IP address, static routes for legacy IP address can be removed from Transit Gateway and VPC route tables.

Obviously this isn’t the solution for rerouting IPs in large scale. Route tables can easily grow very complex and pretty soon hit the default limit of 50 routes in VPC route table. While the limit can be increased up to 1000 routes, it will have a negative impact on VPC network performance.

If the instance ID will change, static route in the subnet route table must be updated. When using Auto Scaling group of 1 instance, to ensure server will stay running, it will become a bit more complex as code for route table update must be included in server start-up routine. This is also limited to routing to EC2s and ENIs. That is ok for most lift-and-shift workloads, but not suitable for cloud-native architectures.

Delegating IAM access with permission boundary

Petri Kallberg — Sun, 17 Oct 2021 14:08:24 +0000

This is first repost in series that covers various details of migrating services from traditional data center to AWS. You can find this and other post from my personal blog https://carriagereturn.nl/

A Brief history of shared AWS accounts

Back in the days, when creating an AWS account was a lot of overhead, common pattern was to have multiple teams sharing a single account. This created a demand to isolate teams from each other by assigning them an IAM policy that would grant access only to their part of the account. While it might have been possible (in theory) to create complex IAM policies to containerize resources by application or team, one fundamental problem did remain. Once you granted a right to create an IAM policy or role, you effectively granted admin level access to everything on the account. IAM has very granular controls for actions, it can not limit what policies you attach to a role, or content of policy documents.

To solve this problem, AWS introduced in July 2018 feature called IAM permission boundary. If this went unnoticed, don't feel back about yourself. If you have been working in modern environments, it has been the best practice to use AWS account as boundary and create accounts dedicated to a single team, application or environment.

IAM permission boundary

There are 3 types of IAM polices that control your access to AWS APIs. First and the most common is normal identity-based policies that are attached to roles and users. Second type is organization service control policies (SCP) that are attached to AWS accounts or OUs. These can limit what the local admins can do on the account and as they are not part the account, they can not be modified by local admins. Typical use-case for SCPs would be limiting regions or services you are allowed to use within given account.

The Third type is permission boundary policy. It is similar to regular IAM policies inside of an account, but the use-case is similar to SCPs, limiting what can be granted in normal policies.

Policy evaluation logic works so, that you are only allowed for an actions if all three policy types allow it. You may remember that single deny will always overwrite any number of allow statements, but another thing to note is that all policy types must also have allow your actions, ie. the effective permissions are the intersection of all policies applied. This is why you typically want to have allow-all -statement in permission boundary or SCP policies, and then deny the parts you don't want to be granted via normal IAM policies.

Setting boundaries for a team

The Goal is to allow users to manage IAM policies, roles, users, groups and instance-profiles so they can work idependently. But at the same time, protect their resources from others sharing the same AWS account. Ownership of an resource is claimed using a tag with unique value to user, team or role. Ie. something that wasn't possible before permission boundaries.

Dissecting the boundary policy

As a demonstration of how IAM permission boundary works, I wrote a cloudformation template that creates an IAM user with AdministratorAccess and permission boundary attached. I've dissected the boundary policy statements below.

First there is normal IAM policy header stuff. Only interesting piece here is ManagedPolicyName. This is going to be restricted in following statements.

  IAMBoundary:
    Type: AWS::IAM::ManagedPolicy
    Properties:
      ManagedPolicyName: !Sub "${AWS::StackName}-boundary"
      Description: IAM permission boundary to force boundary inheritance
      PolicyDocument:
        Version: "2012-10-17"
        Statement:

AllowUnlessOwnedBySomeoneElse

This statement will allow all actions, on all resources, unless there is an owner -tag that has different value than the IAM role or user owner -tag has. Ie. if some resource is missing the owner -tag, they are considered as common property. This is something you have to accept, as it is not possible to tag all resources at creation but create and tagging are 2 separate API calls. It is also assuming that when role or user is created, it will be tagged with team or application specific owner -tag value as StringEqualsIfExists won't apply when aws:PrincipalTag/owner is not present.

          - Sid: 'AllowUnlessOwnedBySomeoneElse'
            Effect: Allow 
            Action: '*'
            Resource: '*'
            Condition:
              StringEqualsIfExists:
                'aws:RequestTag/owner': ${aws:PrincipalTag/owner}
                'aws:ResourceTag/owner': ${aws:PrincipalTag/owner}

DenyBoundaryModification

Next thing is to deny modification of this boundary policy. Deny rule applies only this single policy. You can still grant access to all other IAM policies.

          - Sid: 'DenyBoundaryModification'
            Effect: Deny
            Action:
              - iam:CreatePolicyVersion
              - iam:DeletePolicy
              - iam:DeletePolicyVersion
              - iam:SetDefaultPolicyVersion
            Resource: !Sub "arn:aws:iam::${AWS::AccountId}:policy/${AWS::StackName}-boundary"

ForceBoundaryInheritance

This is the beef of permission boundary. You are allowed to create a role or user only if that has the same permission boundary attached as you have.

          - Sid: 'ForceBoundaryInheritance'
            Effect: Deny 
            Action:
              - iam:CreateUser
              - iam:CreateRole
              - iam:PutUserPermissionsBoundary
              - iam:PutRolePermissionsBoundary
            Resource: '*'
            Condition:
              StringNotEquals:
                'iam:PermissionsBoundary': !Sub "arn:aws:iam::${AWS::AccountId}:policy/${AWS::StackName}-boundary"

DenyBoundaryRemoval

Inheritance isn't enough, but it is also necessary to deny removal of permission boundary.

          - Sid: 'DenyBoundaryRemoval'
            Effect: Deny
            Action:
              - iam:DeleteUserPermissionsBoundary
              - iam:DeleteRolePermissionsBoundary
            Resource: '*'
            Condition:
              StringEquals:
                'iam:PermissionsBoundary': !Sub "arn:aws:iam::${AWS::AccountId}:policy/${AWS::StackName}-boundary"

DenyIAMChangesWithoutPrefix

Final statement of the boundary policy, restricts the IAM resources; users, groups, roles, policies and instance-profiles, to have a prefix of your IAM pricipals owner -tag value. This is ensuring you won't be able to modify or attach other than your own IAM resources.

          - Sid: 'DenyIAMChangesWithoutPrefix'
            Effect: Deny
            Action:
              - iam:Add*
              - iam:Attach*
              - iam:Create*
              - iam:Delete*
              - iam:Detach*
              - iam:Pass*
            NotResource:
              - !Sub "arn:*:iam::${AWS::AccountId}:group/${!aws:PrincipalTag/owner}-*"
              - !Sub "arn:*:iam::${AWS::AccountId}:policy/${!aws:PrincipalTag/owner}-*"
              - !Sub "arn:*:iam::${AWS::AccountId}:role/${!aws:PrincipalTag/owner}-*"
              - !Sub "arn:*:iam::${AWS::AccountId}:user/${!aws:PrincipalTag/owner}-*"
              - !Sub "arn:*:iam::${AWS::AccountId}:instance-profile/${!aws:PrincipalTag/owner}-*"

And that's it! Now you have isolated multiple teams within a single account using IAM permission boundary. You can tag now the IAM user with owner -tag and some value. Then using a different user, create resources, some tagged with the same value as the IAM user created, some with different values, and see how access is only granted to resources matching your tag.

NOTE: There is no need to create team specific boundary policies but the same policy can be attached to all roles and users that need to be isolated. This is thanks to IAM policy variable aws:PrincipalTag.

Too little, too late?

But is this "too little, too late", as now it is also easy to create new accounts using organizations?

If there is an option to have dedicated AWS account, that would be the ultimate boundary. But there are also many large migrations from data center to AWS, where it isn't feasible to create own account for every workload. This is where shared accounts with permission boundaries can become helpful. And not to forget those shared "legacy" accounts that many of us have created in past, and are not quite yet split into single tenancy model ;-)

So, there actually might be more use for IAM boundaries than ever!

P.S. I'd love to hear if you found holes in the boundary that I missed.

P.P.S. Lambda applies automatic permission boundary to application
IAM roles it creates. You can use auto created boundary policy or define your own in SAM template Globals -section.

DEV Community: Petri Kallberg