How-to debug and trace problems in AWS CodeBuild

Following an "AWS Unless" strategy within our company, it was only a matter of time before we would move our builds to AWS CodeBuild and AWS CodePipeline. Migrating our legacy pipelines to AWS CodeBuild turned out to be a pretty straightforward job. For that reason, CodeBuild was easily adopted by our development teams.

However, in general there was one complaint about CodeBuild, troubleshooting problems in the buildspec.yml was hard, mainly because access to the remote session was nonexistent.

That was until last July when AWS Session Manager access for AWS CodeBuild was announced. It's strange but it seems that the release of this awesome feature went by unnoticed!? Maybe that's due to the fact that the press release seems to miss the right semantics? It's definitely a hard post to find even for Google if you don't use the right words. Fingers crossed that this article can bring more attention to this feature :-)

By enabling remote access, AWS Session Manager finally brings debug capabilities to AWS CodeBuild. Besides Session Manager access, the new CodeBuild command codebuild-breakpoint is the key to this new feature.

In Practice

1. Add the permission to use AWS Session manager to AWS CodeBuild:
   

   CodeBuildServiceRole:
     Type: AWS::IAM::Role
     Properties:
       AssumeRolePolicyDocument:
         Statement:
           - Effect: Allow
             Principal:
               Service:
                 - codebuild.amazonaws.com
             Action:
               - sts:AssumeRole
       Path: /
       Policies:
         - PolicyName: log-access
           PolicyDocument:
             Statement:
               - Effect: Allow
                 Action:
                   - logs:CreateLogStream
                   - logs:PutLogEvents
                   - logs:CreateLogGroup
                 Resource:
                   - !Sub arn:aws:logs:${AWS::Region}:${AWS::AccountId}:log-group:/aws/codebuild/*
         - PolicyName: ssm-access
           PolicyDocument:
             Statement:
               - Effect: Allow
                 Action:
                   - ssmmessages:CreateControlChannel
                   - ssmmessages:CreateDataChannel
                   - ssmmessages:OpenControlChannel
                   - ssmmessages:OpenDataChannel
                 Resource: "*"
               - Effect: Allow
                 Action:
                   - s3:GetEncryptionConfiguration
                   - s3:PutObject
                 Resource:
                   - arn:aws:s3:::your-log-bucket-name
                   - arn:aws:s3:::your-log-bucket-name/*
   
         ...
   

2. Add a breakpoint to your buildspec.yml
   

   build:
     commands:
       - ...
       - codebuild-breakpoint
       - ...
   

3. Start a build for debugging

   • Start a build using "Advanced Build Overrides"
   • Under advanced settings choose "Enable Session Connection"

Start a remote session using the Web Console

Start a remote session using the CLI

1. Grab the Build ID (aka Build Run)

2. Get the sessionTarget using the Build Id
   

   aws codebuild batch-get-builds --ids <buildID> --region <region>
   

   IMHO, currently, the documentation for batch-get-builds falls short. Getting the sessionTarget using the CLI could be tricky if you don't use the right settings or if you're not using a recent version of the CLI. Therefore I made a Pull Request to change the documentation to:

   Copy the sessionTarget property value. Note: sessionTarget is only available if output is json or table. If output is set to text look for DEBUGSESSION instead. If the property is missing from the output then update your CLI to a more recent version.

3. Once you have copied the sessionTarget value you can start a new remote session using:
   

   aws ssm start-session --target <sessionTarget> --region <region>
   

4. Debug your build :-)

You're all set. To stop a debug session just execute $ codebuild-resume.

Remark: the CodeBuild policy mentioned above only needs S3 permissions if using Amazon S3 to store your logs. In case it's missing you get an SSM Session with a stuck prompt without any further feedback. So, if using S3 to store your logs be sure to have that policy right.

See also: View a running build in Session Manager

Until next time!

Comments

[Andrew Brown 🇨🇦]

I'm blown away by this feature. I'm currently starting at failing build runs and I'm lucky enough to see you found this obscure but very useful feature.