DEV Community: hayao-k

Effortlessly Export AWS Health Organizational View to CSV with This CLI Tool

hayao-k — Thu, 16 May 2024 15:49:59 +0000

💡 The tools discussed in this article leverage the AWS Health API, which requires a Business or higher-level AWS Support plan.

Introduction

For all AWS Organizations administrators, how do you handle the events notified by AWS Health? I imagine you receive numerous notifications daily when utilizing AWS services.

These events need to be properly managed as they can have a significant impact on the availability and reliability of your systems. Recently, there have been significant events scheduled, such as the Amazon RDS certificate update in August and the end of support for the AWS Lambda Python 3.8 runtime in October.

If you're managing multiple accounts within your organization, manually checking events and gathering relevant information can take time and effort.

I have developed a CLI tool called AWS Health Exporter to address this challenge.

hayao-k / aws-health-exporter

AWS Health Exporter is a command-line tool designed to describe AWS Health events for your organization.

AWS Health Exporter

Health Exporter is a command-line tool designed to describe AWS Health events for your organization. It allows you to filter events by service name and status, and export the details to a CSV file. Optionally, you can echo the CSV content to standard output.

Features

Event Filtering: Filter events by service name, event status, and other criteria to get precisely the data you need.
Entity Filtering: Filter affected entities by status code (IMPAIRED, UNIMPAIRED, UNKNOWN, PENDING, or RESOLVED).
AWS Organizations Support: Works seamlessly with AWS Organizations, allowing you to get a health overview of all accounts.
CSV Export: Automatically formats and exports the data into a CSV file, making it simple to store, share, and analyze.

Prerequisites

AWS credentials with appropriate permissions to access AWS Health and AWS Organizations services
You must have a Business, Enterprise On-Ramp, or Enterprise Support plan from AWS Support to use the…

View on GitHub

Key Features

AWS Health Exporter is a command-line tool for retrieving event information from the organizational view of AWS Health. It allows you to filter events by service name, status, and more and export details of the relevant accounts and resource IDs to a CSV file.

AWS Organizations Support: Retrieves information from the organizational view of AWS Health. It cannot be used with standalone accounts, but there is an option to output data for a single account only.
CSV Export: Data is formatted and exported in CSV format, making it easy to save, share, and analyze.
Event Filtering: Filters events by conditions such as service name and status, making it easier to find the events you're looking for.
Resource Filtering: Only retrieves resources matching specific status codes (IMPAIRED, UNIMPAIRED, UNKNOWN, PENDING, or RESOLVED).

About AWS Health Organizational View

Enabling the organizational view allows you to aggregate AWS Health events for all accounts within the organization. Data is retained for 90 days, and users/roles of the organization's management or delegated administrator accounts can access the information.

You can set it up and refer to it from "Your organization health" in the AWS Health dashboard.

In the organizational view, you can check information for each event, such as:

Affected accounts
Number of affected resources and breakdown of their statuses
Resources affected within each account

This tool can export all this information to a CSV file!

Prerequisites for using the tool

The organizational view of AWS Health is enabled.
AWS authentication credentials to access AWS Health and AWS Organizations
- Authentication credentials for the management or delegated administrator accounts are required to use the organizational view.
A business plan or higher-level AWS support contract
- Required to use the AWS Health API

How to Use

Download the latest binary suitable for your environment from the GitHub repository's releases page.

https://github.com/hayao-k/aws-health-exporter/releases

wget https://github.com/hayao-k/aws-health-exporter/releases/download/v0.8.1/aws-health-exporter_0.8.1_linux_amd64.tar.gz
tar xvf aws-health-exporter_0.8.1_linux_amd64.tar.gz

To use AWS Health Exporter, run the binary with the desired flags. Below are the available flags:

--event-filter, --filter, -f: Filter events by service name, event status, and other criteria.
--status-code, -c: Filter entity by status code. Possible values are IMPAIRED, UNIMPAIRED, UNKNOWN, PENDING and RESOLVED
--echo, -e: Echo CSV content to standard output.
--profile, -p: Specify the AWS credential profile to use.
--account-id, -i: Specify a single account ID to process (optional).
--output-file, --file-name, o: Specify the output CSV file name.

Details of the event filtering option

The --event-filter option allows you to specify complex filtering criteria. Below is a table of the available fields that can be included in the filter criteria:

Field	Description	Possible Values
`service`	Filter events by AWS service name.	e.g., `LAMBDA`, `RDS`, `EKS`
`status`	Filter events by status.	`open`, `closed`, `upcoming`
`category`	Filter events by category.	`issue`, `accountNotification`, `scheduledChange`, `investigation`
`region`	Filter events by region.	AWS region codes, e.g., `us-east-1`
`startTime`	Filter events by start time.	ISO 8601 date format
`endTime`	Filter events by end time.	ISO 8601 date format
`lastUpdatedTime`	Filter events by last updated time.	ISO 8601 date format

For startTime, endTime, and lastUpdatedTime, you can specify a time range using from and to in ISO 8601 date format. Here is the structure for determining the time range:

{from:YYYY-MM-DDTHH:MM:SSZ,to:YYYY-MM-DDTHH:MM:SSZ}

Example Commands

# Describe RDS events with open status and export to CSV
./health-exporter --event-filter service=RDS,status=open

# Describe upcoming LAMBDA events and echo the output to STDOUT
./health-exporter --event-filter service=LAMBDA,status=upcoming --echo

# Describe only events in the Tokyo region and specify their last updated time.
./health-exporter ----event-filter "lastUpdatedTime={from=2024-03-01T00:00:00Z,to=2024-05-02T23:59:59Z},region=ap-northeast-1"

# Get entities with pending status only and specify a custom file name
./health-exporter --status-code PENDING --output-file my_event_details.csv

# Get events using the specified profile
./health-exporter --profile my-profile

# Process only a single account
./health-exporter --account-id 123456789012

Execution Example

When you execute the command, an interactive prompt will be displayed. In the following example, the --event-filter flag extracts only the upcoming status events related to AWS Lambda.

$ health-exporter --event-filter service=LAMBDA,status=upcoming --status-code PENDING
Use the arrow keys to navigate: ↓ ↑ → ← 
? Select an event: 
  ▸ LAMBDA - AWS_LAMBDA_PLANNED_LIFECYCLE_EVENT (us-east-1, 2024-10-14 07:00:00)
    LAMBDA - AWS_LAMBDA_PLANNED_LIFECYCLE_EVENT (ap-northeast-1, 2024-10-14 07:00:00)
    LAMBDA - AWS_LAMBDA_PLANNED_LIFECYCLE_EVENT (ap-northeast-1, 2024-06-12 07:00:00)
    LAMBDA - AWS_LAMBDA_PLANNED_LIFECYCLE_EVENT (ap-southeast-2, 2024-10-14 07:00:00)
↓   LAMBDA - AWS_LAMBDA_PLANNED_LIFECYCLE_EVENT (us-east-1, 2024-06-12 07:00:00)

From the prompt, select the event you want to output. After selection, the tool will gather related account and entity information and output it to a CSV file.

✔ LAMBDA - AWS_LAMBDA_PLANNED_LIFECYCLE_EVENT (us-east-1, 2024-10-14 07:00:00)
Event details have been written to AWS_LAMBDA_PLANNED_LIFECYCLE_EVENT_2024-10-14_07-00-00_us-east-1_PENDING.csv.

The output CSV will contain information such as Account ID, Account Name, Region, Identifier, Status, and Last Updated. In this example, since --status-code PENDING was specified during command execution, only resources with PENDING status are output.

Account ID, Account Name, Region, Identifier, Status, Last Updated
000000000000,account-0000,us-east-1,arn:aws:lambda:us-east-1:000000000000:function:Old_Runtime_Lambda_Function-1PBKPZPFSJ058,PENDING,2024-04-21 20:11:29
111111111111,account-1111,us-east-1,arn:aws:lambda:us-east-1:111111111111:function:Old_Runtime_Lambda_Function-uuTi2u7DbooD,PENDING,2024-04-21 20:11:29
111111111111,account-1111,us-east-1,arn:aws:lambda:us-east-1:111111111111:function:Old_Runtime_Lambda_Function-omdieC8Umobo,PENDING,2024-04-21 20:11:29
222222222222,account-2222,us-east-1,arn:aws:lambda:us-east-1:222222222222:function:Old_Runtime_Lambda_Function-ULZ27BYSQ0MN,PENDING,2024-04-21 20:11:29
222222222222,account-2222,us-east-1,arn:aws:lambda:us-east-1:222222222222:function:Old_Runtime_Lambda_Function-10YNGBMU46VP9,PENDING,2024-04-21 20:11:29
222222222222,account-2222,us-east-1,arn:aws:lambda:us-east-1:222222222222:function:Old_Runtime_Lambda_Function-CEgHAu41udFy,PENDING,2024-04-21 20:11:29
333333333333,account-3333,us-east-1,arn:aws:lambda:us-east-1:333333333333:function:Old_Runtime_Lambda_Function-zNKRpLWP0pXB,PENDING,2024-04-21 20:11:29
333333333333,account-3333,us-east-1,arn:aws:lambda:us-east-1:333333333333:function:Old_Runtime_Lambda_Function-24ES8MRQJ9R6,PENDING,2024-04-21 20:11:29
444444444444,account-4444,us-east-1,arn:aws:lambda:us-east-1:444444444444:function:Old_Runtime_Lambda_Function-134QIS8IYF84K,PENDING,2024-04-21 20:11:29
444444444444,account-4444,us-east-1,arn:aws:lambda:us-east-1:444444444444:function:Old_Runtime_Lambda_Function-B97VeyrZNXIy,PENDING,2024-04-21 20:11:29

Mechanism

Primarily uses 3 AWS Health APIs.

DescribeEventsForOrganization API

Calls the DescribeEventsForOrganization API to retrieve relevant events based on the filter conditions specified on the command line. This API returns only an overview of the events, so information about affected accounts or resources is not included.

DescribeAffectedAccountsForOrganization API

This API retrieves a list of accounts within the organization affected by the selected event.

DescribeAffectedEntitiesForOrganization API

This API returns a list of entities affected by one or more events in one or more accounts within the organization.

When the user selects an event through the interactive prompt, information obtained from these APIs is formatted and output as a CSV file.

I hope this helps you.

AI Chatbot powered by Amazon Bedrock 🚀🤖

hayao-k — Fri, 29 Sep 2023 16:48:30 +0000

Introduction

I have created a sample chatbot application that uses Chainlit and LangChain to showcase Amazon Bedrock.

You can interact with the AI assistant while switching between multiple models.

This sample application has been tested in the following environments:

AWS Region: us-east-1
AWS Copilot CLI: v1.30.1
Python: v3.11.5
boto3: v1.28.57
LangChain: v0.0.305
Chainlit: v0.7.0

Note: To support the GA Version of the Bedrock API, make sure to use versions of LangChain and boto3 that are newer than those mentioned above.

Source Code

Check out my GitHub repository!

hayao-k / Bedrock-AIChatbot-Sample

AI Chatbot powered by Amazon Bedrock 🚀🤖

Overview

Sample chatbot application to experience Amazon Bedrock, using Chainlit and LangChain.

You can interact with the AI assistant while switching between multiple models.

Please ensure you enable access to each model in Amazon Bedrock beforehand.

Features

Customizable Chat Settings: Through Chainlit, it offers a user-friendly interface to select models and adjust settings before initializing the chat.
Model Selection: Amazon Bedrock model list is automatically retrieved and can be selected.
Document Chat: Please refer to the official documentation for the support status of each model.
Image Processing (Anthropic Models Only): Supports image input for Claude 3 models.

Prerequisites

Before deploying this application, ensure you have the following:

AWS Copilot CLI installed
Enable access to each model in Amazon Bedrock

Getting Started

Deploy to AWS App Runner using AWS Copilot CLI. To deploy this application:

git clone https://github.com/hayao-k/Bedrock-AIChatbot-Sample
cd Bedrock-AIChatbot-Sample

…

View on GitHub

Start using Bedrock

Before you can start using Amazon Bedrock, you must request access to each model. To add access to a model, go to the Model Access section in the Bedrock console and click Edit.

Select the models you want to use and save the settings.

Note: Third-party models such as Jurassic-2 and Claude require access through the AWS Marketplace. This means you will also be charged a Marketplace fee for using the model.

It will take some time for each model to become available; models with an Access status of Access granted are ready for use.

Note: Some models, such as Titan Text G1 - Express, are in Preview and may not be immediately available.

How to Deploy

The GitHub repository for the sample application contains a manifest file for deploying the container to AWS App Runner using the AWS Copilot CLI.

Follow these steps to deploy:

Install the AWS Copilot CLI if necessary



sudo curl -Lo /usr/local/bin/copilot https://github.com/aws/copilot-cli/releases/latest/download/copilot-linux && sudo chmod +x /usr/local/bin/copilot

Deploy to App Runner!



 export AWS_REGION=us-east-1
 copilot app init bedrockchat-app
 copilot deploy --name bedrockchat --env dev

If the deployment is successful, access the URL displayed in the message. If the following screen appears, the deployment has succeeded!

To delete an environment, execute the following command:



copilot app delete

How to use the app

The model, temperature, and maximum token size can be changed from the Settings panel.

The sample code allows you to choose between Claude v2, Jurassic-2 Ultra, and Amazon Titan Text G1 - Express. This enables you to compare performance while switching between models.

Note: As of 9/29/2023, Amazon Titan Text G1 - Express is in Preview status. It will be rolled out gradually, so it may not be available for some accounts.

You can enjoy a free-flowing conversation with the AI assistant! The conversation history during the session is retained, allowing it to reply to you while considering the context.

I hope this will be of help to someone else.

Running Amazon S3 Mountpoint Inside a Container

hayao-k — Wed, 22 Mar 2023 12:37:04 +0000

Introduction

Here is a simple example of running Mountpoint for Amazon S3 from inside a container

Created with information as of 3/21/2023 (version: 0.2.0-b8363a4)
Mountpoint for Amazon S3 is currently in alpha release and should not be used in production workloads

Container image is also available in ECR Public gallery.

Dockerfile



FROM rust:1.68.0 as Build

RUN apt-get update && apt-get install -y \
    clang\
    cmake \
    curl \
    fuse \
    git \
    libfuse-dev \
    pkg-config \
 && apt-get clean \
 && rm -rf /var/lib/apt/lists/* \
 && git clone --recurse-submodules https://github.com/awslabs/mountpoint-s3.git \
 && cd mountpoint-s3 \
 && cargo build --release


FROM debian:bullseye-slim
RUN apt-get update && apt-get install -y \
    ca-certificates \
    libfuse-dev \
    sudo \
 && apt-get clean \
 && rm -rf /var/lib/apt/lists/*

COPY --from=build /mountpoint-s3/target/release/mount-s3 /usr/local/bin/mount-s3

RUN chmod 777 /usr/local/bin/mount-s3

RUN useradd -ms /bin/bash mount-s3-user \
 && echo '%sudo ALL=(ALL) NOPASSWD:ALL' >> /etc/sudoers \
 && adduser mount-s3-user sudo

USER mount-s3-user

Getting started

Image Build



docker image build -t mount-s3:latest .

Run



docker container run --privileged --rm -it mount-s3:latest bash

Enjoy



mount-s3-user@ce43831fda04:~$ sudo mount-s3 <bucket_name> /mnt --allow-other --region ap-northeast-1
mount-s3-user@ce43831fda04:~$ ls -l /mnt/test.json
-rw-r--r-- 1 root root 306424 Feb 21 02:42 /mnt/test.json

EC2 on Docker Consideration

If using EC2 IAM roles for AWS credentials, increasing the IMDSv2 hop limit from 1 to 2 in the instance metadata options is recommended.

In a container environment, if the hop limit is 1, the IMDSv2 response does not return because going to the container is considered an additional network hop. To avoid the process of falling back to IMDSv1 and the resultant delay, in a container environment we recommend that you set the hop limit to 2



aws ec2 modify-instance-metadata-options \
    --instance-id i-xxxxxxxxxxxxxxxxx \
    --http-put-response-hop-limit 2 \
    --http-endpoint enabled

I hope this will be of help to someone else.

GitHub Repo

hayao-k / container-mountpoint-s3

container-mountpoint-s3

This is a simple container image to verify the operation of Mountpoint for Amazon S3 in a container environment.

Container image is also available in ECR Public gallery.

Getting started

Image Build

docker image build -t mount-s3:latest .

Run

docker container run --privileged --rm -it mount-s3:latest bash

Enjoy

mount-s3-user@ce43831fda04:~$ sudo mount-s3 <bucket_name> /mnt --allow-other --region ap-northeast-1
mount-s3-user@ce43831fda04:~$ ls -l /mnt/test.json
-rw-r--r-- 1 root root 306424 Feb 21 02:42 /mnt/test.json

EC2 on Docker Consideration

If using EC2 IAM roles for AWS credentials, increasing the IMDSv2 hop limit from 1 to 2 in the instance metadata options is recommended.

In a container environment, if the hop limit is 1, the IMDSv2 response does not return because going to the container is considered an additional network hop. To avoid the process of falling back to IMDSv1 and the resultant delay, in a container environment we recommend that you set the…

View on GitHub

Creating AWS resources for GitHub audit log streaming with CloudFormation

hayao-k — Fri, 17 Mar 2023 13:11:24 +0000

Introduction

GitHub Enterprise Cloud audit logs support log streaming to various cloud providers.

https://docs.github.com/en/enterprise-cloud@latest/admin/monitoring-activity-in-your-enterprise/reviewing-audit-logs-for-your-enterprise/streaming-the-audit-log-for-your-enterprise

Streaming audit logs to Amazon S3 can be done via OpenID Connect. This requires the creation of an OIDC Provider and IAM role on the AWS side and an S3 bucket for log storage.

Please refer to the above document for detailed instructions.

It is not something that is created many times, but I have created a CloudFormation template and hope it will be helpful for those who want to create a new one quickly and easily.

Resources to be created

Logical ID	Type	Description
AuditLogBucket	AWS::S3::Bucket	S3 bucket for audit log
AuditLogbucketPolicy	AWS::S3::BucketPolicy	S3 bucket policy for audit log
AccessLogBucket	AWS::S3::Bucket	S3 bucket for server access log
AccessLogBucketPolicy	AWS::S3::BucketPolicy	S3 bucket policy for server access log
AuditLogEncryptionKey	AWS::KMS::Key	Customer Managed Key for audit log encryption
AuditLogEncryptionKeyAlias	AWS::KMS::Alias	Customer Managed Key alias
AuditLogStremingPolicy	AWS::IAM::ManagedPolicy	IAM policy for audit log streaming
AuditLogStreamingRole	AWS::IAM::Role	IAM Role for audit log streaming
GithubAuditLogOIDCProvider	AWS::IAM::OIDCProvider	GitHub OIDC provider

CloudFormation template

Github repository is here

hayao-k / CFn-github-audit-log-streaming

CFn-github-audit-log-streaming

This CloudFormation template creates the AWS resources required for GitHub Enterprise Cloud audit log streaming.

View on GitHub

AWSTemplateFormatVersion: "2010-09-09"
Description: Create AWS resources required for GitHub Enterprise Cloud audit log streaming

Parameters:
  RetainDays:
    Description: Number of days to keep audit log.
    Type: String
    Default: 365
  EnterpriseName:
    Description: Your GitHub Enterprise Name.
    Type: String
    Default: EXAPMLECORP
  AccessLogPrefix:
    Description: Server access log-prefix of audit log bucket.
    Type: String
    Default: logs/

Metadata: 
  AWS::CloudFormation::Interface: 
    ParameterGroups: 
      - Label: 
          default: GitHub Configuration
        Parameters: 
          - EnterpriseName
      - Label: 
          default: Audit Log Configuration
        Parameters: 
          - RetainDays
          - AccessLogPrefix

Resources:
  # Customer Managed Key for audit log encryption
  AuditLogEncryptionKey:
    Type: AWS::KMS::Key
    Properties: 
      Description: Customer Managed Key for audit log encryption
      Enabled: true
      EnableKeyRotation: True
      KeyPolicy:
        Version: '2012-10-17'
        Id: key-default-1
        Statement:
        - Sid: Allow administration of the key
          Effect: Allow
          Principal:
            AWS: !Sub arn:aws:iam::${AWS::AccountId}:root
          Action:
          - kms:Create*
          - kms:Describe*
          - kms:Enable*
          - kms:List*
          - kms:Put*
          - kms:Update*
          - kms:Revoke*
          - kms:Disable*
          - kms:Get*
          - kms:Delete*
          - kms:ScheduleKeyDeletion
          - kms:CancelKeyDeletion
          Resource: '*'
        - Sid: Allow local use of the key
          Effect: Allow
          Principal:
            AWS: !Sub arn:aws:iam::${AWS::AccountId}:root
          Action:
          - kms:Encrypt
          - kms:Decrypt
          - kms:ReEncrypt*
          - kms:GenerateDataKey*
          - kms:DescribeKey
          Resource: '*'

  # Customer Managed Key ALias
  AuditLogEncryptionKeyAlias:
    Type: AWS::KMS::Alias
    Properties: 
      AliasName: !Sub alias/${AWS::StackName}-key
      TargetKeyId: !Ref AuditLogEncryptionKey

  # S3 bucket for audit log
  AuditLogBucket:
    Type: AWS::S3::Bucket
    DeletionPolicy: Retain
    UpdateReplacePolicy: Retain
    Properties:
      BucketEncryption:
        ServerSideEncryptionConfiguration: 
          - ServerSideEncryptionByDefault: 
              SSEAlgorithm: aws:kms
              KMSMasterKeyID: !GetAtt AuditLogEncryptionKey.Arn
            BucketKeyEnabled: true    
      LifecycleConfiguration:
        Rules:
          - Id: Retain
            Status: Enabled
            ExpirationInDays: !Ref RetainDays
      PublicAccessBlockConfiguration: 
        BlockPublicAcls: true
        BlockPublicPolicy: true
        IgnorePublicAcls: true
        RestrictPublicBuckets: true
      LoggingConfiguration:
        DestinationBucketName: !Ref AccessLogBucket
        LogFilePrefix: !Ref AccessLogPrefix
      VersioningConfiguration:
        Status: Enabled
      OwnershipControls:
        Rules:
          - ObjectOwnership: BucketOwnerEnforced

  # S3 bucket policy for audit log
  AudtiLogBucketPolicy:
    Type: AWS::S3::BucketPolicy
    Properties:
      Bucket: !Ref AuditLogBucket
      PolicyDocument:
        Statement:
        - Effect: Deny
          Principal: '*'
          Action: 's3:*'
          Resource: 
            - !Sub ${AuditLogBucket.Arn}/*
            - !GetAtt AuditLogBucket.Arn
          Condition:
            Bool: 
              aws:SecureTransport: false

  # S3 bucket for server access log
  AccessLogBucket:
    Type: AWS::S3::Bucket
    DeletionPolicy: Retain
    UpdateReplacePolicy: Retain
    Properties:
      BucketEncryption:
        ServerSideEncryptionConfiguration: 
          - ServerSideEncryptionByDefault: 
              SSEAlgorithm: AES256
      LifecycleConfiguration:
        Rules:
          - Id: Retain
            Status: Enabled
            ExpirationInDays: !Ref RetainDays
      PublicAccessBlockConfiguration: 
        BlockPublicAcls: true
        BlockPublicPolicy: true
        IgnorePublicAcls: true
        RestrictPublicBuckets: true
      VersioningConfiguration:
        Status: Enabled
      OwnershipControls:
        Rules:
          - ObjectOwnership: BucketOwnerEnforced

  # S3 bucket policy for server access log
  AccessLogBucketPolicy:
    Type: AWS::S3::BucketPolicy
    Properties:
      Bucket: !Ref AccessLogBucket
      PolicyDocument:
        Statement:
          - Effect: Deny
            Principal: '*'
            Action: 's3:*'
            Resource: 
              - !Sub ${AccessLogBucket.Arn}/*
              - !GetAtt AccessLogBucket.Arn
            Condition:
              Bool: 
                aws:SecureTransport: false
          - Effect: Allow
            Principal:
              Service: logging.s3.amazonaws.com
            Action: s3:PutObject
            Resource:
              - !Sub ${AccessLogBucket.Arn}/${AccessLogPrefix}*
            Condition:
              ArnLike:
                aws:SourceArn: !GetAtt AuditLogBucket.Arn
              StringEquals: 
                aws:SourceAccount: !Ref AWS::AccountId

  # IAM policy for audit log streaming role
  AuditLogStremingPolicy:
    Type: AWS::IAM::ManagedPolicy
    Properties:
      Description: Policy for GitHub Audit Log Streaming
      PolicyDocument:
        Version: 2012-10-17
        Statement:
          - Effect: Allow
            Action:
              - s3:PutObject
            Resource:
              - !Sub ${AuditLogBucket.Arn}/*

  # IAM Role for audit log streaming role
  AuditLogStreamingRole:
    Type: AWS::IAM::Role
    Properties:
      ManagedPolicyArns: 
        - !Ref AuditLogStremingPolicy
      AssumeRolePolicyDocument:
        Statement:
          - Effect: Allow
            Action: sts:AssumeRoleWithWebIdentity
            Principal:
              Federated: !Ref GithubAuditLogOIDCProvider
            Condition:
              StringEquals:
                oidc-configuration.audit-log.githubusercontent.com:sub: !Sub https://github.com/${EnterpriseName}
                oidc-configuration.audit-log.githubusercontent.com:aud: sts.amazonaws.com

  GithubAuditLogOIDCProvider:
    Type: AWS::IAM::OIDCProvider
    Properties:
      Url: https://oidc-configuration.audit-log.githubusercontent.com
      ClientIdList: 
        - sts.amazonaws.com
      ThumbprintList:
        - 7e6db7b7584d8cf2003e0931e6cfc41a3a62d3df

Outputs:
  IAMRole:
    Description: IAM role for audit log streaming role
    Value: !GetAtt AuditLogStreamingRole.Arn
  AuditLogBucket:
    Description: S3 bucket for audit log
    Value: !Ref AuditLogBucket

Deploy

3 parameters are required.

GitHub Enterprise Name (EnterpriseName)
Number of days to keep audit log (RetainDays)
Server access log-prefix of audit log bucket (AccessLogPrefix)

After deploying the stack, set the output S3 bucket name and IAM Role ARN to GitHub Enterprise Cloud.

If the endpoint connection check is successful, save the configuration.

You can see the audit log in the AuditLogBucket!

I hope this will be of help to someone else.

Automate AWS Enterprise Support Activation for Member Accounts

hayao-k — Mon, 13 Mar 2023 07:16:45 +0000

Why do we need automation?

Member accounts added to AWS Organizations after subscribing to Enterprise Support are not enrolled in Enterprise Support.

To register a new member account with Enterprise Support, you must open a support case in the management account.

Example

This example is a simple workflow that executes case creation, close confirmation, and notification with AWS Step Functions.

Input

You can use Amazon EventBrige to trigger the CreateAccount event in AWS Organizations or the CreateManagedAccount event in AWS Control Tower to launch the state machine.

Therefore, the input for the state machine is a single account as follows.

{
  "AccountId": "<Account ID>"
}

State machine definition

The process flow is as follows. Creating cases and checking status uses AWS SDK service integrations.

In the CeateCase state, create a support case with the account ID received from the state machine startup input as the activation target
Execute the DescribeCases state based on the Case ID returned as the result of the CreateCase state task.
- DescribeCases API requires passing a list of case IDs, so use the built-in function States.Array.
- Since the task result is also a list, specify $.Cases[0] in OutputPath.
In the Choice state, check the status of support cases from the DescirbeCases output
- If resolved, proceed to SNS Pulibsh state.
- Otherwise, wait for a specified time in the Wait state and execute DescribeCases again.
In the SNS Publish state, publish a message to the specified SNS Topic

{
  "Comment": "A description of your state machine",
  "StartAt": "CreateCase",
  "States": {
    "CreateCase": {
      "Type": "Task",
      "Parameters": {
        "Subject": "Enterprise Activation Request for Linked account",
        "ServiceCode": "customer-account",
        "SeverityCode": "low",
        "CategoryCode": "other-account-issues",
        "CommunicationBody.$": "States.Format('Please enable Enterprise support for following account ID:\n{}\n', $.AccountId)",
        "Language": "en",
        "IssueType": "customer-service"
      },
      "Resource": "arn:aws:states:::aws-sdk:support:createCase",
      "Next": "DescribeCases"
    },
    "DescribeCases": {
      "Type": "Task",
      "Parameters": {
        "CaseIdList.$": "States.Array($.CaseId)",
        "IncludeResolvedCases": true
      },
      "Resource": "arn:aws:states:::aws-sdk:support:describeCases",
      "Next": "Choice",
      "OutputPath": "$.Cases[0]"
    },
    "Choice": {
      "Type": "Choice",
      "Choices": [
        {
          "Variable": "$.Status",
          "StringEquals": "resolved",
          "Next": "SNS Publish"
        }
      ],
      "Default": "Wait"
    },
    "SNS Publish": {
      "Type": "Task",
      "Resource": "arn:aws:states:::sns:publish",
      "Parameters": {
        "Message.$": "$",
        "TopicArn": "arn:aws:sns:us-east-1:123456789012:your-sns-topic"
      },
      "End": true
    },
    "Wait": {
      "Type": "Wait",
      "Seconds": 30,
      "Next": "DescribeCases"
    }
  }
}

On Workflow Studio, it is displayed as follows.

Execution example

I hope this will be of help to someone else.

How to restore EC2 instance with multiple ENIs attached from AWS Backup

hayao-k — Thu, 12 Jan 2023 03:27:19 +0000

Goal of this post

Backup EC2 instances with multiple ENIs attached with AWS Backup
Restore EC2 instances with multiple ENIs attached, when restored from a recovery point

How?

Run the StartRestoreJob API, e.g., from the AWS CLI or SDK
Restore jobs launched from the console cannot customize the network i\nterface

EC2 instance backup with multiple ENIs attached

EC2 instances with multiple ENIs attached can also be backed up with AWS Backup. Backup data is stored as AMI, but AMI does not contain network interface information.

However, the metadata of the recovery point includes the network interface information. Recovery point metadata can be checked with the GetRecoveryPointRestoreMetadata API.

The following is an example of execution with the AWS CLI.

$ aws backup get-recovery-point-restore-metadata --backup-vault-name Default --recovery-point-arn arn:aws:ec2:us-west-2::image/ami-xxxxxxxxxxxxxxxxx
{
    "BackupVaultArn": "arn:aws:backup:us-west-2:123456789012:backup-vault:Default",
    "RecoveryPointArn": "arn:aws:ec2:us-west-2::image/ami-xxxxxxxxxxxxxxxxx",
    "RestoreMetadata": {
        "CapacityReservationSpecification": "{\"CapacityReservationPreference\":\"open\"}",
        "CpuOptions": "{\"CoreCount\":2,\"ThreadsPerCore\":1}",
        "CreditSpecification": "{\"CpuCredits\":\"unlimited\"}",
        "DisableApiTermination": "false",
        "EbsOptimized": "true",
        "HibernationOptions": "{\"Configured\":false}",
        "InstanceInitiatedShutdownBehavior": "stop",
        "InstanceType": "t4g.micro",
        "Monitoring": "{\"State\":\"disabled\"}",
        "NetworkInterfaces": "[{\"AssociatePublicIpAddress\":true,\"DeleteOnTermination\":true,\"Description\":\"\",\"DeviceIndex\":0,\"Groups\":[\"sg-xxxxxxxxxxxxxxxxx\"],\"Ipv6AddressCount\":0,\"Ipv6Addresses\":[],\"NetworkInterfaceId\":\"eni-aaaaaaaaaaaaaaaaa\",\"PrivateIpAddress\":\"172.31.62.169\",\"PrivateIpAddresses\":[{\"Primary\":true,\"PrivateIpAddress\":\"172.31.62.169\"}],\"SecondaryPrivateIpAddressCount\":0,\"SubnetId\":\"subnet-xxxxxxxxxxxxxxxxx\",\"InterfaceType\":\"interface\",\"Ipv4Prefixes\":[],\"Ipv6Prefixes\":[]},{\"AssociatePublicIpAddress\":true,\"DeleteOnTermination\":false,\"Description\":\"\",\"DeviceIndex\":1,\"Groups\":[\"sg-xxxxxxxxxxxxxxxxx\"],\"Ipv6AddressCount\":0,\"Ipv6Addresses\":[],\"NetworkInterfaceId\":\"eni-bbbbbbbbbbbbbbbbb\",\"PrivateIpAddress\":\"172.31.54.130\",\"PrivateIpAddresses\":[{\"Primary\":true,\"PrivateIpAddress\":\"172.31.54.130\"}],\"SecondaryPrivateIpAddressCount\":0,\"SubnetId\":\"subnet-xxxxxxxxxxxxxxxxx\",\"InterfaceType\":\"interface\",\"Ipv4Prefixes\":[],\"Ipv6Prefixes\":[]}]",
        "Placement": "{\"AvailabilityZone\":\"us-west-2d\",\"GroupName\":\"\",\"Tenancy\":\"default\"}",
        "RequireIMDSv2": "false",
        "SecurityGroupIds": "[\"sg-xxxxxxxxxxxxxxxxx\"]",
        "SubnetId": "subnet-xxxxxxxxxxxxxxxxx",
        "VpcId": "vpc-xxxxxxxxxxxxxxxxx",
        "aws:backup:request-id": "xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx"
    }
}

In the above, you can see that eni-aaaaaaaaaaaaaaaaa and eni-bbbbbbbbbbbbbbbbb information is included.

How to restore from a recovery point

When launching a restore job in the AWS Backup console, it is not possible to restore an EC2 instance with multiple ENIs attached. This is because the console limits the customizable parameters to the following.

https://docs.aws.amazon.com/aws-backup/latest/devguide/restoring-ec2.html

The AWS Backup console allows you to restore Amazon EC2 recovery points with the following parameters and settings you can customize:

Instance type

Amazon VPC

Subnet

Security groups

IAM role

Shutdown behavior

Stop–hibernate behavior

Termination protection

T2/T3 unlimited

Placement group name

EBS-optimized instance

Tenancy

RAM disk ID

Kernel ID

User data

Deletion on termination

To restore an EC2 instance with other customized parameters, including the network interface, you must execute the StartRestoreJob API with metadata, e.g., from the AWS CLI or SDK.

Use the AWS Backup API, CLI, or SDK to restore Amazon EC2 recovery points
Use StartRestoreJob. This option allows you to restore all 38 parameters, including the 22 parameters that are not customizable on the console.

The following is an example of execution with the AWS CLI.

$ aws backup start-restore-job \
  --recovery-point-arn "arn:aws:ec2:us-west-2::image/ami-xxxxxxxxxxxxxxxxx" \
  --iam-role-arn "arn:aws:iam::123456789012:role/service-role/AWSBackupDefaultServiceRole" \
  --metadata file://metadata.json  
{
    "RestoreJobId": "XXXXXXXX-XXXX-XXXX-XXXX-XXXXXXXXXXXX"
}

You can specify parameters in metadata.json as follows.

Example of specifying a private IP address

Please note that if a backup source instance exists, the private IP address must be changed to avoid duplicate addresses.

{
    "VpcId": "vpc-xxxxxxxxxxxxxxxxx",
    "Monitoring": "{\"State\":\"disabled\"}",
    "CapacityReservationSpecification": "{\"CapacityReservationPreference\":\"open\"}",
    "InstanceInitiatedShutdownBehavior": "stop",
    "DisableApiTermination": "false",
    "CreditSpecification": "{\"CpuCredits\":\"unlimited\"}",
    "HibernationOptions": "{\"Configured\":false}",
    "EbsOptimized": "true",
    "Placement": "{\"AvailabilityZone\":\"us-west-2d\",\"GroupName\":\"\",\"Tenancy\":\"default\"}",
    "aws:backup:request-id": "xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx",
    "InstanceType": "t4g.micro",
    "NetworkInterfaces": "[{\"DeleteOnTermination\":true,\"Description\":\"\",\"DeviceIndex\":0,\"Groups\":[\"sg-xxxxxxxxxxxxxxxxx\"],\"Ipv6AddressCount\":0,\"Ipv6Addresses\":[],\"PrivateIpAddresses\":[{\"Primary\":true,\"PrivateIpAddress\":\"172.31.62.169\"}],\"SubnetId\":\"subnet-xxxxxxxxxxxxxxxxx\",\"InterfaceType\":\"interface\"},{\"DeleteOnTermination\":false,\"Description\":\"\",\"DeviceIndex\":1,\"Groups\":[\"sg-xxxxxxxxxxxxxxxxx\"],\"Ipv6AddressCount\":0,\"Ipv6Addresses\":[],\"PrivateIpAddresses\":[{\"Primary\":true,\"PrivateIpAddress\":\"172.31.54.130\"}],\"SubnetId\":\"subnet-xxxxxxxxxxxxxxxxx\",\"InterfaceType\":\"interface\"}]"
}

Example of specifying ENI-ID

Please note that the ENI must be detached from the instance and in Available status if the backup source ENI is to be used.

{
    "VpcId": "vpc-xxxxxxxxxxxxxxxxx",
    "Monitoring": "{\"State\":\"disabled\"}",
    "CapacityReservationSpecification": "{\"CapacityReservationPreference\":\"open\"}",
    "InstanceInitiatedShutdownBehavior": "stop",
    "DisableApiTermination": "false",
    "CreditSpecification": "{\"CpuCredits\":\"unlimited\"}",
    "HibernationOptions": "{\"Configured\":false}",
    "EbsOptimized": "true",
    "Placement": "{\"AvailabilityZone\":\"us-west-2d\",\"GroupName\":\"\",\"Tenancy\":\"default\"}",
    "aws:backup:request-id": "xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx",
    "InstanceType": "t4g.micro",
    "NetworkInterfaces": "[{\"DeleteOnTermination\":true,\"Description\":\"\",\"DeviceIndex\":0,\"Groups\":[\"sg-xxxxxxxxxxxxxxxxx\"],\"Ipv6AddressCount\":0,\"Ipv6Addresses\":[],\"PrivateIpAddresses\":[{\"Primary\":true,\"PrivateIpAddress\":\"172.31.62.169\"}],\"SubnetId\":\"subnet-xxxxxxxxxxxxxxxxx\",\"InterfaceType\":\"interface\"},{\"DeleteOnTermination\":false,\"Description\":\"\",\"DeviceIndex\":1,\"Ipv6AddressCount\":0,\"Ipv6Addresses\":[],\"NetworkInterfaceId\":\"eni-bbbbbbbbbbbbbbbbb\"}]"
}

Reference

https://aws.amazon.com/premiumsupport/knowledge-center/aws-backup-ec2-restore-cli/

I hope this will be of help to someone else.

How to use AWS Parameters and Secrets Lambda Extension

hayao-k — Wed, 19 Oct 2022 08:59:10 +0000

What is AWS Parameters and Secrets Lambda Extension?

https://aws.amazon.com/jp/about-aws/whats-new/2022/10/aws-parameters-secrets-lambda-extension/

This extension can be used to retrieve parameters from the AWS Systems Manager Parameter Store and secrets from the AWS Secrets Manager.

What makes you happy?

Until now, parameters and secrets were obtained in the Lambda function process using the AWS SDK or other means.

With this extension, these values can be cached and reused during the lifecycle of a Lambda function. This reduces the latency and cost of retrieving parameters and secrets.

Basic usage

Please refer to each documents for details.

https://docs.aws.amazon.com/systems-manager/latest/userguide/ps-integration-lambda-extensions.html

https://docs.aws.amazon.com/secretsmanager/latest/userguide/retrieving-secrets_lambda.html

Set Layer of the extension to Lambda function

Lambda Extension is made available by configuring Lambda Layers. In Managed Console, AWS Parameters and Secrets Lambda Extension could be selected in the AWS layer.

When configuring from the CLI or other means, specify the ARN of the published Layer. A list of ARNs for each region is provided in the documentation.

Write HTTP GET code in the function

Using this Extension eliminates processing in the AWS SDK, but the code to retrieve the value with an HTTP GET request is still required. See the second half of this post for the sample code.

Change IAM policy for execution role

The extension uses the credentials of the IAM role used to execute the Lambda function itself. Therefore, an appropriate IAM policy must be set up to retrieve parameters and secrets. For example, for the Parameter Store, ssm:GetParameter and kms:Decrypt (when using SecureString) are required.

(Optional) Set environment variables for functions

TTL for the cache, log level, etc., can be controlled by setting environment variables for the Lambda function.

Sample Code

This is an example of referencing Amazon Linux 2 AMI public parameters.

Notes are as follows.

/ in the parameter name must be encoded
The extension's local HTTP server port starts at default 2773
- It can be changed via the environment variable PARAMETERS_SECRETS_EXTENSION_HTTP_PORT
Header 'X-Aws-Parameters-Secrets-Token' with AWS_SESSION_TOKEN environment variable must be added
- If not specified, it will be 401 unauthorized.

const https = require('http');

exports.handler = function(event, context, callback) {

    const options = {
        hostname: 'localhost',
        port: 2773,
        path: '/systemsmanager/parameters/get/?name=%2Faws%2Fservice%2Fami-amazon-linux-latest%2Famzn-ami-hvm-x86_64-gp2',
        headers: {
            'X-Aws-Parameters-Secrets-Token': process.env.AWS_SESSION_TOKEN
        },
        method: 'GET'
    };

    const req = https.request(options, res => {
        res.on('data', d => {
            console.log("Response from cache: "+d);
            return d;
        });
    });

    req.on('error', error => {
        console.error(error);
    });

    req.end();
};

The log of the execution result looks like this You got the parameter values!

[AWS Parameters and Secrets Lambda Extension] 2022/10/19 06:51:08 PARAMETERS_SECRETS_EXTENSION_LOG_LEVEL is not present. Log level set to info.
[AWS Parameters and Secrets Lambda Extension] 2022/10/19 06:51:08 INFO Systems Manager Parameter Store and Secrets Manager Lambda Extension 1.0.94
[AWS Parameters and Secrets Lambda Extension] 2022/10/19 06:51:08 INFO Serving on port 2773
EXTENSION   Name: AWSParametersAndSecretsLambdaExtension    State: Ready    Events: [INVOKE,SHUTDOWN]
START RequestId: bb5bcc53-38cc-42d7-9dc5-xxxxxxxxxxxx Version: $LATEST
[AWS Parameters and Secrets Lambda Extension] 2022/10/19 06:51:08 INFO ready to serve traffic
2022-10-19T06:51:09.247Z    bb5bcc53-38cc-42d7-9dc5-xxxxxxxxxxxx    INFO    Response from cache: {"Parameter":{"ARN":"arn:aws:ssm:ap-northeast-1::parameter/aws/service/ami-amazon-linux-latest/amzn-ami-hvm-x86_64-gp2","DataType":"text","LastModifiedDate":"2022-10-04T17:56:51.889Z","Name":"/aws/service/ami-amazon-linux-latest/amzn-ami-hvm-x86_64-gp2","Selector":null,"SourceResult":null,"Type":"String","Value":"ami-0fb16641312307fa9","Version":49},"ResultMetadata":{}}
END RequestId: bb5bcc53-38cc-42d7-9dc5-xxxxxxxxxxxx
REPORT RequestId: bb5bcc53-38cc-42d7-9dc5-xxxxxxxxxxxx  Duration: 796.05 ms Billed Duration: 797 ms Memory Size: 128 MB Max Memory Used: 76 MB  Init Duration: 324.74 ms

I hope this will be of help to someone else.

Backup AWS Support case history to your Wiki

hayao-k — Tue, 27 Sep 2022 03:07:59 +0000

Why do you need a backup?

AWS Support case histories are retained for up to 12 months.

https://aws.amazon.com/premiumsupport/faqs/

Q: How long is case history retained?

Case history information is available for 12 months after creation.

Therefore, if you want to store the answers as knowledge in the long term, you need a backup.

This post summarises how to convert support case answers into markdown format and post them to your Wiki via the API.

Architecture

Detecting support case closures with EventBridge event rules
Use AWS Support API in AWS Lambda to get case details
Formatted into Markdown format and posted to your Wiki

Note: Using AWS Support API requires a subscription to one of the following support plans: Business, Enterprise On-Ramp, or Enterprise.

This post is about GROWI, the OSS wiki, but it can be applied to any software or service that can post via API.

https://growi.org/en/

Points for Implementation

EventBridge event rule

Set to detect events where event-name is ResolveCase.

{
  "source": ["aws.support"],
  "detail-type": ["Support Case Update"],
  "detail": {
    "event-name": ["ResolveCase"]
  }
}

See documentation for examples of events.

https://docs.aws.amazon.com/awssupport/latest/user/event-bridge-support.html#example-event-logs-for-support-cases

{
    "version": "0",
    "id": "1aa4458d-556f-732e-ddc1-4a5b2fbd14a5",
    "detail-type": "Support Case Update",
    "source": "aws.support",
    "account": "111122223333",
    "time": "2022-02-21T15:51:31Z",
    "region": "us-east-1",
    "resources": [],
    "detail": {
        "case-id": "case-111122223333-muen-2022-7118885805350839",
        "display-id": "1234563851",
        "communication-id": "",
        "event-name": "ResolveCase",
        "origin": ""
    }
}

Getting support case information

The event passed from EventBrige contains a Case ID, so use DescribeCases API to get the details. Communications is obtained separately, so includeCommunications should be False.

def describe_case(case_id):
    response = support.describe_cases(
        caseIdList=[
            case_id
        ],
        includeResolvedCases=True,
        includeCommunications=False
    )
    return response['cases'][0]

def lambda_handler(event, context):
    case_info = describe_case(event['detail']['case-id'])

Cases excluded from backup

Increased service quotas and Enterprise support activation would not require backup.

def lambda_handler(event, context):
    case_info = describe_case(event['detail']['case-id'])

    if case_info['serviceCode'] == "service-limit-increase" or \
       case_info['subject'] == "Enterprise Activation Request for Linked account.":
        return {
            'Result': 'Service limit increase or Enterprise support activation will not be posted.'
        }

Get communication history with support

Use DescribeCommunications API. The data is retrieved from the most recent communication, so the order is reordered and concatenated.

Certain symbols of three or more consecutive characters are displayed as headers or horizontal lines in Markdown notation. They are replaced to avoid involuntary conversions.

def describe_communications(case_id):
    body = ""
    paginator = support.get_paginator('describe_communications')
    page_iterator = paginator.paginate(caseId=case_id)

    for page in page_iterator:
        for communication in page['communications']:
            body = re.sub(
                r'-{3,}|={3,}|\*{3,}|_{3,}', "...", communication['body']
            ) + '\n\n---\n\n' + body

    communications = '## Communications\n' + body
    return communications

Post to GROWI

The API reference is below.

https://docs.growi.org/en/api/rest-v3.html

New pages are created using createPage (POST /pages).

{
  "body": "string",
  "path": "/",
  "grant": 1,
  "grantUserGroupId": "5ae5fccfc5577b0004dbd8ab",
  "pageTags": [
    {
      "_id": "5e2d6aede35da4004ef7e0b7",
      "name": "daily",
      "count": 3
    }
  ],
  "createFromPageTree": true
}

Only body (the body of the article) and path (the path to the post) are required.
The grant specifies the extent to which the page is public (1: public, 2: only people who know the link, etc.).
In practice, you should also include the access_token (api_key).

def create_payload(account_id, case_info):
    token = os.environ['API_KEY']
    title = '# ' + case_info['subject'] + '\n'
    information = '## Case Information\n' + \
        '* Account ID: ' + account_id + '\n' + \
        '* Case ID: ' + case_info['displayId'] + '\n' +\
        '* Create Date ' + case_info['timeCreated'] + '\n' + \
        '* Severity: ' + case_info['severityCode'] + '\n' + \
        '* Service: ' + case_info['serviceCode'] + '\n' + \
        '* Category: ' + case_info['categoryCode'] + '\n'
    communications = describe_communications(case_info['caseId'])
    return {
        'access_token': token,
        'path': '/PathYouWantToPost/' + case_info['subject'],
        'body': title + information + communications,
        'grant': 1,
    }

def lambda_handler(event, context):
    case_info = describe_case(event['detail']['case-id'])
    payload = create_payload(event['account'], case_info)
    url = os.environ['API_URL']
    headers = {
        'Content-Type': 'application/json',
    }
    req = Request(url, json.dumps(payload).encode('utf-8'), headers)

Lambda function example

The function execution role must be granted AWS Support referencing rights.

from logging import getLogger, INFO
import json
import os
from urllib.request import Request, urlopen
from urllib.error import URLError, HTTPError
import re
from botocore.exceptions import ClientError
import boto3

logger = getLogger()
logger.setLevel(INFO)

support = boto3.client('support')

def describe_communications(case_id):
    body = ''
    try:
        paginator = support.get_paginator('describe_communications')
        page_iterator = paginator.paginate(caseId=case_id)
    except ClientError as err:
        logger.error(err.response['Error']['Message'])
        raise

    for page in page_iterator:
        for communication in page['communications']:
            body = re.sub(
                r'-{3,}|={3,}|\*{3,}|_{3,}', "...", communication['body']
            ) + '\n\n---\n\n' + body

    communications = '## Communications\n' + body
    return communications

def create_payload(account_id, case_info):
    token = os.environ['API_KEY']
    title = '# ' + case_info['subject'] + '\n'
    information = '## Case Information\n' + \
        '* Account ID: ' + account_id + '\n' + \
        '* Case ID: ' + case_info['displayId'] + '\n' +\
        '* Create Date ' + case_info['timeCreated'] + '\n' + \
        '* Severity: ' + case_info['severityCode'] + '\n' + \
        '* Service: ' + case_info['serviceCode'] + '\n' + \
        '* Category: ' + case_info['categoryCode'] + '\n'
    communications = describe_communications(case_info['caseId'])
    return {
        'access_token': token,
        'path': '/PathYouWantToPost/' + case_info['subject'],
        'body': title + information + communications,
        'grant': 1,
    }

def describe_case(case_id):
    try:
        response = support.describe_cases(
            caseIdList=[
                case_id
            ],
            includeResolvedCases=True,
            includeCommunications=False
        )
    except ClientError as err:
        logger.error(err.response['Error']['Message'])
        raise
    else:
        return response['cases'][0]

def lambda_handler(event, context):
    case_info = describe_case(event['detail']['case-id'])

    if case_info['serviceCode'] == "service-limit-increase" or \
       case_info['subject'] == "Enterprise Activation Request for Linked account.":
        return {
            'Result': 'Service limit increase or Enterprise support activation will not be posted.'
        }

    payload = create_payload(event['account'], case_info)
    url = os.environ['API_URL']
    headers = {
        'Content-Type': 'application/json',
    }
    req = Request(url, json.dumps(payload).encode('utf-8'), headers)

    try:
        response = urlopen(req)
        response.read()
    except HTTPError as e:
        return {
            'Result': f'''Request failed: {e.code}, {e.reason})'''
        }
    except URLError as e:
        return {
            'Result': f'''Request failed: {e.reason})'''
        }
    else:
        return {
            'Result' : 'Knowledge posted.'
        }

The results of the POST are as follows. Sorry, I can't share the case's contents, so it's mostly blacked out.

I hope this will be of help to someone else.

Migrating CloudFront OAI to OAC using CloudFormation

hayao-k — Wed, 21 Sep 2022 12:20:13 +0000

Goals of this post

Describes the CloudFormation template modifications required to migrate CloudFront's Origin access identity (OAI) to Origin Access Control (OAC).

OAC is a new access control method for setting S3 buckets as origins in CloudFront.

Amazon CloudFront introduces Origin Access Control (OAC) | Networking & Content Delivery

aws.amazon.com

Previously we had used Origin Access Identity (OAI) to restrict access to origin S3 buckets to CloudFront only.

OAI is currently treated as Legacy. Migration from OAI to OAC is recommended to support security best practices and new regions.

Sample Template

See my GitHub repository!

hayao-k / CFn-sample-cloudfront-oac

CFn-sample-cloudfront-oac

CloudFormation template for building a static site delivery environment with CloudFront and S3. OAC is used for origin access control.

static-site-deploy-default.yaml

Template for using the default CloudFront domain name.

static-site-deploy-cname.yaml

Template for delivery on a custom domain using a TLS certificate issued by AWS Certificate Manager (ACM). You will need to specify the CNAME and ACM identifier when deploying.

View on GitHub

Examples of modifications

Migrating existing CloudFormation templates from OAI to OAC requires, as a minimum, the following modifications.

Creating an OAC
Modifying the bucket policy
Modifying Distribution Origin

Creating an OAC

Create an AWS::CloudFront::OriginAccessControl. Description of the OriginAccessControlConfig is an optional field.



Resources:
  CloudFrontOriginAccessControl:
    Type: AWS::CloudFront::OriginAccessControl
    Properties: 
      OriginAccessControlConfig:
        Description: Default Origin Access Control
        Name: !Ref AWS::StackName
        OriginAccessControlOriginType: s3
        SigningBehavior: always
        SigningProtocol: sigv4

https://docs.aws.amazon.com/AWSCloudFormation/latest/UserGuide/aws-resource-cloudfront-originaccesscontrol.html

Delete AWS::CloudFront::CloudFrontOriginAccessIdentity as it will no longer be required after migration.

Modifying the bucket policy

In the following example, the policy for OAI has been removed, but it is a recommended migration procedure to include both OAI and OAC policies. This will prevent CloudFront from losing access to S3 buckets during migration to OAC. Please take action as necessary.

Before



 # S3 bucket policy to allow access from CloudFront OAI
  AssetsBucketPolicy:
    Type: AWS::S3::BucketPolicy
    Properties:
      Bucket: !Ref AssetsBucket
      PolicyDocument:
        Statement:
        - Action: s3:GetObject
          Effect: Allow
          Resource: !Sub ${AssetsBucket.Arn}/*
          Principal:
            AWS: !Sub arn:aws:iam::cloudfront:user/CloudFront Origin Access Identity ${CloudFrontOriginAccessIdentity}

After



# S3 bucket policy to allow access from CloudFront OAC
  AssetsBucketPolicy:
    Type: AWS::S3::BucketPolicy
    Properties:
      Bucket: !Ref AssetsBucket
      PolicyDocument:
        Statement:
        - Action: s3:GetObject
          Effect: Allow
          Resource: !Sub ${AssetsBucket.Arn}/*
          Principal:
            Service: cloudfront.amazonaws.com
          Condition:
            StringEquals:
              AWS:SourceArn: !Sub arn:aws:cloudfront::${AWS::AccountId}:distribution/${AssetsDistribution}

https://docs.aws.amazon.com/AmazonCloudFront/latest/DeveloperGuide/private-content-restricting-access-to-s3.html

Modifying Distribution Origin

OAI sets S3OriginConfig for Distribution Origin, but OAC requires OriginAccessControlId to be specified.

Before



  AssetsDistribution:
    Type: AWS::CloudFront::Distribution
    Properties:
      DistributionConfig:
        Origins:
        - Id: S3Origin
          DomainName: !GetAtt AssetsBucket.DomainName
          S3OriginConfig:
            OriginAccessIdentity: !Sub origin-access-identity/cloudfront/${CloudFrontOriginAccessIdentity}

Afeter



  AssetsDistribution:
    Type: AWS::CloudFront::Distribution
    Properties:
      DistributionConfig:
        Origins:
        - Id: S3Origin
          DomainName: !GetAtt AssetsBucket.DomainName
          S3OriginConfig:
            OriginAccessIdentity: ''
          OriginAccessControlId: !GetAtt CloudFrontOriginAccessControl.Id

https://docs.aws.amazon.com/AWSCloudFormation/latest/UserGuide/aws-properties-cloudfront-distribution-origin.html

Note: As of September 2022, an empty OriginAccessIdentity must be specified in S3OriginConfig.

Deleting S3OriginConfig will result in the following error when updating the stack.



Resource handler returned message: "Invalid request provided: Exactly one of CustomOriginConfig and S3OriginConfig must be specified"

Examples of actual migration

Now let's update the existing CloudFormation stack using the modified template. Update the stack to ensure that the change set is as expected.

As described before, removing the bucket policy for OAI and migrating to OAC will result in Access Denied during distribution updates.

After updating the distribution, the S3 bucket access settings can be seen to have been updated to OAC.

The S3 bucket policy is also updated as expected.

Access from the browser seems to be okay.

I hope this will be of help to someone else.

Get daily billing amounts by account with Cost Explorer API

hayao-k — Sun, 26 Jun 2022 15:42:00 +0000

Goal of this article

Use Cost Explorer API to get the daily billing amount for each account and output the following data in CSV.

Account Id	Account Name	2022/4/1	2022/4/2	2022/4/3	...	2022/4/30
000000000000	account-0000	42.792716528	40.124716527	43.123416527	...	50.922465287
111111111111	account-1111	32.263379809	30.235379809	31.263353594	...	22.133798094
222222222222	account-2222	751.71034839	720.51234839	772.62033294	...	651.71042035
333333333333	account-3333	4.6428	5.1234	7.8765	...	6.2234
444444444444	account-4444	407.74542211	420.12345211	395.12499518	...	417.99454118
555555555555	account-5555	386.78950595	400.12500509	352.89924506	...	370.75102656
...

An equivalent CSV can be downloaded from the AWS Cost Explorer console. This post describes how to retrieve the data daily automatically.

Example API Request

This is a minimal example of using AWS Lambda to retrieve the daily billing amount for the current month.

import datetime
import boto3

def lambda_handler(event, context):
    today = datetime.date.today()
    start = today.replace(day=1).strftime('%Y-%m-%d')
    end = today.strftime('%Y-%m-%d')
    ce = boto3.client('ce')
    response = ce.get_cost_and_usage(
        TimePeriod={
            'Start': start,
            'End' :  end,
        },
        Granularity='DAILY',
        Metrics=[
            'NetUnblendedCost'
        ],
        GroupBy=[
            {
                'Type': 'DIMENSION',
                'Key': 'LINKED_ACCOUNT'
            }
        ]
    )
    return response['ResultsByTime']

One point to note is that there is a time lag before the cost for a given day is determined. For example, if the cost for April 25 is obtained on April 26, it may be less than the actual billing amount.

The timing of when the AWS data will be updated is not disclosed, but it appears that on April 27, the costs for April 25 will be almost finalized.

Processing with Pandas

Since the API response is a nested JSON, we will consider an example of processing it into a CSV using Pandas.

When using Pandas with AWS Lambda, Lambda Layers must be used.

Since each element contains billing data daily, it is processed with a for statement one day at a time.
After flattening the data using pandas.json_normalize, it is concatenated with the billing amount using pandas.concat.
After further renaming the column to the billing date, the results are merged using the Account Id as the key.

    merged_cost = pandas.DataFrame(
        index=[],
        columns=['Account Id']
    )

    for index, item in enumerate(response):
        normalized_json = pandas.json_normalize(item['Groups'])
        split_keys = pandas.DataFrame(
            normalized_json['Keys'].tolist(),
            columns=['Account Id']
        )
        cost = pandas.concat(
            [split_keys, normalized_json['Metrics.NetUnblendedCost.Amount']],
            axis=1
        )
        renamed_cost = cost.rename(
            columns={'Metrics.NetUnblendedCost.Amount': item['TimePeriod']['Start']}
        )
        merged_cost = pandas.merge(merged_cost, renamed_cost, on='Account Id', how='right')

   print(merged_cost)

Account Id  ...     2022-04-25
0   000000000000  ...  15.4985752779
1   111111111111  ...         0.2176
2   222222222222  ...   6.5567854795
3   333333333333  ...   6.6300957379
4   444444444444  ...   8.2720868504
..           ...  ...            ...
19  777777777777  ...  10.0121863554
18  888888888888  ...   6.5976412116
20  999999999999  ...    6.493243618
[20 rows x 26 columns]

Example of Lambda function

After processing with for statement, the list of account names obtained from AWS Organizations API is merged and output in CSV.

from logging import getLogger, INFO
import os
import datetime
import boto3
import pandas
from botocore.exceptions import ClientError

logger = getLogger()
logger.setLevel(INFO)

def upload_s3(output, key, bucket):
    try:
        s3_resource = boto3.resource('s3')
        s3_bucket = s3_resource.Bucket(bucket)
        s3_bucket.upload_file(output, key, ExtraArgs={'ACL': 'bucket-owner-full-control'})
    except ClientError as err:
        logger.error(err.response['Error']['Message'])
        raise

def get_ou_ids(org, parent_id):
    ou_ids = []

    try:
        paginator = org.get_paginator('list_children')
        iterator = paginator.paginate(
            ParentId=parent_id,
            ChildType='ORGANIZATIONAL_UNIT'
        )
        for page in iterator:
            for ou in page['Children']:
                ou_ids.append(ou['Id'])
                ou_ids.extend(get_ou_ids(org, ou['Id']))
    except ClientError as err:
        logger.error(err.response['Error']['Message'])
        raise
    else:
        return ou_ids

def list_accounts():
    org = boto3.client('organizations')
    root_id = 'r-xxxx'
    ou_id_list = [root_id]
    ou_id_list.extend(get_ou_ids(org, root_id))
    accounts = []

    try:
        for ou_id in ou_id_list:
            paginator = org.get_paginator('list_accounts_for_parent')
            page_iterator = paginator.paginate(ParentId=ou_id)
            for page in page_iterator:
                for account in page['Accounts']:
                    item = [
                        account['Id'],
                        account['Name'],
                    ]
                    accounts.append(item)
    except ClientError as err:
        logger.error(err.response['Error']['Message'])
        raise
    else:
        return accounts

def get_cost_json(start, end):
    ce = boto3.client('ce')
    response = ce.get_cost_and_usage(
        TimePeriod={
            'Start': start,
            'End' :  end,
        },
        Granularity='DAILY',
        Metrics=[
            'NetUnblendedCost'
        ],
        GroupBy=[
            {
                'Type': 'DIMENSION',
                'Key': 'LINKED_ACCOUNT'
            }
        ]
    )
    return response['ResultsByTime']

def lambda_handler(event, context):
    today = datetime.date.today()
    start = today.replace(day=1).strftime('%Y-%m-%d')
    end = today.strftime('%Y-%m-%d')
    key = 'daily-cost-' + today.strftime('%Y-%m') + '.csv'
    output_file = '/tmp/output.csv'
    bucket = os.environ['BUCKET']
    account_list = pandas.DataFrame(list_accounts(), columns=['Account Id', 'Account Name'])
    daily_cost_list = get_cost_json(start, end)

    merged_cost = pandas.DataFrame(
        index=[],
        columns=['Account Id']
    )

    for index, item in enumerate(daily_cost_list):
        normalized_json = pandas.json_normalize(item['Groups'])
        split_keys = pandas.DataFrame(
            normalized_json['Keys'].tolist(),
            columns=['Account Id']
        )
        cost = pandas.concat(
            [split_keys, normalized_json['Metrics.NetUnblendedCost.Amount']],
            axis=1
        )
        renamed_cost = cost.rename(
            columns={'Metrics.NetUnblendedCost.Amount': item['TimePeriod']['Start']}
        )
        merged_cost = pandas.merge(merged_cost, renamed_cost, on='Account Id', how='outer')

    daily_cost = pandas.merge(account_list, merged_cost, on='Account Id', how='right')
    daily_cost.to_csv(output_file, index=False)
    upload_s3(output_file, key, bucket)

Now all that is left is setting an arbitrary startup schedule in EventBridge and a Lambda function as the target.

Configure actions-runner-controller with proxy in private EKS cluster

hayao-k — Sun, 12 Jun 2022 17:31:07 +0000

Goals of this post

Run actions-runner-controller on Private EKS cluster
Communication with GitHub API is through on-premises proxy server

This post is validated with the following configuration.

Amazon EKS: Kubernetes 1.22
actions-runner-controller: v0.23.0
cert-manager v1.8.0
eksctl: 0.93.0
kubectl: v1.21.2
AWS CLI: 2.5.7

What is actions-runner-controller?

Controller for operating GitHub Actions self-hosted runners on Kubernetes cluster.

actions / actions-runner-controller

Kubernetes controller for GitHub Actions self-hosted runners

Actions Runner Controller (ARC)

About

Actions Runner Controller (ARC) is a Kubernetes operator that orchestrates and scales self-hosted runners for GitHub Actions.

With ARC, you can create runner scale sets that automatically scale based on the number of workflows running in your repository, organization, or enterprise. Because controlled runners can be ephemeral and based on containers, new runner instances can scale up or down rapidly and cleanly. For more information about autoscaling, see "Autoscaling with self-hosted runners."

You can set up ARC on Kubernetes using Helm, then create and run a workflow that uses runner scale sets. For more information about runner scale sets, see "Deploying runner scale sets with Actions Runner Controller."

People

Actions Runner Controller (ARC) is an open-source project currently developed and maintained in collaboration with the GitHub Actions team, external maintainers @mumoshu and @toast-gear, various contributors, and the awesome community.

If you think the…

View on GitHub

The following describes the flow of building actions-runner-controller on private EKS cluster.

Create private EKS Cluster

Set up kubectl, eksctl, and AWS CLI in your working environment in advance. See below for detailed instructions.

kubectl: https://kubernetes.io/docs/tasks/tools/

eksctl: https://docs.aws.amazon.com/eks/latest/userguide/eksctl.html

AWS CLI: https://docs.aws.amazon.com/cli/latest/userguide/getting-started-install.html

Please refer to the following document to create a private cluster with eksctl.

https://eksctl.io/usage/eks-private-cluster/

In this case, created an EKS cluster using the following config file.

apiVersion: eksctl.io/v1alpha5
kind: ClusterConfig

metadata:
  name: actions-runner
  region: ap-northeast-1
  version: "1.22"

privateCluster:
  enabled: true
  skipEndpointCreation: true

vpc:
  subnets:
    private:
      ap-northeast-1a:
        id: subnet-xxxxxxxxxxxxxxxxx
      ap-northeast-1c:
        id: subnet-yyyyyyyyyyyyyyyyy
      ap-northeast-1d:
        id: subnet-zzzzzzzzzzzzzzzzz

For a private EKS cluster, the following VPC endpoints are required.

ECR (ecr.api, ecr.dkr)
S3 (Gateway)
EC2
STS
CloudWatch Logs

If you enable a private cluster with eksctl (privateCluster.enabled), these endpoints will be created automatically.

In this case, I had a requirement to use a VPC already connected via Direct Connect, so skip endpoint creation with privateCluster.skipEndpointCreation to use the existing VPC and endpoints.

NOTE: if you specify an existing VPC, eksctl will edit the route table.

However, if the subnet is associated with the main route table, eksctl will not edit the route table and will fail to create the cluster. Therefore, it is necessary to create and associate a route table explicitly.

eksctl communicates with the AWS API via a proxy server. However, communication to private EKS cluster endpoints does not use a proxy server, so the following environment variable should be set.

export https_proxy=http://xxx.xxx.xxx.xxx:yyyy
export no_proxy=.eks.amazonaws.com

The following command creates a cluster. Creating a private cluster takes longer than creating a regular cluster. This configuration took about 22 minutes, but if you are creating VPC or VPC endpoints, it will take even longer, so adjust the timeout value.

$ eksctl create cluster --timeout 30m -f private-cluster.yaml

Register Managed Node Groups

Add UserData to the launch template used by the managed node group and configure the Docker daemon to use a proxy.

This template is based on the following blog.

https://yomon.hatenablog.com/entry/2020/10/eks_docker_inside_proxy#%E3%83%9E%E3%83%8D%E3%83%BC%E3%82%B8%E3%83%89%E5%9E%8B%E3%83%8E%E3%83%BC%E3%83%89%E3%82%B0%E3%83%AB%E3%83%BC%E3%83%97%E3%81%ABProxy%E8%A8%AD%E5%AE%9A

AWSTemplateFormatVersion: "2010-09-09"
Parameters:
  ClusterIp:
    Type: String
  ProxyIP:
    Type: String
  ProxyPort:
    Type: Number
  SshKeyPairName:
    Type: String

Resources:
  EKSManagedNodeInPrivateNetworkLaunchTemplate:
    Type: AWS::EC2::LaunchTemplate
    Properties: 
      LaunchTemplateName: eks-managednodes-in-private-network
      LaunchTemplateData: 
        InstanceType: t3.small
        KeyName: !Ref SshKeyPairName
        TagSpecifications: 
          - ResourceType: instance
            Tags:
              - Key: Name 
                Value: actions-runner-nodegroup
          - ResourceType: volume
            Tags:
              - Key: Name
                Value: actions-runner-nodegroup

        UserData: !Base64 
          "Fn::Sub": |
            Content-Type: multipart/mixed; boundary="==BOUNDARY=="
            MIME-Version:  1.0

            --==BOUNDARY==
            Content-Type: text/cloud-boothook; charset="us-ascii"

            #Set the proxy hostname and port
            PROXY=${ProxyIP}:${ProxyPort}
            MAC=$(curl -s http://169.254.169.254/latest/meta-data/mac/)
            VPC_CIDR=$(curl -s http://169.254.169.254/latest/meta-data/network/interfaces/macs/$MAC/vpc-ipv4-cidr-blocks | xargs | tr ' ' ',')

            #Create the docker systemd directory
            mkdir -p /etc/systemd/system/docker.service.d

            #Configure docker with the proxy
            cloud-init-per instance docker_proxy_config tee <<EOF /etc/systemd/system/docker.service.d/http-proxy.conf >/dev/null
            [Service]
            Environment="HTTP_PROXY=http://$PROXY"
            Environment="HTTPS_PROXY=http://$PROXY"
            Environment="NO_PROXY=${ClusterIp},$VPC_CIDR,localhost,127.0.0.1,169.254.169.254,.internal,s3.amazonaws.com,.s3.ap-northeast-1.amazonaws.com,api.ecr.ap-northeast-1.amazonaws.com,dkr.ecr.ap-northeast-1.amazonaws.com,ec2.ap-northeast-1.amazonaws.com,ap-northeast-1.eks.amazonaws.com"
            EOF

            #Reload the daemon and restart docker to reflect proxy configuration at launch of instance
            cloud-init-per instance reload_daemon systemctl daemon-reload 
            cloud-init-per instance enable_docker systemctl enable --now --no-block docker
            --==BOUNDARY==

After creating the launch template, create a Managed Node Group with the following config file.

apiVersion: eksctl.io/v1alpha5
kind: ClusterConfig

metadata:
  name: actions-runner
  region: ap-northeast-1

managedNodeGroups:
- name: t3s-proxy
  desiredCapacity: 1
  privateNetworking: true
  launchTemplate:
    id: lt-xxxxxxxxxxxxxxxxx
    version: "1"

Run create nodegroup.

$ eksctl create nodegroup -f nodegroup.yaml

Installing actions-runner-controller

actions-runner-controller uses cert-manager to manage certificates for Admission Webhooks, so it must be installed in advance.

$ kubectl apply -f https://github.com/cert-manager/cert-manager/releases/download/v1.8.0/cert-manager.yaml

To use actions-runner-controller in a proxy environment, it is necessary to configure the proxy server information in the following three locations.

manager container of controller-manager
runner container of RunnerDeployment
sidecar (dind) container of RunnerDeployment

I deployed actions-runner-controller with kubectl this time.

1 sets the https_proxy environment variable to the Deployment definition of controller-manager.
2 and 3 set the https_proxy environment variable to the definition of RunnerDeployment.

Edit the manifest file of actions-runner-controller as follows.

33836     spec:
33837       containers:
33838       - args:
33839         - --metrics-addr=127.0.0.1:8080
33840         - --enable-leader-election
33841         command:
33842         - /manager
31843         env:
31844         - name: GITHUB_TOKEN
31845           valueFrom:
31846             secretKeyRef:
31847               key: github_token
31848               name: controller-manager
31849               optional: true
31850         - name: GITHUB_APP_ID
31851           valueFrom:
31852             secretKeyRef:
31853               key: github_app_id
31854               name: controller-manager
31855               optional: true
31856         - name: GITHUB_APP_INSTALLATION_ID
31857           valueFrom:
31858             secretKeyRef:
31859               key: github_app_installation_id
31860               name: controller-manager
31861               optional: true
31862         - name: GITHUB_APP_PRIVATE_KEY
31863           value: /etc/actions-runner-controller/github_app_private_key
+ 31864       - name: http_proxy
+ 31865         value: "http://xxx.xxx.xxx.xxx:xxxx"
+ 31866       - name: https_proxy
+ 31867         value: "http://xxx.xxx.xxx.xxx:xxxx"
+ 31868       - name: no_proxy
+ 31869         value: "172.20.0.1,*.eks.amazonaws.com"
31870         image: summerwind/actions-runner-controller:v0.22.3
31871         name: manager

Run kubectl create.

$ kubectl create -f actions-runners-controller.yaml

NOTE: Due to a known issue with Kubernetes, we must use create (or replace when updating) instead of apply.

https://github.com/actions-runner-controller/actions-runner-controller/issues/1317

GitHub API can be authenticated using the GitHub App or Personal Access Token (PAT).

In this case, we use PAT authentication. Set the appropriate scope for the token depending on the type of runner and create a secret.

kubectl create secret generic controller-manager \
    -n actions-runner-system \
    --from-literal=github_token=${GITHUB_TOKEN}

NOTE: PAT must be pre-authorized before accessing an Organization with SAML SSO configured.

https://docs.github.com/ja/enterprise-cloud@latest/authentication/authenticating-with-saml-single-sign-on/authorizing-a-personal-access-token-for-use-with-saml-single-sign-on

RunnerDeployment is defined below.

apiVersion: actions.summerwind.dev/v1alpha1
kind: RunnerDeployment
metadata:
  name: runner-deploy
spec:
  template:
    spec:
      enterprise: <enterprise-name>
      labels:
        - runner-container
      env:
        - name: RUNNER_FEATURE_FLAG_EPHEMERAL
          value: "true"
        - name: http_proxy
          value: "http://xxx.xxx.xxx.xxx:yyyy"
        - name: https_proxy
          value: "http://xxx.xxx.xxx.xxx:yyyy"
        - name: no_proxy
          value: "172.20.0.1,10.1.2.0/24,*.eks.amazonaws.com"
      dockerEnv:
        - name: http_proxy
          value: "http://xxx.xxx.xxx.xxx:yyyy"
        - name: https_proxy
          value: "http://xxx.xxx.xxx.xxx:yyyy"
        - name: no_proxy
          value: "172.20.0.1,10.1.2.0/24,*.eks.amazonaws.com"
---
apiVersion: actions.summerwind.dev/v1alpha1
kind: HorizontalRunnerAutoscaler
metadata:
  name: runner-deployment-autoscaler
spec:
  scaleDownDelaySecondsAfterScaleOut: 300
  scaleTargetRef:
    name: runner-deploy
  minReplicas: 1
  maxReplicas: 5
  metrics:
  - type: PercentageRunnersBusy
    scaleUpThreshold: '0.75'
    scaleDownThreshold: '0.25'
    scaleUpFactor: '2'
    scaleDownFactor: '0.5'

When a container is used in a GitHub Actions job, the docker container for dind, launched as a RunnerDeployment sidecar, is used. Specify dockerEnv so that this dind container can perform image pulls.

After deployment, confirm that the pod is successfully started and is visible from GitHub.

$ kubectl apply -f RunnerDeployment.yaml

$ kubectl get pod
NAME                            READY   STATUS    RESTARTS   AGE
pod/runner-deploy-xxxxx-xxxxx   2/2     Running   0          53s

If controller-manager does not start properly, check the settings in 1 or GitHub; if Runner does not start properly, check the settings in 2 or 3.

Differences between Amazon ECR and Inspector image scanning capabilities

hayao-k — Thu, 27 Jan 2022 14:42:16 +0000

Introduction

At AWS re:Invent 2021, the vulnerability management service Amazon Inspector was redesigned and released as the all-new Amazon Inspector (v2). The new Inspector not only scans EC2 but also scans container images stored in Amazon ECR.

Improved, Automated Vulnerability Management for Cloud Workloads with a New Amazon Inspector
https://aws.amazon.com/jp/blogs/aws/improved-automated-vulnerability-management-for-cloud-workloads-with-a-new-amazon-inspector/

Differences

With the introduction of image scanning by Inspector, the ECR scanning function is now called Basic scanning, while the Inspector scanning function is called Enhanced scanning. The main differences between Basic scanning and Enhanced scanning are as follows.

	Basic scanning	Enhanced scanning
Vulnerability Detection Target	OS packages only	OS and programming language packages
Vulnerability Detection Timing	When the image is pushed	When vulnerabilities occur
Pricing	Free	Not free
Amazon EventBrdige Integration	Yes (Scan result summary only)	yes
AWS Security Hub Integration	No	Yes
AWS Organizations Integration	No	Yes

Vulnerability Detection Target

Basic scanning provides a scan using the CVE database of the open-source Clair project. Only OS packages are targeted for vulnerability detection.

Enhanced scanning is capable of detecting vulnerabilities in programming language packages in addition to OS packages. Supported programming languages are as follows.

C#
Golang
Java
JavaScript
PHP
Python
Ruby
Rust

Supported operating systems and programming languages

Vulnerability Detection Timing

Basic scanning can be triggered when an image is pushed (Scan on push) or manually. Manual scans are limited to once every 24 hours for each image.

For Enhanced scanning, continuous scans can be used for repositories. Continuous scanning automatically scans whenever an image is pushed and whenever the Amazon Inspector vulnerability database is updated. This means that vulnerabilities can be detected at about the same time as vulnerability information is updated.

For Enhanced scanning, you can define whether you want to enable continuous scanning or only scan on push in the repository name scan filter. Manual scan execution is not possible with Enhanced scanning.

Pricing

Basic scanning is free of charge, but Enhanced scanning is a paid feature, with the following monthly fees for the Tokyo region as of 12/2021

Per first container image scanned during a push to ECR: $0.11
Number of Continuous scans for a container image: $0.01 per scan

Pricing page: https://aws.amazon.com/inspector/pricing/

Integration with Amazon EventBridge

When Basic scanning is complete, an event is sent to EventBridge, and you can get a summary of the scan results.

Amazon ECR events and EventBridge

Enhanced scanning is enabled, the following events are sent to EventBridge.

Event for a repository scan frequency change
Event for an initial image scan (equivalent to a Basic scanning)
Event for an image scan finding update (created, updated, closed)

Enhanced scanning differs from Basic scanning in that an event is issued each time a vulnerability is found. For more details, please refer to the following document.

Enhanced scanning

Integration with AWS Security Hub

In environments where AWS Security Hub is enabled, integration with Amazon Inspector is also automatically enabled. Vulnerabilities discovered by Enhanced scanning are automatically sent to AWS Security Hub and can be included in existing security operations workflows.

Integration with AWS Security Hub

AWS Organizations support

New Amazon Inspector also supports integration with AWS Organizations. Delegated administrator accounts can enable EC2 scans and ECR scans (Enhanced scanning) for all member accounts in an organization to manage vulnerabilities. It also supports the automatic activation of new accounts added to the organization.

Enabling scans for member accounts

Points to note when using Enhanced scanning

Manually scanning cannot be performed

Manually scanning cannot be performed in an environment with Enhanced scanning enabled.

Per-repository scan settings deprecated

Setting up a repository-level image scan has been deprecated. The use of scan filters is recommended even when using Basic scanning. The scan filter settings will take precedence if the repository-level and the registry scan filters are set. Continuous Scan setting for Enhanced Scanning can only be specified in the scan filter.

Continuous scanning is available up to 30 days after the image is pushed

When a continuous scan is configured, the image will be scanned for 30 days after being pushed to the repository. If the image has not been updated in the last 30 days, the continuous scan for that image will be paused.

Basic scanning cannot be used together

The scan settings are for the entire registry. It is not possible to switch the scan type for each repository. Also, if you enable Enhanced scanning, you will not see the results of previous Basic scanning in the console.

The results are not lost, and you can refer to them again by changing the scan type back to Basic scanning.

Try Enhanced scanning

Settings

From the Private registry in the Amazon ECR console, click Edit for Scanning.

Select Enhanced scanning as the scan type. Note that this is a setting for the registry, so it cannot be used in conjunction with the basic scan. For both continuous scan and scan on push, you can use scan filters to narrow down the repositories to be scanned. In this example, we set the filter to target repository names starting with test/, but you can also target all repositories.

Click Confirm when you see a message about additional charges for Enhanced scanning.

Make sure that the scan settings have changed from Basic to Enhanced version.

If you check the Account management page of the new Inspector console, you will see that ECR container scanning is Enabled.

You can also check the coverage of repositories that have Enhanced scanning enabled in the Inspector dashboard. Four repositories in this environment had "scan on push" enabled in Basic scanning, so it has been carried over. Since the per-repository image scan setting has been deprecated, it is preferable to disable and use scan filters to manage the coverage.

Operation check

Let's push the image. Since Enhanced scanning can detect language-specific vulnerabilities, I used a container image of a Java application. Enter the repository name to match the scan filter you have just set.

You will see that the scan frequency for the created repository is set to Continuous.

I pushed an image containing an old Java application for testing to detect the vulnerability.

$ aws ecr get-login-password | docker login --username AWS --password-stdin 123456789012.dkr.ecr.ap-northeast-1.amazonaws.com
Login Succeeded

$ docker tag test/java-sample-app:v1.1.0 123456789012.dkr.ecr.ap-northeast-1.amazonaws.com/test/java-sample-app:v1.1.0
$ docker push 123456789012.dkr.ecr.ap-northeast-1.amazonaws.com/test/java-sample-app:v1.1.0
The push refers to repository [123456789012.dkr.ecr.ap-northeast-1.amazonaws.com/test/java-sample-app]
c5baabd61e59: Pushed 
88e64033fc7f: Pushed 
0466be121ce3: Pushed 
b87942114db6: Pushed 
v1.1.0: digest: sha256:69a58fe6b25d21015da0f170ffa6934f2ec2827562238ecbde902ldkgi2d082b size: 1165

If you check the Inspector console, you will see that it detects quite many vulnerabilities.

Let's look at one of the critical vulnerabilities from All Findings. We are detecting a vulnerability in Jackson, a library for processing JSON in Java. Basic scanning does not detect vulnerabilities in programming language packages such as this.

The results were also linked to Security Hub.

Enhanced scanning be performed on an existing image?

When I enabled Enhanced Scanning, I noticed that the initial scan is also performed on some images stored in the existing repository. As far as I can tell, the initial scan will be performed on the stored images if the following conditions are met.

The repository is the target of a continuous scan
The image has been pushed within 30 days