DEV Community: Kay

Hacktoberfest 2023 Pledge

Kay — Wed, 18 Oct 2023 00:45:52 +0000

This will be my sixth Hacktoberfest!

How does Amazon Managed Service for Prometheus relate to Amazon CloudWatch?

Kay — Sun, 09 Oct 2022 02:03:48 +0000

Amazon CloudWatch

CloudWatch provides end-to-end observability across logs, metrics, and traces for applications running on EC2, AWS container services (EKS, ECS), Lambda, and other AWS services.
CloudWatch can discover and collect Prometheus metrics as CloudWatch metrics to provide options for our customers to query and alarm on Prometheus metrics.
You should use CloudWatch if you are looking for a comprehensive observability service that brings together logs, metrics, tracing, dashboarding, and alerting in a unified experience that encompasses AWS services, EC2, containers, and serverless.

Amazon Managed Service for Prometheus

Amazon Managed Service for Prometheus, which is specifically optimized for monitoring container-based workloads, offers a Prometheus-compatible APIs for ingesting and querying your Prometheus metrics.
Amazon Managed Service for Prometheus is a metric-only service and does not collect logs or distributed trace data. You can export selected CloudWatch metrics to Amazon Managed Service for Prometheus in order to use PromQL as the common query language for querying and alarming on all your stored metrics.
You should use Amazon Managed Service for Prometheus if you want a service that is fully compatible with the Prometheus open source project. You should also choose Amazon Managed Service for Prometheus if you are already running Prometheus and are looking to eliminate that ongoing operational cost while also improving security.

See https://aws.amazon.com/prometheus/faqs/.

CloudWatch metrics + AWS Distro for OpenTelemetry (ADOT) + Amazon Managed Service for Prometheus + Amazon Managed Grafana

This blog post could be a good start for people who are interested in CloudWatch metrics + AWS Distro for OpenTelemetry (ADOT) + Amazon Managed Service for Prometheus + Amazon Managed Grafana -
Viewing Amazon CloudWatch metrics with Amazon Managed Service for Prometheus and Amazon Managed Grafana.

Gotchas when building GitHub self-hosted runners with AWS official AMIs/container images for Python apps

Kay — Sat, 01 Oct 2022 05:54:45 +0000

The following are some gotchas when setting up specific Python versions on GitHub self-hosted runners which are based on AWS official AMIs and container images.

Official Amazon Linux 2 AMI and official Amazon Linux 2 container image do not have the same Python runtimes setup.
- Amazon Linux 2 AMI (ECS-optimized AMI /aws/service/ecs/optimized-ami/amazon-linux-2/recommended) has python2.7 and python3.7 by default.
- Amazon Linux 2 container image amazonlinux/amazonlinux (public.ecr.aws/amazonlinux/amazonlinux) has only python2.7; it does not have python3/pip installed by default.
Amazon Linux 2 AMI has python3 pointed to python3.7 by default. When changing python3 to point to other Python3 version (e.g. setting update-alternatives --install /usr/bin/python3 python3 /usr/bin/python3.8 1), it may break cfn-signal so no signal will be sent to ASG at EC2 launch (i.e. no healthy instances will be registered).
Note that this will be an issue if you use ASG for the runners (not ECS Fargate).
Keep python to point to python2.7, as yum does not support Python3. You will see this error if setting python to point to Python3:
```
> yum
File "/usr/bin/yum", line 30
except KeyboardInterrupt, e:
                       ^
SyntaxError: invalid syntax
```
See also https://stackoverflow.com/questions/11213520/yum-crashed-with-keyboard-interrupt-error.
actions/setup-python@v4 does not support arm64 (https://raw.githubusercontent.com/actions/python-versions/main/versions-manifest.json).
See related Open issue https://github.com/actions/setup-python/issues/108.

Note: (3) and (4) are not Amazon Linux specific. Just issues experienced when building the Amazon official image based self-hosted runners.

Some notes about Bottlerocket Security

Kay — Mon, 26 Sep 2022 23:34:47 +0000

This post includes some notes about Bottlerocket security.

CIS Hardening Benchmark for Bottlerocket
FIPS Support / Validation
Does Bottlerocket have integration with AWS Inspector?
Is OS host logs available? Does it have integration with CloudWatch Log?
Reduced attack surface, verified software, enforced permission boundaries
ECS/EBS encryption vs. OS crypto

CIS Hardening Benchmark for Bottlerocket

Bottlerocket now has a Center for Internet Security (CIS) Benchmark. The CIS Benchmark is a catalog of security-focused configuration settings that help Bottlerocket customers configure or document any non-compliant configurations in a simple and efficient manner. The CIS Benchmark for Bottlerocket includes both Level 1 and Level 2 configuration profiles.

FIPS Support / Validation

Issue (Open): https://github.com/bottlerocket-os/bottlerocket/issues/1667

FIPS compliance is our second most requested feature, behind CIS (which is in progress), and I'm planning to focus on it once the CIS benchmark is complete.

Does Bottlerocket have integration with AWS Inspector?

Bottlerocket is now supported by AWS inspector in commercial regions.

Is OS host logs available? Does it have integration with CloudWatch Log?

No. There is no current plan to add a logging agent to the host OS.

Issue (Open) https://github.com/bottlerocket-os/bottlerocket/issues/850

Comments from Maintainers:

We have no current plans to add a logging agent to the host OS.

When talking with many EKS customers, we found that a common pattern is to use Kubernetes’ facilities for log streaming, even for system level logs. Another method is to use Fluent Bit as covered in this blog post. These are our suggested methods for customers to get both container logs as well as other logs off the box.

Reduced attack surface, verified software, enforced permission boundaries

Bottlerocket contains less software, and notably eliminates some components you might expect: Bottlerocket doesn’t have SSH, any interpreters like Python, or even a shell; it is expected that Bottlerocket to be "hands-off" most of the time, and removing components like this makes it harder for an attacker to gain a foothold in the system. Beyond removal of software, Bottlerocket also reduces the attack surface of the operating system by applying software hardening techniques like:
- building position-independent executables (PIE),
- using relocation read-only (RELRO) linking, and
- building all first-party software with memory-safe languages like Rust and Go.
Bottlerocket uses SELinux in enforcing mode to restrict modifications to itself even from privileged containers. SELinux is an implementation of Mandatory Access Control (MAC) enforced by the Linux kernel, and limits the set of actions processes can take. Today, Bottlerocket’s SELinux policy is intended to restrict orchestrated containers from causing undesired and unexpected changes to the operating system. Going forward, we want to extend this policy to apply to all categories of persistent threats.

See

ECS/EBS encryption vs. OS crypto

Bottlerocket operates with 2 default storage volumes - standard EBS encryption applicable

The root device, holds the active and passive partition sets. It also contains the bootloader, the dm-verity hash tree for verifying the immutable root filesystem, and the data store for the Bottlerocket API.
The data device is used as persistent storage for container images, container orchestration, host-containers, and bootstrap containers.

Bottlerocket cryptographically verifies itself

The operating system is composed of a disk image that is verified on boot with dm-verity; unexpected changes to the contents of the disk image will cause the operating system to fail to boot.

Bottlerocket uses its own software updater rather than a more common Linux package manager. Updates to Bottlerocket are vended from a repository that follows The Update Framework (TUF) specification; TUF mitigates common classes of attacks against software repositories present in traditional package manager systems.

Source: https://aws.amazon.com/blogs/containers/bottlerocket-a-special-purpose-container-operating-system/

AWS Lambda Gotchas

Kay — Fri, 23 Sep 2022 01:34:15 +0000

This post includes few gotchas on AWS Lambda.

1) Lambda Python Runtimes - Python 3.6/3.7 are Amazon Linux 1 and Python 3.8/3.9 are Amazon Linux 2

Python 3.6/3.7 are Amazon Linux 1 and Python 3.8/3.9 are Amazon Linux 2.

In general it should be fine to upgrade from Python 3.6 to 3.9.

But there are cases you'll need to make some changes. For example, if you have code utilizing some sys call, e.g. curl - curl is not installed in Amazon Linux 2 by default.

2) AWS CLI not allowing valid JSON in payload parameter with lambda invoke

If you see error like Invalid base64:, it could be because since awscli 2, payloads need to be base64 encoded when invoking a Lambda function.

By default, the AWS CLI version 2 now passes all binary input and binary output parameters as base64-encoded strings. A parameter that requires binary input has its type specified as blob (binary large object) in the documentation.

You will need to pass in also --cli-binary-format raw-in-base64-out. For example:

aws lambda invoke --function-name testsms \
    --invocation-type Event \
    --cli-binary-format raw-in-base64-out \
    --payload '{"key": "test"}' response.json

3) Cannot do `ping` from Lambda Function

See AWS Lambda FAQs

Lambda attempts to impose as few restrictions as possible on normal language and operating system activities, but there are a few activities that are disabled: Inbound network connections are blocked by AWS Lambda, and for outbound connections, only TCP/IP and UDP/IP sockets are supported, and ptrace (debugging) system calls are blocked. TCP port 25 traffic is also blocked as an anti-spam measure.

4) Code storage for uploaded Lambda functions (`CodeStorageExceededException`)

The Lambda service stores your function code in an internal S3 bucket that's private to your account. Each AWS account is allocated 75 GB of storage in each Region (and can be increased up to Terabytes). Code storage includes the total storage used by both Lambda functions and layers. If you reach the quota, you receive a CodeStorageExceededException when you attempt to deploy new functions.

See Lambda quotas.

To see the storage used

From AWS Lambda console > Dashboard
From AWS CLI:
```
aws lambda list-versions-by-function --function-name myTestFunction
aws lambda get-layer-version --layer-version --layer-name TestLayer --version-number 2
```
This returns each published version of the function/layer together with the $LATEST version. The CodeSize attribute shows the total number of bytes used by code storage of this function/layer.

See Monitoring Lambda code storage.

Amazon EC2 On-Demand Instance vCPUs limits

Kay — Thu, 22 Sep 2022 01:06:17 +0000

On-Demand Instance vCPUs limits

There is a limit on the number of running On-Demand Instances per AWS account per Region. On-Demand Instance limits are managed in terms of the number of vCPUs that your running On-Demand Instances are using, regardless of the instance type. Each limit specifies the vCPU limit for one or more instance families.

To see the current vCPUs limits of your account from AWS EC2 console
- Enter "vcpu" in the "Find limits" to shortlist the limits
To find out the current running instances (of the same instance family of the EC2 type you want to check), go to EC2 console.
Calculate how many vCPUs you nee
- You can use the vCPU limits calculator to see what will be the numbers for adding instances of certain type.
Request a limit increase
- Even though EC2 automatically increases your On-Demand Instance limits based on your usage, you can request a limit increase if necessary. See Request a limit increase for details.

Amazon EC2 Image Builder Gotchas

Kay — Thu, 22 Sep 2022 00:56:27 +0000

Gotchas

In AWS::ImageBuilder::ContainerRecipe, Parameters is supported only from Console and AWS CLI, but not from CloudFormation. Confirmed with AWS support.
In AWS::ImageBuilder::ContainerRecipe, ParentImage (or Base image in Console) cannot reference another AWS account's ECR repo - this is not mentioned in AWS documentation. Confirmed with AWS support.
If something is not right at early stage (e.g. parse file in Component Data), you will see Internal Failure in CloudFormation console, but the errors will not be logged in S3 nor CloudWatch Logs.
CloudWatch Logs - /aws/imagebuilder/${ImageName}
- Logging things happen in the EC2 instance of the build only.
- Not for Component Data syntax error, version conflict, etc.
Tags are not inherited from the CloudFormation stack for all Image Builder resources (Component, Image Recipe / Container Recipe, Infrastructure configuration, Distribution, Image Pipeline).
Adding, removing, renaming Tags, need to change Version; otherwise Internal Failure will be shown in the CloudFormation console, nothing in S3 log nor CW logs. Also need to update upstream resource's version e.g. Recipe Version.
Whenever a change to Recipe (include Git Component)
- Build time ~30 mins

Accessing an API Gateway private REST API in another AWS account using an interface VPC endpoint

Kay — Thu, 22 Sep 2022 00:40:25 +0000

This repository kyhau/access-private-apigw-in-another-account provides a working example for calling a private API Gateway REST API from another AWS account, including CloudFormation templates, API test code and GitHub Actions workflows.

Concept

To use an interface VPC endpoint to access an API Gateway private REST API that's in another AWS account, do the following:

Create an interface endpoint in a VPC in one account (account A).
Create an API Gateway private REST API in a second account (account B).
Configure a resource policy for the private REST API that allows the interface endpoint to invoke the API.
Test the setup by calling the private REST API from account A. There are 4 ways to call the API and are covered in the Lambda function ApiTesterFunction (ApiTester.py).

For details see How can I access an API Gateway private REST API in another AWS account using an interface VPC endpoint?

Deployment Example

The workflow deploy-apigw-vpce.yaml deploys the interface VPC endpoint for API Gateway execute-api associated to VPC subnet(s) in Account-A.
The workflow deploy-apigw.yaml deploys a simple API Gateway private REST API to Account-B. The stack deploys deploy a resource policy for the private REST API that allows the interface endpoint to invoke the API.
The workflow deploy-lambda-api-tester.yaml deploys a Lambda function to VPC subnet(s) in Account-A, for testing the

Testing the API endpoints with the Lambda function ApiTesterFunction

The Lambda function ApiTesterFunction (ApiTester.py) tests the 4 endpoints/approaches accessing the API in another AWS account:

Default APIGW endpoint; work only if private DNS enabled for your interface endpoint
```
https://dummyapiid.execute-api.ap-southeast-2.amazonaws.com/v0/mock
```

API Gateway Route 53 Alias for VPC endpoint associated

https://dummyapiid-vpce-12345678901234567.execute-api.ap-southeast-2.amazonaws.com/v0/mock

Public DNS name with a Host header

https://vpce-12345678901234567-abcd1234.execute-api.ap-southeast-2.vpce.amazonaws.com/v0/mock with headers Host=dummyapiid.execute-api.ap-southeast-2.amazonaws.com

Public DNS name with the x-apigw-api-id header

https://vpce-12345678901234567-abcd1234.execute-api.ap-southeast-2.vpce.amazonaws.com/v0/mock with headers x-apigw-api-id=dummyapiid

Example of the Lambda function (ApiTesterFunction) execution log

Some notes about Amazon EKS IAM OIDC Provider

Kay — Mon, 18 Apr 2022 23:52:55 +0000

Step 1

iam:*OpenIDConnectProvider* permissions are not required when creating an EKS cluster with CreateCluster, which creates an OpenID Connect provider URL (OpenID Connect issuer URL) for the cluster (e.g. https://oidc.eks.ap-southeast-2.amazonaws.com/id/ABCABC111222333444ABCABC11122233).

And in CloudTrail, there are no *OpenIDConnectProvider* events logged.

Step 2

After (1), the cluster has an OpenID Connect issuer URL associated with it. To use IAM roles for service accounts, an IAM OIDC provider must exist for the cluster. See here.

Then you need to run ekctl utils associate-iam-oidc-provider, e.g.

$ eksctl utils associate-iam-oidc-provider --cluster=k-test-oicd --approve --region=ap-southeast-2 --profile test-oidc

A Open ID Provider with the same URL as (1) is created.

For this step, this role needs to have at least the following permissions.

iam:CreateOpenIDConnectProvider
iam:GetOpenIDConnectProvider
iam:TagOpenIDConnectProvider

CloudTrail does NOT show the events as well (e.g. CreateOpenIDConnectProvider).

X-Ray tracing from SQS to Lambda

Kay — Mon, 08 Feb 2021 23:57:55 +0000

SQS supports X-Ray tracing but it does not propagate the trace to Lambda function. Lambda always starts a new trace with an immutable facade segment.

This is a known issue, see:

aws-xray-sdk-node: https://github.com/aws/aws-xray-sdk-node/issues/208
aws-xray-sdk-dotnet: https://github.com/aws/aws-xray-sdk-dotnet/issues/110

Workaround

There were some workaround discussions in the above issues.

The following is a solution I implemented for a demo:

Create a new segment to replace the facade segment created by Lambda, and
assign with the retrieved trace ID and parent ID from the trace header of the SQS segment.

Example in typescript:

import { Handler, SQSEvent, SQSRecord } from "aws-lambda"
import { Segment, setSegment, utils } from "aws-xray-sdk"
...

// Create a new Segment
const traceHeaderStr = sqsRecord.attributes.AWSTraceHeader
const traceData = utils.processTraceData(traceHeaderStr)
const sqsSegmentEndTime = Number(sqsRecord.attributes.ApproximateFirstReceiveTimestamp) / 1000

const lambdaSegment = new Segment(
    functionName,
    traceData.root,
    traceData.parent
)
lambdaSegment.origin = "AWS::Lambda::Function"
lambdaSegment.start_time = lambdaExecStartTime - (lambdaExecStartTime - sqsSegmentEndTime)
lambdaSegment.addPluginData({
    function_arn: functionArn,
    region: sqsRecord.awsRegion,
    request_id: awsRequestId
})

// Set it as the current Segment
setSegment(lambdaSegment)

// Do something

// Close the segment
lambdaSegment.close()

From Trace Map console:

Example code can be found here on GitHub kyhau/aws-tools/b/X-Ray/xray-sqs-to-lambda/handler.ts.

saml2aws-multi: a simple tool providing an easy-to-use command line interface for saml2aws

Kay — Sat, 30 Jan 2021 00:04:38 +0000

saml2aws-multi is a simple tool I created for using saml2aws more effectively on day-to-day tasks. saml2aws-multi provides an easy-to-use command line interface to support login and retrieve AWS temporary credentials for multiple roles of different accounts with saml2aws.

Example:

Usage

$ awslogin --help
Usage: awslogin [OPTIONS] COMMAND [ARGS]...

  Get credentials for multiple accounts with saml2aws

Options:
  -k, --keyword TEXT              Pre-select roles with the given keyword(s)
  -f, --profile-name-format [RoleName|RoleName-AccountAlias]
                                  Profile name format  [default: RoleName]
  -r, --refresh-cached-roles      [default: False]
  -t, --session-duration TEXT     Session duration in seconds
  -d, --debug                     [default: False]
  --help                          Show this message and exit.

Commands:
  chained  List chained role profiles specified in ~/.aws/config
  switch   Switch default profile
  whoami   Who am I?

For detailed instruction and source see kyhau/saml2aws-multi.

Hactoberfest 2020 Swag Arrived

Kay — Sat, 09 Jan 2021 00:40:56 +0000

Just received my Hacktoberfest 2020 swag. Looking forward to participate again this year.