DEV Community: Sarma

Vector-Database: Qdrant-cluster on ECS-Fargate

Sarma — Wed, 31 Dec 2025 03:36:08 +0000

When do you need this?

When SaaS is not viable due to compliance-reasons or for network-performance.
When Vector-DB should be in the same aws-account as the rest of the solution
When No kubernetes.

Questions? See the article # 2 which has great detail, as to why certain things HAVE to be that way! This article (# 1) just focuses on cdk-snippets to quickly get started (for those very familiar with CDK).

All of this (CDK, ECS & Clusters) is too much to handle?
Seek professional-services from Qdrant.tech; If not, me/my current-employer (an AWS-partner).

Short Summary

Run a 4/or/6-node Qdrant-cluster on ECS, across 2-or-3 AZs
Keep it very simple, by making AWS take care of almost everything re: availability, uptime, AZ-balancing.
No matter how many failed nodes are replaced, use a single "endpoint" (to connect to the Qdrant-cluster).
No EBS. Rely on EFS for "native" Snapshots, for protecting data.
Securing access to Qdrant-Dashboard via ALB + Qdrant-native API-Key.

how do I .. ?

A article # 2 has FULL DETAILs on the Critical-Design & Key-requirements that influenced/constrained/forced the final implementation.
A article # 3 re: Snapshots.
A separate GitLab-repo contains the full CDK-Construct.
Assumption: You'll OK to CUSTOM-build the Qdrant Container-IMAGE (using a custom Dockerfile) using Qdrant's github. article # 4 for a sensible/defensible Dockerfile.

Below is the key "lego-blocks" to get started, for CDK-experts.

Questions? See the article # 2 which has great detail, as to why certain things HAVE to be that way!

Key variables

bashScriptFromArticle4 = .. local-path-inside-docker__to_bash-script-from-article-4 ..;

ecsClusterName = .. .. + '-' + cpuArchStr;
fargateContainerName = .. + '-' + cpuArchStr;
fargateServiceName = 'ECSSvc-'+ fargateContainerName;
taskDefContainerName = 'cont-'+ fargateContainerName;

domainName = ecsClusterName + ".local";
qdrantFQDN = 'qdrant-cluster.'+ domainName;

nativeApiKeySecret = new aws_secretsmanager.Secret( .. );
// excludeCharacters = "~`@#$%^&*+={}[]()|\\:;'\"”<>,/? ";

Fargate Task Definition

new aws_ecs.FargateTaskDefinition( .. .. {
    family: fargateContainerName, //// Equivalent to FargateTaskName/taskname
    cpu: ECS_FARGATE_CPU_SIZING + 512,
    memoryLimitMiB: ECS_FARGATE_MEMORY_SIZING_GB * 1024 + ( 1 * 1024 ),
    enableFaultInjection: (basicProps.tier === 'dev' ),
    ephemeralStorageGiB: 22, // ValidationError: Ephemeral storage size must be between 21GiB and 200GiB
    runtimePlatform: { cpuArchitecture: ecsCpuArchitecture, operatingSystemFamily: aws_ecs.OperatingSystemFamily.LINUX },
    volumes: [{
        name: fargateContainerName,  //// This name -MUST- referenced in the aws_ecs.FargateService.
        efsVolumeConfiguration: {
            fileSystemId: efsFileSystem.fileSystemId,
            rootDirectory: "/",
            transitEncryption: "ENABLED",
            authorizationConfig: { accessPointId: efsAccessPoint.accessPointId, iam: "ENABLED" }
        },
    }],
});

Fargate Service

new aws_ecs.FargateService( .. .., {
    cluster: myECSCluster,
    serviceName: fargateServiceName +'-'+ label,
    taskDefinition: qdrantTaskDefinition,
    desiredCount: desiredCount,
    minHealthyPercent: 0,  // Allow the single task to be stopped during deployment
    maxHealthyPercent: 200, // Allow 1 new task to start before stopping the old one
    vpcSubnets: { onePerAz: true, subnetType: aws_ec2.SubnetType.PRIVATE_WITH_EGRESS, availabilityZones: vpc.availabilityZones },
    securityGroups: [fargateSvcSecurityGroup],
    assignPublicIp: false,
    enableExecuteCommand: true,         // Enable ECS Exec for debugging
    propagateTags: aws_ecs.PropagatedTagSource.SERVICE,
    availabilityZoneRebalancing: aws_ecs.AvailabilityZoneRebalancing.ENABLED,
    circuitBreaker: { enable: true, rollback: true },  // Enable deployment circuit breaker
});

Container for Fargate-Task

Refer to article 4 for how to docker build the Docker-image.

const containerImage = aws_ecs.ContainerImage.fromEcrRepository(ecrRepo, qdrantImageTag);

//// Add Container to Task Definition
dockerLabels = {
    'app.cluster': ecsClusterName,
    'app.service': fargateServiceName,
    'app.task-definition': fargateContainerName, // must match family's name of task-definition
    'app.component': 'vector-database',
    'app.version': cdkContextBuildQdrantImageTag,
    'app.environment': basicProps.tier,
};

portMappings = [
    { containerPort: 6333, protocol: aws_ecs.Protocol.TCP, name: 'qdrant-rest', appProtocol: aws_ecs.AppProtocol.http },
    { containerPort: 6334, protocol: aws_ecs.Protocol.TCP, name: 'qdrant-grpc', appProtocol: aws_ecs.AppProtocol.http },
    { containerPort: 6335, protocol: aws_ecs.Protocol.TCP, name: 'qdrant-cluster', appProtocol: aws_ecs.AppProtocol.http },
];

containerEnvironmentVariables = {
    // runtime environment-variables used by Qdrant-VectorDB docker-container
    QDRANT__CLUSTER__ENABLED: "true",
    QDRANT__SERVICE__API_KEY: nativeApiKeySecret.secretValue.unsafeUnwrap(),
};

qdrantPrimaryNodeTaskDefinition.addContainer( 'QdrantContainer', {
    containerName: taskDefContainerName,
    image: containerImage,
    user: "1000:1000",
    command: [ '/bin/sh', '-c', bashScriptFromArticle4 ],
    cpu: constantsCdk.ECS_FARGATE_CPU_SIZING,
    memoryLimitMiB: ECS_FARGATE_MEMORY_SIZING_GB * 1024, // in MB
    environment: containerEnvironmentVariables,
    dockerLabels: { ...dockerLabels, 'app.instance': 'primary-node' },
    portMappings,
    essential: true,
    logging: containerLogs,
    healthCheck: commonInsideContainerHealthCheck,
});

SecurityGroup

const fargateSvcSecurityGroup = new aws_ec2.SecurityGroup( cdkScope, 'QdrantSecurityGroupForFargate', {
    vpc: vpc,
    securityGroupName: ecsClusterName + '-SvcInbound-' + fargateContainerName,
    description: `For FARGATE-service - inbound 6333,6334 only, NO outbound. Cluster: ${ecsClusterName}, Container: ${fargateContainerName}`,
    allowAllOutbound: false, // Deny/Allow -- Explicitly all outbound traffic
});

//// Add inbound rules for Qdrant ports
fargateSvcSecurityGroup.addIngressRule(aws_ec2.Peer.ipv4(vpc.vpcCidrBlock), aws_ec2.Port.tcp(6333),
    'Allow inbound traffic on port 6333 (Qdrant REST API)'
);
//// Add outbound HTTPS for ECR image pulling
fargateSvcSecurityGroup.addEgressRule(aws_ec2.Peer.anyIpv4(), aws_ec2.Port.tcp(443),
    'Allow outbound HTTPS for ECR image pulling, whether or NOT using VPC Endpoints'
);
//// Cluster-replication traffic; REF: https://qdrant.tech/documentation/guides/distributed_deployment/#enabling-distributed-mode-in-self-hosted-qdrant
fargateSvcSecurityGroup.addIngressRule(fargateSvcSecurityGroup, aws_ec2.Port.tcp(6334),
    'Allow INTRA-CLUSTER Replication traffic on port 6334 (Qdrant gRPC API)'
);
fargateSvcSecurityGroup.addEgressRule(fargateSvcSecurityGroup, aws_ec2.Port.tcp(6334),
            'Allow INTRA-CLUSTER Replication traffic on port 6334 (Qdrant gRPC API)'
);
fargateSvcSecurityGroup.addIngressRule(fargateSvcSecurityGroup, aws_ec2.Port.tcp(6335),
    'Allow INTRA-CLUSTER Replication traffic on port 6335 (Qdrant Cluster-Replication traffic)'
);
fargateSvcSecurityGroup.addEgressRule(fargateSvcSecurityGroup, aws_ec2.Port.tcp(6335),
    'Allow INTRA-CLUSTER Replication traffic on port 6335 (Qdrant Cluster-Replication traffic)'
);

//// Allow Fargate tasks to access EFS
for ( const efsSecurityGroup of efsSecurityGroups ) {
    efsSecurityGroup.addIngressRule(fargateSvcSecurityGroup, aws_ec2.Port.tcp(2049), 'Allow NFS access from Fargate tasks');
    fargateSvcSecurityGroup.addEgressRule(efsSecurityGroup, aws_ec2.Port.tcp(2049), 'Allow NFS access to EFS');
}

Vector-Database: Qdrant-cluster (Fargate) - Dockerfile

Sarma — Wed, 31 Dec 2025 03:33:55 +0000

This is the last of the article set. Go to "Get-Started" article # 1

Why on earth did I chose to create my own Dockerfile?

Technically, yes its "my own". But, its Not a "new" Dockerfile. I just slashed-n-slashed-n-slashed Qdrant's official Dockerfile until I got it to this. It can be simplified further by removing 'GPU' support.
I initially tried using Qdrant's as-is Dockerfile, but that made the CDK-design extremely complicated. I then added a small "starter" bash-script for the container-image, and voila! a simple CDK !
There was NO way I was trusting all the 3rd-party images and 3rd-party packages that Qdrant was installing in their Dockerfile.
My Dockerfile just relies on debian-slim, cargo-chef and mold. That is it. And, adds nslookup only to simplify the cluster deployment-design.
All my base-images HAD to be Verified images from public.ecr.aws. A religious decision. Dockerhub-images over my dead-body.

What if Qdrant's official docker-image is mandatory?

Perhaps your enterprise mandates the use of Qdrant's official ready-to-use Docker-images, rather than using the Dockerfile from a suspicious no-name guy (ahem. me).

OPTION 1: Use Qdrant's official as a "base" image, and incorporate the (following) bash-script into a new "derivative" container-image, and use that bash-script as the "entrypoint". You also need to ensure nslookup cli-command is available in your new image (the bash-script relies on it).

OPTION 2: You'll need 3 Fargate Services (to ensure 2 shards that are replicated across 2-or-3 AZs). Details are out of scope of this article-set. YES, I got this working too initially, but the COMPLEX set of aws-resources was un-maintainable. That led me to the DRASTICALLY-simplified design of just 1 Fargate-Service with just 1 Fargate-Task with just 1 Container (which is what this set of articles is about).

HOW-TO

Copy the bash-script (below) into the Qdrant git-cloned folder. FYI: It must be the "entrypoint" for the running container.
Also, copy the CUSTOM Dockerfile below into the Qdrant git-cloned folder.
docker build the container-image.
Store container-image in an ECR-Repo. Tag it appropriately.

Explaining the "entrypoint" Bash-script

This bash-script does a DNS lookup of qdrant-cluster.my-ecs-cluster.local (stored as the value of bash-variable QdrantClusterFQDN)

if the lookup returns no records, then container should run:

./qdrant --bootstrap http://${QdrantClusterFQDN}:6335

else if 1+ record(s) is/are returned, then container should run:

./qdrant --uri http://${QdrantClusterFQDN}:6335

else, for all other scenarios of the nslookup cmd, dump the response of that command and print an error message

Could Qdrant-corp have taken care of this simple logic within their binary? It would make cluster-setup easier (but then, their livelihood is gone).

Why?

NOTE: Per official documentation, the 1st node must be started as:

./qdrant --bootstrap http://${qdrantPrimaryFQDN}:6335

Except for the very-1st node (that uses the above cmd), all other "nodes/Containers" must be started as:

./qdrant --uri http://${qdrantPrimaryFQDN}:6335

That is, --uri replaces ~~--bootstrap~~

A big-problem happens when the 1st node itself fails and we need a replacement node in its place, in an automated manner! If the replacement-node uses --bootstrap (by definition/configuration), hell breaks loose. All such exceptional scenarios are well handled by the script.

bash-script

Note: replace my-ecs-cluster with the name of YOUR ecs-cluster.
Note: Following is for arm64 with/without Nvidia-Gpu.

#!/bin/bash -f
set -e

MyClusterName=""my-ecs-cluster"
QdrantClusterFQDN="qdrant-cluster.${MyClusterName}.local"

# Install required DNS tools
echo "Installing DNS utilities..."
apt-get update && \
apt-get install -y --no-install-recommends dnsutils && \
apt-get autoremove -y && \
apt-get autoclean && \
rm -rf /var/lib/apt/lists/* /var/cache/apt/* /tmp/* /var/tmp/*

echo "Performing DNS lookup for ${QdrantClusterFQDN}..."
if DNS_RESULT=$(dig +short "$QdrantClusterFQDN" A 2>/dev/null) && [ -n "$DNS_RESULT" ]; then
    RECORD_COUNT=$(echo "$DNS_RESULT" | grep -cE '^[0-9]+\.[0-9]+\.[0-9]+\.[0-9]+$' 2>/dev/null || echo 0)

    if [ "$RECORD_COUNT" -eq 0 ]; then
        echo "No valid IP records found. Bootstrapping totally-new Qdrant cluster..."
        exec ./qdrant --bootstrap "http://${QdrantClusterFQDN}:6335"
    else
        echo "Found ${RECORD_COUNT} DNS record(s). Joining existing cluster..."
        exec ./qdrant --uri "http://${QdrantClusterFQDN}:6335"
    fi
else
    echo "dig command failed or returned empty result. Full output:"
    dig "$QdrantClusterFQDN" A
    echo "FATAL-Error Not able to determine if this Qdrant-node is part of a cluster or not"
    exit 1
fi

Dockerfile

For arm64 cpu-architecture.

Warning: Using an XLARGE Codebuild-instance (32cpus, 128GB RAM), it still took 20 minutes to build the image !!

# Enable GPU support.
# This option can be set to `nvidia` or `amd` to enable GPU support.
# This option is defined here because it is used in `FROM` instructions.
ARG GPU

# Use AWS ECR Public Gallery's official Rust image
FROM public.ecr.aws/docker/library/rust:1.90-bookworm AS base

# Install cargo-chef manually for better security control
RUN cargo install cargo-chef --locked

FROM base AS planner
WORKDIR /qdrant
COPY . .
RUN cargo chef prepare --recipe-path recipe.json

FROM base AS builder
WORKDIR /qdrant

# Install build dependencies
RUN apt-get update \
    && apt-get install -y --no-install-recommends \
        clang \
        lld \
        cmake \
        protobuf-compiler \
        libprotobuf-dev \
        protobuf-compiler-grpc \
        jq \
        pkg-config \
        gcc \
        g++ \
        libc6-dev \
        libunwind-dev \
        curl \
        ca-certificates \
    && apt-get clean \
    && rm -rf /var/lib/apt/lists/* /var/cache/debconf/* \
    && rustup component add rustfmt

# `ARG`/`ENV` pair is a workaround for `docker build` backward-compatibility.
#
# https://github.com/docker/buildx/issues/510
ARG BUILDPLATFORM
ENV BUILDPLATFORM=${BUILDPLATFORM:-linux/arm64}

# Pin mold version and verify checksum for Cpu-Architecture
ARG MOLD_VERSION=2.36.0
# ARG MOLD_SHA256_ARM64="4b20b3fac90ad3f5b5d4c0b65a6e3f4b4f10c6f7f1a8f9a8b1b1b1b1b1b1b1b1"

RUN mkdir -p /opt/mold && \
    cd /opt/mold && \
    TARBALL="mold-$MOLD_VERSION-aarch64-linux.tar.gz" && \
    curl -sSLO "https://github.com/rui314/mold/releases/download/v$MOLD_VERSION/$TARBALL" && \
    tar -xf "$TARBALL" --strip-components 1 && \
    rm "$TARBALL"

    # echo "$MOLD_SHA256_ARM64  $TARBALL" | sha256sum -c - && \


# `ARG`/`ENV` pair is a workaround for `docker build` backward-compatibility.
#
# https://github.com/docker/buildx/issues/510
ARG TARGETPLATFORM
ENV TARGETPLATFORM=${TARGETPLATFORM:-linux/arm64}

# Set protobuf environment variables
ENV PROTOC=/usr/bin/protoc
ENV PROTOC_INCLUDE=/usr/include

# Select Cargo profile (e.g., `release`, `dev` or `ci`)
ARG PROFILE=release

# Enable crate features
ARG FEATURES

# Pass custom `RUSTFLAGS` (e.g., `--cfg tokio_unstable` to enable Tokio tracing/`tokio-console`)
ARG RUSTFLAGS

# Select linker (e.g., `mold`, `lld` or an empty string for the default linker)
ARG LINKER=mold

# Enable GPU support
ARG GPU

# Download and extract web UI
COPY tools/ tools/
COPY docs/ docs/
RUN mkdir /static && STATIC_DIR=/static ./tools/sync-web-ui.sh

# Cook dependencies
COPY --from=planner /qdrant/recipe.json recipe.json
RUN PATH="$PATH:/opt/mold/bin" \
    PROTOC=/usr/bin/protoc \
    PROTOC_INCLUDE=/usr/include \
    CARGO_BUILD_JOBS=32 \
    RUSTFLAGS="${LINKER:+-C link-arg=-fuse-ld=}$LINKER $RUSTFLAGS -C codegen-units=16" \
    cargo chef cook --profile $PROFILE ${FEATURES:+--features} $FEATURES --features=stacktrace ${GPU:+--features=gpu} --recipe-path recipe.json

COPY . .
# Include git commit into Qdrant binary during build
# ARG GIT_COMMIT_ID

### LAPTOP-BUILDS: memory-optimized settings for ARM64 (via --jobs cli-arg to cargo-build)
### Modified RUSTFLAGS to include:
###     -C opt-level=2 instead of the default -C opt-level=3 (less memory intensive)
###     -C codegen-units=4 instead of the default -C codegen-units=1 (allows for better memory distribution)
### For speed-optimized high-spec AWS-CodeBuild (arm64: 32 vCPUs, 64GB RAM)
### RUSTFLAGS optimized for speed without target-cpu=native to avoid SIGILL:
###     -C codegen-units=16 for optimal parallelization with 32 vCPUs
###     -C target-feature=+v8a for ARM64 v8 optimizations
###     -C opt-level=3 for maximum optimization (we have enough memory)
RUN PATH="$PATH:/opt/mold/bin" \
    RUSTFLAGS="${LINKER:+-C link-arg=-fuse-ld=}$LINKER $RUSTFLAGS -C opt-level=3 -C codegen-units=16 -C target-feature=+v8a -C link-arg=-Wl,--threads=$(nproc)" \
    cargo build --profile $PROFILE ${FEATURES:+--features} $FEATURES --features=stacktrace ${GPU:+--features=gpu} --bin qdrant \
        --jobs 32 \
    && PROFILE_DIR=$(if [ "$PROFILE" = dev ]; then echo debug; else echo $PROFILE; fi) \
    && mv target/$PROFILE_DIR/qdrant /qdrant/qdrant
    ### Note: Since we are NOT cross-compiling, we do NOT need `/$(cargo --print-target-triple)` in the path above.

# Generate SBOM
RUN cargo install cargo-sbom --jobs 32 && \
    cargo sbom > qdrant.spdx.json

# Dockerfile does not support conditional `FROM` directly.
# To workaround this limitation, we use a multi-stage build with a different base images which have equal name to ARG value.

# Base image for Qdrant.
FROM public.ecr.aws/docker/library/debian:bookworm-slim AS qdrant-cpu

# ### Only needed if NVIDIA GPU is required !!
# # Base images for Qdrant with nvidia GPU support.
# FROM nvidia/opengl:1.2-glvnd-devel-ubuntu22.04 AS qdrant-gpu-nvidia
# # Set non-interactive mode for apt-get.
# ENV DEBIAN_FRONTEND=noninteractive
# # Set NVIDIA driver capabilities. By default, all capabilities are disabled.
# ENV NVIDIA_DRIVER_CAPABILITIES compute,graphics,utility
# # Copy Nvidia ICD loader file into the container.
# COPY --from=builder /qdrant/lib/gpu/nvidia_icd.json /etc/vulkan/icd.d/
# # Override maintainer label. Nvidia base image have it's own maintainer label.
# LABEL maintainer="Qdrant Team <info@qdrant.tech>"


# ### Only needed if AMD-GPU is required !!
# FROM rocm/dev-ubuntu-22.04 AS qdrant-gpu-amd
# # Set non-interactive mode for apt-get.
# ENV DEBIAN_FRONTEND=noninteractive
# # Override maintainer label. AMD base image have it's own maintainer label.
# LABEL maintainer="Qdrant Team <info@qdrant.tech>"

FROM qdrant-cpu AS qdrant
# ### Only needed if NVidia or AMD-GPU is required !!
# FROM qdrant-${GPU:+gpu-}${GPU:-cpu} AS qdrant

# Install GPU dependencies
ARG GPU

RUN if [ -n "$GPU" ]; then \
    apt-get update \
    && apt-get install -y \
    libvulkan1 \
    libvulkan-dev \
    vulkan-tools \
    ; fi

# Install additional packages into the container.
# E.g., the debugger of choice: gdb/gdbserver/lldb.
ARG PACKAGES

RUN apt-get update \
    && apt-get install -y --no-install-recommends ca-certificates tzdata libunwind8 $PACKAGES \
    && apt-get clean \
    && rm -rf /var/lib/apt/lists/* /var/cache/debconf/* /var/lib/dpkg/status-old

# Copy Qdrant source files into the container. Useful for debugging.
#
# To enable, set `SOURCES` to *any* non-empty string. E.g., 1/true/enable/whatever.
# (Note, that *any* non-empty string would work, so 0/false/disable would enable the option as well.)
ARG SOURCES

# Dockerfile does not support conditional `COPY` instructions (e.g., it's impossible to do something
# like `if [ -n "$SOURCES" ]; then COPY ...; fi`), so we *hack* conditional `COPY` by abusing
# parameter expansion and `COPY` wildcards support. 😎

ENV DIR=${SOURCES:+/qdrant/src}
COPY --from=builder ${DIR:-/null?} $DIR/

ENV DIR=${SOURCES:+/qdrant/lib}
COPY --from=builder ${DIR:-/null?} $DIR/

ENV DIR=${SOURCES:+/usr/local/cargo/registry/src}
COPY --from=builder ${DIR:-/null?} $DIR/

ENV DIR=${SOURCES:+/usr/local/cargo/git/checkouts}
COPY --from=builder ${DIR:-/null?} $DIR/

ENV DIR=""

ARG APP=/qdrant

ARG USER_ID=0

RUN if [ "$USER_ID" != 0 ]; then \
        groupadd --gid "$USER_ID" qdrant; \
        useradd --uid "$USER_ID" --gid "$USER_ID" -m qdrant; \
        mkdir -p "$APP"/storage "$APP"/snapshots; \
        chown -R "$USER_ID:$USER_ID" "$APP"; \
    fi

COPY --from=builder --chown=$USER_ID:$USER_ID /qdrant/qdrant "$APP"/qdrant
COPY --from=builder --chown=$USER_ID:$USER_ID /qdrant/qdrant.spdx.json "$APP"/qdrant.spdx.json
COPY --from=builder --chown=$USER_ID:$USER_ID /qdrant/config "$APP"/config
COPY --from=builder --chown=$USER_ID:$USER_ID /qdrant/tools/entrypoint.sh "$APP"/entrypoint.sh
COPY --from=builder --chown=$USER_ID:$USER_ID /static "$APP"/static

WORKDIR "$APP"

USER "$USER_ID:$USER_ID"

ENV TZ=Etc/UTC \
    RUN_MODE=production

EXPOSE 6333
EXPOSE 6334

LABEL org.opencontainers.image.title="Qdrant"
LABEL org.opencontainers.image.description="Official Qdrant image"
LABEL org.opencontainers.image.url="https://qdrant.com/"
LABEL org.opencontainers.image.documentation="https://qdrant.com/docs"
LABEL org.opencontainers.image.source="https://github.com/qdrant/qdrant"
LABEL org.opencontainers.image.vendor="Qdrant"

CMD ["./entrypoint.sh"]

Leaving the last CMD intact, since we will be overriding entrypoint.

APPENDIX

Get-Started article # 1
A article # 2 has FULL DETAILs on the Critical-Design & Key-requirements that influenced/constrained/forced the final implementation.
A article # 3 re: Snapshots.
A separate GitLab-repo contains the full CDK-Construct.
Assumption: You'll OK to CUSTOM-build the Qdrant Container-IMAGE (using a custom Dockerfile) using Qdrant's github. article # 4 for a sensible/defensible Dockerfile.

/ End

Snapshots & Data-RESTORE: Vector-Database: Qdrant-Cluster

Sarma — Wed, 31 Dec 2025 03:31:37 +0000

This belongs to an article set. Go to "Get-Started" article # 1

To state the obvious, it VERY IMPORTANT to very-frequently TEST the validity of SnapShots - manually. All Vector-databases are Not as robust as good ol' SQL-Databases, to fully trust the backups/snapshots.

The following manual steps walk you thru' (first) how to take/make snapshots and then to "restore/recover" from a snapshot.

(A) HOW-TO - creating snapshots

OPTION 1: in an AUTOMATED manner

We need a new Lambda. See sample code at bottom.

Option 2: Manually

If you want to MANUALLY trigger a snapshot of an entire-collection, just simply use the Qdrant's built-in RestApi:

POST /collections/${collection_name}/snapshots

Do this either via your choice of RestApi-client, or via Qdrant's Dashboard (via ALB like httpS://my-custom-alb-domain.mycompany.com/dashboard)

(B) HOW-TO:- viewing snapshots

Use the Qdrant-Dashboard:
1. Important: Use the API-Key/Token for security!!
2. Important: Use httpS !
Invoke POST collections/${collection_name}/points/scroll
- Save post-response to a temporary local file, to help you track what you will do per this article.
Confirm (using the post-response) that you have a NEW snapshot, that we shall refer to as snapshot # 1 (a.k.a. baseline ) of the collection.
- FYI: You can easily recognize snapshots based on the TIMESTAMP that is part of the snapshot's name.
If relying on automated-snapshots, wait for the next new snapshot to be created.

(C) HOW-TO:- prep for VERIFYING the snapshots

Make a change to your vector database.
- If you are using Qdrant's test_collection, then insert a new Point: PointStruct(id=6, vector=[0.30, 0.05, 0.10, 0.40], payload={"city": "Bengaluru"}) (Don't ask me about the numbers)
Trigger a new snapshot (manually or via scheduled-automation).
POST collections/${collection_name}/points/scroll (repeat)
- --COMPARE-- this response with the contents of a temporary local file.
Confirm you have a NEW snapshot (called snapshot # 2) of the collection (AFTER the above update).

(D) HOW-TO:- Restore/Recover

Snapshots stored locally inside Qdrant-cluster.

FYI: Qdrant stores ALL snapshots that you create/trigger, under /qdrant/snapshots, but YOU should rely on Qdrant's API (../scroll) to list/access/download/upload snapshots.

Invoke the following POST api (with json-body) to do a restore using baseline a.k.a. snapshot # 1 :-

PUT /collections/${collection_name}/snapshots/recover`
{
  "location": "file:///qdrant/storage/snapshots/${collection_name}/${collection_name}-YYYY-MM-DD-HH-MM-SS.snapshot"
}

Do a query to verify that data change (that you did in sub-section (C)) .. is MISSING !
Restore from an LATEST snapshot # 2.
Do a query to verify that data change (that you did in sub-section (C)) .. is BACK !

From s3-bucket

Warning: The Fargate-Task's(Container) role will need S3-access. Out of scope of this article.

Restore from an old snapshot # 1/baseline using this example:-

PUT /collections/${collection_name}/snapshots/recover`
{
    "location": "https://??????????.s3.us-east-2.amazonaws.com/${collection_name}-1046358132872257-2025-12-10-20-31-22.snapshot"`
}

Do a query to verify that data change (that you did in sub-section (C)) .. is MISSING !
Restore from an LATEST snapshot # 2.
Do a query to verify that data change (that you did in sub-section (C)) .. is BACK !

(E) HOW-TO:- Lambda to create AUTOMATED-Snapshots

This Lambda will trigger snapshots periodically, and will also COPY those snapshots to an S3-bucket. FYI: I instead prefer mounting an EFS-FileSystem onto /qdrant/snapshots/ of the Container (runnning as a Fargate-Task), and having EFS replicated to a 2nd region.

import os
import json
import boto3
import traceback
from datetime import datetime, timezone

import requests

from aws_lambda_powertools import Logger, Tracer, Metrics
from aws_lambda_powertools.logging import correlation_paths
from aws_lambda_powertools.metrics import MetricUnit

logger = Logger()
tracer = Tracer()
metrics = Metrics()

s3_client = boto3.client('s3')
secrets_client = boto3.client('secretsmanager')

### -------------------------------------------
def get_api_key(secrets_manager_arn: str) -> str:
    """Retrieve API key from AWS Secrets Manager"""
    if not secrets_manager_arn or secrets_manager_arn.strip() == "":
        raise ValueError("Secrets Manager ARN not provided")

    logger.info(f"Retrieving API key from Secrets Manager ARN: {secrets_manager_arn}")
    response = secrets_client.get_secret_value(SecretId=secrets_manager_arn)
    api_key = response['SecretString'].strip()

    if not api_key:
        raise ValueError("API key not found in secret")

    logger.info("API key retrieved successfully ✅")
    return api_key

### -------------------------------------------
def get_qdrant_url(qdrant_fqdn: str) -> str:
    """Construct Qdrant URL using Service Discovery FQDN"""
    if not qdrant_fqdn:
        raise ValueError("QDRANT_FQDN environment variable not set")

    qdrant_url = f"http://{qdrant_fqdn}:6333"
    logger.info(f"Qdrant URL: {qdrant_url}")
    return qdrant_url

### -------------------------------------------
def determine_snapshot_frequency() -> str:
    """
    Determine which snapshot frequency to use based on current time
    Returns: '15min', 'hourly', 'daily', or 'monthly'
    """
    now = datetime.now(timezone.utc)

    # Monthly: 1st of month at 8 AM UTC
    if now.day == 1 and now.hour == 8 and now.minute < 15:
        return 'monthly'

    # Daily: Every day at 8 AM UTC
    if now.hour == 8 and now.minute < 15:
        return 'daily'

    # Hourly: Top of every hour
    if now.minute < 15:
        return 'hourly'

    # Default: 15-minute intervals
    return '15min'

### -------------------------------------------
def get_collections(qdrant_url: str, api_key: str) -> list:
    """Get list of all collections"""
    collections_url = f"{qdrant_url}/collections"
    headers = {"api-key": api_key}

    logger.info(f"Getting collections from {collections_url}")
    response = requests.get(collections_url, headers=headers)

    if response.status_code != 200:
        raise Exception(f"Failed to get collections: {response.status_code} - {response.text}")

    collections_data = response.json()
    collections = [col["name"] for col in collections_data.get("result", {}).get("collections", [])]
    logger.info(f"Found collections: {collections}")
    return collections

### -------------------------------------------
def create_collection_snapshot(qdrant_url: str, collection_name: str, api_key: str) -> dict:
    """
    Create a snapshot for a specific collection
    Returns snapshot metadata
    """
    snapshot_url = f"{qdrant_url}/collections/{collection_name}/snapshots"
    headers = {"api-key": api_key}

    logger.info(f"Creating snapshot for collection '{collection_name}' at {snapshot_url}")
    response = requests.post(snapshot_url, headers=headers)

    if response.status_code not in [200, 201]:
        raise Exception(f"Failed to create snapshot for collection '{collection_name}': {response.status_code} - {response.text}")

    snapshot_data = response.json()
    logger.info(f"Snapshot created for collection '{collection_name}': {json.dumps(snapshot_data)}")

    return snapshot_data

### -------------------------------------------
def download_collection_snapshot(qdrant_url: str, collection_name: str, snapshot_name: str, api_key: str) -> bytes:
    """Download collection snapshot from Qdrant"""
    download_url = f"{qdrant_url}/collections/{collection_name}/snapshots/{snapshot_name}"
    headers = {"api-key": api_key}

    logger.info(f"Downloading snapshot from {download_url}")
    response = requests.get(download_url, headers=headers, stream=True)

    if response.status_code != 200:
        raise Exception(f"Failed to download snapshot: {response.status_code}")

    return response.content

### -------------------------------------------
def upload_to_s3(bucket_name: str, s3_key: str, data: bytes) -> None:
    """Upload snapshot to S3"""
    logger.info(f"Uploading to s3://{bucket_name}/{s3_key}")
    s3_client.put_object(Bucket=bucket_name, Key=s3_key, Body=data)
    logger.info(f"Upload complete ✅")

### -------------------- main -----------------------
@logger.inject_lambda_context(correlation_id_path=correlation_paths.API_GATEWAY_REST)
@tracer.capture_lambda_handler
@metrics.log_metrics(capture_cold_start_metric=True)
def lambda_handler(event, context):
    """
    Lambda function to create and manage Qdrant snapshots
    Implements RPO of 15 minutes with S3 lifecycle-based retention
    """

    logger.info('^' * 160)

    try:
        # Get environment variables
        secrets_manager_arn = os.getenv('AK_ARN')
        if not secrets_manager_arn:
            raise ValueError("AK_ARN environment-variable NOT set")
        qdrant_fqdn = os.getenv('QDRANT_FQDN')
        if not qdrant_fqdn:
            raise ValueError("QDRANT_FQDN environment-variable NOT set")
        tier = os.getenv('TIER')
        if not tier:
            raise ValueError("TIER environment-variable NOT set")
        cpu_arch = os.getenv('CPU_ARCH')
        if not cpu_arch:
            raise ValueError("CPU_ARCH environment-variable NOT set")
        # ecs_cluster_name = os.getenv('ECS_CLUSTER_NAME')
        # fargate_container_name = os.getenv('FARGATE_CONTAINER_NAME')

        # Validate required environment variables
        # if not all([secrets_manager_arn, qdrant_fqdn, tier, cpu_arch, ecs_cluster_name, fargate_container_name]):
        if not all([secrets_manager_arn, qdrant_fqdn, tier, cpu_arch]):
            missing = [k for k, v in {
                'AK_ARN': secrets_manager_arn,
                'QDRANT_FQDN': qdrant_fqdn,
                'TIER': tier,
                'CPU_ARCH': cpu_arch,
                # 'ECS_CLUSTER_NAME': ecs_cluster_name,
                # 'FARGATE_CONTAINER_NAME': fargate_container_name
            }.items() if not v]
            raise ValueError(f"Missing required environment variables: {', '.join(missing)}")

        # Get API key and Qdrant URL
        api_key = get_api_key(secrets_manager_arn)
        qdrant_url = get_qdrant_url(qdrant_fqdn)

        # Determine snapshot frequency
        frequency = determine_snapshot_frequency()
        logger.info(f"Snapshot frequency: {frequency}")

        # Get collections
        collections = get_collections(qdrant_url, api_key)

        if not collections:
            logger.info("No collections found - skipping snapshot creation")
            return {
                'statusCode': 200,
                'headers': {'Content-Type': 'application/json'},
                'body': json.dumps({
                    'message': 'No collections found - no snapshots created',
                    'frequency': frequency
                })
            }

        # Get S3 bucket name
        bucket_name = os.getenv('SNAPSHOT_S3_BUCKET_NAME')
        if not bucket_name:
            raise ValueError("SNAPSHOT_S3_BUCKET_NAME environment-variable NOT set")

        # Create snapshots for each collection
        snapshots_created = []
        total_size = 0

        for collection_name in collections:
            try:
                # Create collection snapshot
                snapshot_data = create_collection_snapshot(qdrant_url, collection_name, api_key)
                snapshot_name = snapshot_data.get('result', {}).get('name')

                if not snapshot_name:
                    logger.error(f"Snapshot name not found for collection '{collection_name}'")
                    continue

                # Download snapshot
                snapshot_bytes = download_collection_snapshot(qdrant_url, collection_name, snapshot_name, api_key)
                logger.info(f"Downloaded snapshot for '{collection_name}': {len(snapshot_bytes)} bytes")

                # Upload to S3
                timestamp = datetime.now(timezone.utc).strftime('%Y%m%d-%H%M%S')
                s3_key = f"{tier}-{cpu_arch}/{frequency}/{collection_name}/{snapshot_name}"
                # s3_key = f"{tier}-{cpu_arch}/{frequency}/{collection_name}/{timestamp}-{snapshot_name}"
                upload_to_s3(bucket_name, s3_key, snapshot_bytes)

                snapshots_created.append({
                    'collection': collection_name,
                    'snapshot_name': snapshot_name,
                    's3_location': f"s3://{bucket_name}/{s3_key}",
                    'size_bytes': len(snapshot_bytes)
                })
                total_size += len(snapshot_bytes)

            except Exception as e:
                logger.error(f"Failed to create snapshot for collection '{collection_name}': {str(e)}")
                continue

        # Emit metrics
        metrics.add_metric(name="SnapshotsCreated", unit=MetricUnit.Count, value=len(snapshots_created))
        metrics.add_metric(name="TotalSnapshotSize", unit=MetricUnit.Bytes, value=total_size)

        return {
            'statusCode': 200,
            'headers': {'Content-Type': 'application/json'},
            'body': json.dumps({
                'message': f'Collection snapshots created successfully',
                'frequency': frequency,
                'snapshots_created': len(snapshots_created),
                'collections': collections,
                'snapshots': snapshots_created,
                'note': 'Retention managed by S3 lifecycle policies'
            })
        }

    except Exception as e:
        traceback.print_exc()
        logger.exception("❌ Error creating snapshot")
        metrics.add_metric(name="SnapshotError", unit=MetricUnit.Count, value=1)

        return {
            'statusCode': 500,
            'headers': {'Content-Type': 'application/json'},
            'body': json.dumps({'error': f'Error creating snapshot: {str(e)}'})
        }

APPENDIX

Get-Started article # 1
A article # 2 has FULL DETAILs on the Critical-Design & Key-requirements that influenced/constrained/forced the final implementation.
A article # 3 re: Snapshots.
A separate GitLab-repo contains the full CDK-Construct.
Assumption: You'll OK to CUSTOM-build the Qdrant Container-IMAGE (using a custom Dockerfile) using Qdrant's github. article # 4 for a sensible/defensible Dockerfile.

/ End

Design: Vector-Database: Qdrant-Cluster on Fargate

Sarma — Wed, 31 Dec 2025 03:00:59 +0000

This is 2nd part of a set of articles:

Get-Started article # 1
This article # 2 has FULL DETAILs on the Critical-Design & Key-requirements that influenced/constrained/forced the final implementation.
A article # 3 re: Snapshots.
A separate GitLab-repo contains the full CDK-Construct.
Assumption: You'll OK to CUSTOM-build the Qdrant Container-IMAGE (using a custom Dockerfile) using Qdrant's github. article # 4 for a sensible/defensible Dockerfile.

All of this (CDK & Clusters) is too much to handle? Seek professional-services from Qdrant.tech; If not, me/my current-employer.

Summary of Challenges:

This is a Database!
Need multi-AZ resiliency when cluster is created.
Must automatically maintain multi-AZ resiliency even after innumerable Node-Failures; How to delegate to AWS (ECS), to maintain this multi-AZ balancing by automatically REPLACING nodes, ensuring resiliency & DLP (data-loss protection).
Need high-speed robust multi-AZ replication (of storage/data)
Avoid re-inventing the wheel, that too .. on AWS Cloud
Avoid deploying any of my own mechanisms for:-
1. Resiliency, incl. automatically spinning up a new "node", if any node becomes unhealthy.
2. inter-node / inter-process Communication
3. Replication of Storage-layer or just database-data
4. Monitoring & Alerting
Storage-Persistence challenges
1. Qdrant, like any database, does file-Locking (highly parallel threads attempting to lock the file-system)
2. Multiple copies of data across AZs in sync all the time, almost instantly ready to copy to a new replica-node on any AZ-outage.
3. EBS Volumes can NOT be replicated to another AZ or to another Region.
4. A "crash" (of a single-node Qdrant-Vector-DB) can leave the contents of the EBS Volumes in a "failed-data-integrity" status that may prevent the replacement node from spinning up; This is a classic "Crashed-Database problem" which has NO simple solution.
Self-managed cluster:
1. Using Fargate-Tasks/Containers as "nodes" (instead of traditional EC2s)
2. Qdrant requires a cluster's "nodes/tasks/containers" .. .. to be spun up in certain SEQUENCE; in a specific order.
  - Worse, Qdrant MANDATES that each SUBSEQUENT "node/task/container" to be given information about ANY one of the "preceding" ones.
3. If the 1st Qdrant "node/task/container" that was created goes down, replacing it requires special-handling! That is, replacement of "startup/1st" node/task/container uses a different Entrypoint command (compared to the "origina" Entrypoint).
4. Fargate-Tasks/Containers, by design, have ephemeral PRIVATE-IPs. That's good!
5. Database comes with a "Qdrant-Dashboard" (website); Its built-in into the database! And, we need access to it, but do NOT want the database to use public-IPs!
6. Snapshots-based restore/recovery (usually a manual activity) requires use of this "Qdrant-Dashboard".
7. Challenge: Fargate tasks have ephemeral private IPs, no predictable FQDNs/server-names. Connecting all "nodes/tasks/containers" together as a cluster
  - Best Solution is ECS Service Discovery :- It creates predictable DNS names (e.g., qdrant-cluster.my-ecs-cluster-name.local)
  - Alternatives are: Parameter Store Coordination , EFS-Based Coordination (Leverage existing EFS) , ALB + Health Check Discovery (Use load balancer for discovery)
8. Blame "Raft" protocol and/or Qdrant, but .. .. you are REQUIRED to fully bring up 1st node/task/container, and then wait for it to finish initialization, before bringing up any other node/task/container.
  - Can it be designed simpler !? Sure! You are welcome anytime to put in a PR on Qdrant's official-GitHub.
9. No more EFS (instead of EBS) as primary filesystem, for ALL Task-instances/Containers. Reason: Qdrant will detect it and stop working.

Must-have Requirements

Per Qdrant's documentation, with their reliance on "Raft" protocol, they recommend a cluster of minimum 3 "nodes/tasks/containers". Studying Qdrant's documentation clarifies that, it should be 3 per shard, and that for future-proofing they recommend starting with 2 shards per collection.
- So, in total 6! That is, 2 nodes/containers/tasks in each AZ, each containing a different Shard. Thereby, each Shard is replicated across 3 AZs.
- You MAY be able to make do with 4, but RAFT's behavior needs studying.
Backup Snapshots
1. Must have 100% integrity.
2. Ideally, avoid taking down the "database/cluster/writers" before taking a snapshot
3. Leverage the database's INHERENT-ability to create snapshots.
4. Avoid storage-level snapshots (given the decades-long history of the "Crashed-Database problem")
Maintanence Scenarios:-
1. Automation of snapshots as appropriate for D.R. R.P.O; We chose an RPO of 15 minutes.
2. RETENTION: Snapshots made every 15 minutes; Last 24 hourly-snapshots to be retained; Last 30 daily-snapshots to be retained; Last 12 monthly-snapshots to be retained.
3. Based on your IT-process sophistication and based on the size of your collection(s), choose between Shard-level snapshots -versus- collection-level snapshots. We chose collection-snapshots. Note: Avoid storage-level snapshots.
4. Resilient-storage of snapshots across regions. FYI: Typical choices are S3, EFS & AWS-Backup. "Standard" EBS can Not do this.
Security:
1. httpS from outside Vpc, via a single-endpoint no matter what changes happen within Qdrant-cluster.
2. "native" ApiKey, to protect the "Qdrant Dashboard" & Qdrant-apis.
Monitoring:
1. Basic "node" monitoring (O/S level)
2. Database monitoring (Qdrant-API-level)
3. Shard-health monitoring (Qdrant-API-level)
4. Replacement-of-nodes monitoring
5. Periodic "101% Healthy" monitoring-checks (actually trying to create a new "test-collection", insert data & run a query).
Software Licensing (for use within a Proprietary product sold to Clients)
1. Will Qdrant's licensing support this Enterprise-grade Cluster being used in a commercial-product? Answer is yes, as of Dec 2025.
Advanced:
1. if I have a large-cluster, then I should be utilizing every single Qdrant-node (for faster "sharded" queries).
2. Not sure how Qdrant leverages the 3-copies (across 3 AZs) of each shard for improving query-performance.
3. But, should you ever want to use a VPC-Lambda to split-up queries across ALL Qdrant-nodes, the Lambda must be able to get the list of PRIVATE-IPs of each Qdrant-node.

Decisions

Use pure CDK / pure CloudFormation.
Avoid Custom-Resources in CDK/CloudFormation at all costs.
Avoid having to "do stuff" after deployment of aws-resoures. Avoid using Lambdas completely w.r.t. ANY Cluster modifications (post-deployment).
1. The cluster will need to run 24x7x365 and must self-adjust. Having lambdas run periodically is Never viable for resilience.
2. Periodically doing Scale-up/down will significantly raise the risk of Data-Loss. No more Scaling-down on a schedule, daily.
  1. Mechanism to take a snapshot (of --EACH-- shard or of --EACH-- collection). This is important due to implications of SHARDING.
  2. Ability to save SnapShots onto an EFS-filesystem (either on-demand or scheduled)
  3. Restore a collection from a SnapShot instead (avoid DIY).
These articles require you to custom-build your own Docker-container-image (from Qdrant's source-code in GitHub).
- If your personal-requirements/constraints are forcing you to avoid changing/enhancing the Dockerfile (so that any "drop-in" official Qdrant image will also work) .. you'll then need 3 different ECS-services. Details are out of scope of these articles.
Only production will use a Qdrant-6-node-Cluster. All other environments will use a single-node Qdrant-container/Task (which has a scheduled daily downtime).
Should clusters be considered too expensive, then a single-node Qdrant Vector-DB along with Snapshots can potentially support an RPO of 15-minutes, but this will require automation to thoroughly test the "snapshots".
Each Fargate-Task will have just one Container; In these articles, they are synonymous.
Native non-storage SnapShots is mandatory from day 1, needed no matter what the final architecture & design.
Snapshots (with zero-overhead) should be automatically stored resiliently across regions. Choices are S3, EFS & AWS Backup. Simplest is EFS (with multi-region replication), by mounting to /qdrant/snapshots into the container, so we chose EFS JUST for storing snapshots only.

End.

Stop using ANY Container-image

Sarma — Sat, 04 Oct 2025 18:48:03 +0000

Summary

Stop using:

docker run --privileged --rm tonistiigi/binfmt --install all ;

❌ Warning !! This is an unverified 3rd-party container-image
Speaking in his favor, Tonis Tiigi puts in a lot of effort 👍 to keep this Container-image updated.
FYI: Life and software is a marathon that no single person can keep doing all alone for long.

docker run --rm --privileged public.ecr.aws/k1t8c0v2/multiarch/qemu-user-static:latest --reset -p yes

❌ Warning !! Un-verified 3rd-party container-image. It is NOT being supported❌ and became outdated❌ a few years ago.

docker run --rm --privileged multiarch/qemu-user-static --reset -p yes

This .. is was .. (in contrast to tonistiigi) community-supported. As is so common, it is NOT being supported❌ and became outdated❌ a few years ago.

Instead (for 2026) use ✅

    sudo apt update ;
    sudo apt install -y qemu-system-arm qemu-user-static binfmt-support ;

    docker run --privileged --rm public.ecr.aws/eks-distro-build-tooling/binfmt-misc:qemu-v7.0.0 --install all ;

How can you simply test/verify, if this will work for you?
See Appendix below.

Switch from apt to dnf or other package-manager as appropriate for your needs.

Why bother?

(Want an alternative to -NOT- do the above, see next section.)

This is a Security topic. Either you get it, and eagerly (repeat: eagerly) implement the recommendations.
Or, you are part of the problem.

Here's another way to look at this article:

You: But.. but.. Mx. Police officer! Everyone else is going 25xph over the speed limit.
Police Officer: Fact is -YOU- are going 25xph over the speed limit. Here's your speeding ticket/चालान.

If you'd like to STILL argue/defend yourself about how the speeding-ticket says you being discriminated against ..
.. stop reading this.

This is a Security topic. Either you get it, and eagerly (repeat: eagerly) implement the recommendations.
Or, you are part of the problem.

How many times have I seen people point this out:- That His Excellency, the current US President, frequently says "I've heard SO MANY other people say.. .." on something most people have Not heard about.
We -ALL- do the SAME EXACT thing (like demonstrated in this article below) sooo frequently .. but, Mr. President, we are holier than thou.

This is a Security topic. Either you get it, and eagerly (repeat: eagerly) implement the recommendations.
Or, you are part of the problem.

"One guy" (one of us) writes -ONE- blog about using a container owned by tonistiigi.
Another "One guy" (one of us) writes -ONE- post about using a container owned by k1t8c0v2.
For years on end, ..
(repeating it) For years on end, ..
developers keep repeating "I've heard -SO MANY- people use .. .. .."
Exactly like in an echo chamber, these 3rd-party container-images get broadcast all over, and magically they becomes trustworthy!

In this day and age, where the lineage of SBOM is so important, why do you use 3rd party container-images?

What is "SBOM", you ask?!
You are part of the problem.

Alternative

There is an alternative to NOT doing anything from this article.
This alternative was so well demonstrated during a recent media-covered DATA-LEAK incident about a "Tea App".
On face-value, it appears that the Product-Manager(s), senior-most management and/or developers did Nothing to prevent the leak.
No Security whatsoever.
Everything about women expressing private opinions, including their IDs, were exposed. The women were humiliated.

So, do you know what the company owning "Tea App" did?
They took advantage of the narrative, that the "Men who were detailed in the Tea App, conspired with the so many other Men of this world, and hacked the app, to take some kind of revenge on those women".
"The product-manager(s), senior-management and/or developers were GOOD people with the proverbial golden ring HALO above their heads".

"The road to hell is paved with good intentions" is a meaningless jumble of words.

Bottomline: Instead of "fixing potential security" problems, get a Communication Major/Degree and "indulge in Media Spin" like the above couple of paras, instead.
That is the only other alternative (to this article) that I am aware of.

Why should I trust the above recommendation?

AWS Owns and Maintains this specific container image.
AWS uses it in the context of EKS (at least for now).
Added benefit: Within AWS, you will Not hit limits on your "docker pull/run.." !
- FYI: I use it extensively in my Pipelines, which pull this image so so many times.

Appendix - How to SIMPLY confirm that the replacement works?

Within an O/S on X86 ..

    OtherChipArch="linux/arm64"
    sudo update-binfmts --display | grep arm ;
    docker buildx ls ;

    echo "Sanity-check the ARM64-chipset execution via X86-Docker-in-Docker..."
    docker run --rm --privileged --platform ${OtherChipArch}  public.ecr.aws/sam/build-python3.12:latest python --version

You know how to do the other way around.

Appendix -- For those who love mega-deep-dives

The ABOVE-docker-cmd installs the QEMU binaries + registers them with binfmt_misc, enabling QEMU to execute non-native file formats for emulation

    ## These `update-binfmt` cmds are --NOT-- sufficient.  We need the "PRIVILEGED" docker-cmd !!!
    ## Register QEMU interpreters for multi-platform builds
    ## CLI-manpage: https://manpages.debian.org/testing/binfmt-support/update-binfmts.8.en.html
    sudo update-binfmts --enable qemu-arm ;
    sudo update-binfmts --enable qemu-armeb ;
    sudo update-binfmts --display | grep arm ;

As learn more fine nuances, I'll add it here.

/ EoF

Running AWS Glue locally on a Windows Laptop

Sarma — Tue, 16 Jan 2024 03:58:39 +0000

by Udaybhaskar Sarma Seetamraju
ToSarma@gmail.com
Dec 31 2023

Highest-level Context

If you are into “Shift-Left” (whether re: Testing, Security, or Replicating-problems-on-developer-laptop, etc ..), then this article is for you.

Objective: Towards enabling up to 5x developer-productivity by allowing developers to robustly SIMULATE the Cloud-environment on a Windows-Laptop.

Quick Summary

Aiming for very simple single command, based on python-scripts --> to execute your python-code as a Glue Job’s build locally on your Microsoft-Windows Laptop.

Even if your company disallows AWS CLI credentials, I have a section on how to significantly raise your productivity, in debugging/developing your python-code.

To state the obvious, everything in here is 100% Python-Scripts.

Note: You should aim to have your software work on arm64 containers, which invariably is cheapest compute on cloud. More below.

Problem Statements

Using Git for capturing ALL code-changes while simultaneously copying-n-pasting into AWS Glue-Studio (for testing/troubleshooting) is painful, error prone and frustrating.
As time progresses, your Glue Job will become complicated and require more than one Python-script.Worse, you have one or more folder-hierarchies, all of which contain PY files that you need to import!
Many developers prefer to develop/test/troubleshoot their code as “plain python”, and Not as a Glue-Job.There is No good reason to deny such developers from doing just that.. .. while ensuring that code will work without any issues inside AWS Glue running locally on Windows-Laptop, and eventually work without issues inside Glue on AWS.
1. When running as “plain python”, there should be No runtime dependencies (like “import awsglue”).
2. When running as “plain python”, there should be No spark-dependency.
3. When running as “plain python”, all inputs/files should be on local laptop’s filesystem. All output should be written to local filesystem only.
4. When running as “plain python”, all information from Glue-Catalog should be available OFFline (as a Python Dict object)
How to proactively ensure the Glue Job will work on all chip-architectures - without having to scramble later? How to explicitly utilize all x86_64/amd64/arm64 architectures locally on your Windows-Laptop?
If the Enterprise does Not allow Laptops to have AWS-Credentials (in ~/.aws/credentials file);Even so, how can I EFFICIENTLY test/debug the python-code file locally on my laptop, even as it needs access to Glue Catalog and/or S3 buckets?

Get started!

Based on your needs on AWS choose between these 2 lines below:

$env:BUILDPLATFORM="linux/amd64"
$env:BUILDPLATFORM="linux/arm64"

Then run these commands in a Powershell window:

$env:DOCKER_DEFAULT_PLATFORM="${BUILDPLATFORM}"
$env:TARGETPLATFORM="${DOCKER_DEFAULT_PLATFORM}"

$WORK_AREA=~
cd $WORK_AREA
git clone https://gitlab.com/tosarma/macbook-m1.git

To try out a sample ..

cd macbook-m1
cd AWS-Glue/src

pip3 install docker
python3 $WORK_AREA/macbook-m1/AWS-Glue/bin/run-glue-job-LOCALLY.py  sample-glue-job.py

WARNING: Without the benefit of “docker cli”, you get ZERO visibility into the progress of docker-activity. This is due to use of un-friendly Docker’s Python APIs, because of which the python-code _ WILL _ _ HANG _ for a long time!

To repeat, “run-glue-job-LOCALLY.py” will hang with NO output, for roughly 2-to-5 minutes (depending on how much CPU and MEMORY you have allocated to the Docker-Desktop, as well as speed of your internet connection).

Important: I only tested using Python3.11; No other Python version tested.

Ready to use it for your own Glue-Script?

First, read the full details within the macbook-m1/AWS-Glue/README.md file.
Copy the *.py files in the macbook-m1/AWS-Glue/src/common subfolder into --> YOUR project’s TOPMOST-folder.
- ATTENTION: the files ./src/common/*.py must exist in your project, after you are done copying.
Make sure to edit your PY file, to look like the example file provided (sample-glue-job.py)
From your project root, run:

cd <your-project-ROOT-folder>

python3 $WORK_AREA/macbook-m1/AWS-Glue/bin/run-glue-job-LOCALLY.py `
          path/to/your/file.py

If you are having problems importing the new files under ./src/common, then try adding this command below and then retry the above command.

$env:PYTHONPATH="your-project's-root-folder"

Is your Python code-base consisting of multiple files across multiple folder-hierarchies?
See section below titled “Complex folder-hierarchies?”

Want to change the CLI-arguments?

Three simple steps:

Edit the file macbook-m1/AWS-Glue/src/common/cli_utils.py
Look inside “process_all_argparse_cli_args()” and make changes in that function.
Look inside “process_std_glue_cli_args()” and make changes inside that function.
- Note: make sure to make similar changes in above steps 2 & 3.

Example:-

You would like to support a new cli-arg as:
- --JOB_NAME 123_ABC
Insert a new line at (say) line # 125 for JOB_NAME.
- This will ensure your code will get the value 123_ABC when running __ INSIDE __ AWS-Glue !!
Insert a new line at (say) line # 78 for --JOB_NAME
- This will allow you to run your python-code as a PLAIN python-command and read this CLI-arg.
- See more re: this in a following section titled “running as a PLAIN python-command”

Never edit the file:
macbook-m1/AWS-Glue/src/common/glue_utils.py

The files “common.py” and “names.py” in that same folder can be edited. Feel free to play around with them.

Tips, Issues & Errors

See Appendix sections, for tips on configuring Docker-DESKTOP.

Question: Want to automatically cleanup/delete the Docker-containers - after they exit?
Answer: INSERT the cli-arg “--cleanup” BEFORE the python-script-name, to that “run-glue-job-LOCALLY.py” script.

Advanced User - Complex folder-hierarchies?

Is your Python code-base consisting of multiple files across multiple folder-hierarchies?
Are you aware that Glue requires you to ZIP up all those OTHER python-files into a single Zip-file?
FYI only - this requirement is driven by Spark!

That script “run-glue-job-LOCALLY.py” will automatically do that for you --> that is, it will automatically look UNDER the current-working-directory, and find all **/*.py files and put them in a temporary ZIP-file.
The script will then automatically pass it on to Glue-inside-Docker (running on your laptop).

If you need to import PY files in parent/ancestor levels, you are in an UN-fortunate NO-Solution sitution. FYI: Unlike on MacOS/Linux, you are _ NOT _ able to take advantage of MacOS/Linux “symlink” (“ln -s”) to those parent/ancestor files, where git CLI can preserve these “symlinks” as exactly just that.

No AWS Credentials on your laptop?

For security-reasons, many companies are denying developers the AWS-credentials for AWS-CLI use.

That means you have a showstopper -> re: locally testing/debugging your python-code, for scenarios like:

COPY INPUT-files from S3-buckets --> into the “current-working directory”.
COPY OUTPUT-files from the “current-working directory” --> into S3-buckets.
Lookup Glue Catalog.
.. etc ..

To workaround this restriction ..
You need to write code that detects whether its running on a Windows-Laptop -versus- actually running inside AWS-Cloud.
In other words, you need to “Short-Circuit” all that code that interacts with AWS-APIs (Glue-Catalog, S3, ..) and mock the expected response from those AWS-APIs.

If you use my script “run-glue-job-LOCALLY.py”, it automatically sets an environment-variable called “running_on_LAPTOP” when running your python-code inside a Docker-Glue container on your laptop!!

How-To “short-circuit”:

if ( os.environ.get('running_on_LAPTOP') ):
    print( "!!!!!!!!!!!!!!!!! running on laptop !!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!" )
    .. ### assume S3-get is already done and file is available in current-directory
    .. ### assume Glue-Catalog-Query is already done and ..
       ###      the "JSON-Response" is available in current-directory as a JSON-file
    ..
else:
    ..
    ..

To state the obvious, on AWS-Cloud, AWS-GLUE does __ NOT __ support environment-variables.
So, this environment-variable called “running_on_LAPTOP” will be UN-defined when running inside AWS-Cloud.

Running as a plain python command

EXAMPLE:
I’m going to use the same “macbook-m1/AWS-Glue/src/sample-glue-job.py” file, to show how to run as PLAIN Python-program.

FYI: My python-code in that sample-glue-job.py expects the following 6 CLI-arguments (with their values).

If you do _ NOT _ like this list of CLI-args, see section above titled “Want to change the CLI-args?”

python3 sample-glue-job.py --ENV sandbox ^
     --commonDatabaseName MyCOMMONDATABASENAME ^
     --glueCatalogEntryName MyDICT ^
     --rawBucket MyRAWBUCKET ^
     --processedBucket MyPROCESSEDBUCKET  ^
     --finalBucket MyFINALBUCKET

If you want the ability to run both as a plain python-script as well as run it inside AWS-glue, you __ MUST __ replicate the structure and code within this “macbook-m1/AWS-Glue/src/sample-glue-job.py”.

APPENDIX

Running out of Disk-space or Memory?

Screenshot below shows the recommended “high” settings.
After building images, you can reduce:

“CPU” can be lowered to “2”.
“Memory” can be lowered to “4GB”.

End of Article.

Running AWS Glue locally on a MacBook-M1

Sarma — Mon, 08 Jan 2024 06:40:19 +0000

by Udaybhaskar Sarma Seetamraju
ToSarma@gmail.com
Dec 31 2023

Highest-level Context

If you are into “Shift-Left” (whether re: Testing, Security, or Replicating-problems-on-developer-laptop, etc ..), then this article is for you.

For the very first time that you switch to an M1-chipset based MacBooks (from intel-chip based MacBooks) .. Productivity is significantly impacted when doing development/testing/troubleshooting “locally” on your laptop. Out-of-scope of this article is supporting those switching from Windoze.

Towards enabling up to 5x developer-productivity by allowing developers to robustly SIMULATE the Cloud-environment on a laptop — I have the following series of articles re: M1-chipset based MacBooks:

Running AWS CodeBuild locally on MacBook-M1.
- Running Containers based on older Ubuntu 20.04 (released in the year 2020) as well as on the newer Ubuntu 22.04 (released in the year 2022)
- Running Containers based on arm64-based Linux
(This) Running AWS Glue locally on MacBook-M1. Various scenarios covered like: you do Not have “aws credentials” on your Laptop (forcing you to mock all the AWS API calls like S3 GET, Glue-Catalog queries, etc..)
New Security-related Best-Practices when creating arm64/aarch64 Docker-Images on a MacBook-M1.

Quick Summary

Aiming for very simple single command, based on bash-shell scripts --> to execute your python-code as a Glue Job’s build locally on your MacBook-M1.

In addition, I have a section on how to significantly raise your productivity, in debugging/developing your python-code, even if your company denies your AWS CLI credentials.

To state the obvious, everything here is 100% Python + Bash-Scripts.

Note: You should aim to have your software work on arm64 containers, which invariably is cheapest compute on cloud. More below.

Problem Statements

Using Git for capturing ALL code-changes while simultaneously copying-n-pasting into AWS Glue-Studio (for testing/troubleshooting) is painful, error prone and frustrating.
As time progresses, your Glue Job will become complicated and require more than one Python-script. Worse, you have one or more folder-hierarchies, all of which contain PY files that you need to import!
Many developers prefer to develop/test/troubleshoot their code as “plain python”, and Not as a Glue-Job. There is No good reason to deny such developers from doing just that.. .. while ensuring that code will work without any issues inside AWS Glue running locally on MacBook-M1, and eventually work without issues inside Glue on AWS.
1. When running as “plain python”, there should be No runtime dependencies (like “import awsglue”).
2. When running as “plain python”, there should be No spark-dependency.
3. When running as “plain python”, all inputs/files should be on local laptop’s filesystem. All output should be written to local filesystem only.
4. When running as “plain python”, all information from Glue-Catalog should be available OFFline (as a Python Dict object)
How to proactively ensure the Glue Job will work on all chip-architectures - without having to scramble later? How to explicitly utilize all x86_64/amd64/arm64/aarch64 architectures locally on laptop?
If the Enterprise does Not allow Laptops to have AWS-Credentials (in ~/.aws/credentials file); Even so, how can I EFFICIENTLY test/debug the python-code file locally on my laptop, even as it needs access to Glue Catalog and/or S3 buckets?

Get started!

export BUILDPLATFORM="linux/aarch64"
ONLY when for running on MacBook-M1 laptop, if you'd like to take advantage of native-performance boost !!

Based on your needs on AWS choose between these 2:
export BUILDPLATFORM="linux/~~amd64~~"
export BUILDPLATFORM="linux/arm64"/

export DOCKER_DEFAULT_PLATFORM="${BUILDPLATFORM}"
export TARGETPLATFORM="${DOCKER_DEFAULT_PLATFORM}"

WORK_AREA=~
cd ${WORK_AREA}
git clone https://gitlab.com/tosarma/macbook-m1.git

To try out a sample ..

cd macbook-m1
cd AWS-Glue/src
${WORK_AREA}/macbook-m1/AWS-Glue/bin/run-glue-job-LOCALLY.sh  sample-glue-job.py

No Bash? Want Python instead?

Just replace the “.sh” with “.py” — in the script name “run-glue-job-LOCALLY” (as shown above).
And, of course, you must insert “python3” at the very beginning of the CLI (this is a platform-independent advice).

Important - Note these:

I only tested using Python3.11; No other Python version tested.
PRE-REQUISITES:
- pip3 install docker

Ready to use it for your own Glue-Script?

First, read the full details within the macbook-m1/AWS-Glue/README.md file.
Copy the *.py files in the macbook-m1/AWS-Glue/src/common subfolder into --> YOUR project’s TOPMOST-folder.
- ATTENTION: the files ./src/common/*.py must exist in your project, after you are done copying.
Make sure to edit your PY file, to look like the example file provided (sample-glue-job.py)
From your project root, run:

cd <your-project's-root-folder>

${WORK_AREA}/macbook-m1/AWS-Glue/bin/run-glue-job-LOCALLY.sh \
          path/to/your/file.py

If you are having problems importing the new files under ./src/common, then try adding this command below and then retry the above command.

export PYTHONPATH="your-project's-root-folder"

Is your Python code-base consisting of multiple files across multiple folder-hierarchies?
See section below titled “Complex folder-hierarchies?”

Want to change the CLI-arguments?

Three simple steps:

Edit the file macbook-m1/AWS-Glue/src/common/cli_utils.py
Look inside “process_all_argparse_cli_args()” and make changes in that function.
Look inside “process_std_glue_cli_args()” and make changes inside that function.
- Note: make sure to make similar changes in above steps 2 & 3.

Example:-

You would like to support a new cli-arg as:
*  --JOB_NAME 123_ABC
Insert a new line at (say) line # 125 for JOB_NAME.
- This will ensure your code will get the value 123_ABC when running __ INSIDE __ AWS-Glue !!
Insert a new line at (say) line # 78 for --JOB_NAME
- This will allow you to run your python-code as a PLAIN python-command and read this CLI-arg.
- See more re: this in a following section titled “running as a PLAIN python-command”

Never edit the file:
macbook-m1/AWS-Glue/src/common/glue_utils.py

The files “common.py” and “names.py” in that same folder can be edited. Feel free to play around with them.

Tips, Issues & Errors

See Appendix sections, for tips on configuring Docker-DESKTOP.

Question: Want to automatically cleanup/delete the Docker-containers - after they exit?
Answer: INSERT the cli-arg “--cleanup” BEFORE the python-filename, to that “run-glue-job-LOCALLY.sh” script.

Advanced User - Complex folder-hierarchies?

That script “run-glue-job-LOCALLY.sh” will automatically do that for you --> that is, it will automatically look UNDER the current-working-directory, and find all **/*.py files and put them in a temporary ZIP-file.
The script will then automatically pass it on to Glue-inside-Docker (running on your laptop).

If you need to import PY files in parent/ancestor levels, I recommend that you add a “symlink” (Linux command “ln -s”) to those files, and put that symlink in your current-working-folder.
Git will preserve these “symlinks” as exactly just that. It will NOT convert them into files. So, feel better already!

No AWS Credentials on your laptop?

For security-reasons, many companies are denying developers the AWS-credentials for AWS-CLI use.

That means you have a showstopper -> re: locally testing/debugging your python-code, for scenarios like:

COPY INPUT-files from S3-buckets --> into the “current-working directory”.
COPY OUTPUT-files from the “current-working directory” --> into S3-buckets.
Lookup Glue Catalog.
.. etc ..

To workaround this restriction ..
You need to write code that detects whether its running on a MacBook-M1 -versus- actually running inside AWS-Cloud.
In other words, you need to “Short-Circuit” all that code that interacts with AWS-APIs (Glue-Catalog, S3, ..) and mock the expected response from those AWS-APIs.

If you use my script “run-glue-job-LOCALLY.sh”, it automatically sets an environment-variable called “running_on_LAPTOP” when running your python-code inside a Docker-Glue container on your laptop!!

How-To “short-circuit”:

if ( os.environ.get('running_on_LAPTOP') ):
    print( "!!!!!!!!!!!!!!!!! running on laptop !!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!" )
    .. ### assume S3-get is already done and file is available in current-directory
    .. ### assume Glue-Catalog-Query is already done and ..
       ###      the "JSON-Response" is available in current-directory as a JSON-file
    ..
else:
    ..
    ..

Running as a plain python command

EXAMPLE:
I’m going to use the same “macbook-m1/AWS-Glue/src/sample-glue-job.py” file, to show how to run as PLAIN Python-program.

FYI: My python-code in that sample-glue-job.py expects the following 6 CLI-arguments (with their values).

If you do _ NOT _ like this list of CLI-args, see section above titled “Want to change the CLI-args?”

python3 sample-glue-job.py --ENV sandbox \
     --commonDatabaseName MyCOMMONDATABASENAME \
     --glueCatalogEntryName MyDICT \
     --rawBucket MyRAWBUCKET \
     --processedBucket MyPROCESSEDBUCKET  \
     --finalBucket MyFINALBUCKET

APPENDIX

Docker-Desktop settings for `aarch64`-chipset

See screenshot below.
Turn ON the setting titled “Use containerd for pulling and storing images”!
Note: for other scenarios, you may have to turn it OFF.
I can’t help explain this crazy conflicting instructions.
As of 2023, this is a Docker-on-MacBook issue, resolvable only by Docker + Apple Corp.

Running out of Disk-space or Memory?

Screenshot below shows the recommended “high” settings.
After building images, you can reduce:

“CPU” can be lowered to “2”.
“Memory” can be lowered to “4GB”.

FYI only - To run on a MacBook-M1, many amd64 emulated containers like Neo4j v4.x will frequently fail, unless you provide Docker with a minimum of 5 cpus and 8GB of RAM!

End of Article.

Running AWS CodeBuild locally on a MacBook-M1

Sarma — Mon, 08 Jan 2024 04:35:58 +0000

by Udaybhaskar Sarma Seetamraju
ToSarma@gmail.com
Dec 31 2023

Highest-level Context

If you are into “Shift-Left” (whether re: Testing, Security, or Replicating-problems-on-developer-laptop, etc ..), then this article is for you.

(This) Running AWS CodeBuild locally on MacBook-M1.
- Running Containers based on older Ubuntu 20.04 (released in the year 2020) as well as on the newer Ubuntu 22.04 (released in the year 2022)
- Running Containers based on arm64-based Linux
Running AWS Glue locally on MacBook-M1. Various scenarios covered like: you do Not have “aws credentials” on your Laptop (forcing you to mock all the AWS API calls like S3 GET, Glue-Catalog queries, etc..)
New Security-related Best-Practices when creating arm64/aarch64 Docker-Images on a MacBook-M1.

Summary

Aiming for very simple set of commands, based on bash-shell scripts --> to start a CodeBuild’s build locally on your MacBook-M1 laptop.

You may ask: Why even bother with arm64/aarch64 docker images, especially when we can set the following ENV-Variables and successfully emulate x86/amd64 chipsets on MacBook-M1?

export BUILDPLATFORM="linux/amd64"
export DOCKER_DEFAULT_PLATFORM="${BUILDPLATFORM}"
export TARGETPLATFORM="${DOCKER_DEFAULT_PLATFORM}"

My response: You should aim to have your software work on linux/arm64 containers, which invariably is cheapest compute on cloud.

Next, I have tips on how to significantly raise your productivity, in debugging/developing your buildspec.yaml, even if your company denies your AWS CLI credentials.

Yes, this article is based on AWS own official Documentation.

But .. ..

those AWS instructions (link above) are a very complex for anything other than AmazonLinux2/2023 on x86, with “gotchas” and challenges of using Ubuntu-vs-AmazonLinux on a MacBook-M1 Laptop.

Based on very simple CLI-arguments, the bash-scripts offered in this article will help automatically download these images (if missing) and execute a build using the buildspec file in current-working directory.

Very-Advanced User: Short summary

You need 2 container-images are required to start a CodeBuild’s build locally on your laptop.
1. amazon/aws-codebuild-local:latest (This is the “Engine” a.k.a. AWS-CodeBuild’s Agent/Platform on your laptop) 
2. If AmazonLinux2/2023 is your Build O/S: pull the image  public.ecr.aws/codebuild/amazonlinux2-x86_64-standard:5.0
3. If Ubuntu 20.04 or 22.04 is your Build-env O/S, then you’ll need locally custom-build the following Docker-Images from AWS official source-code:
  - aws/codebuild/standard:5.0
  - aws/codebuild/standard:7.0
  - No, you can NOT find these images anywhere! 
Optionally, instead of image #2 above, if you want aarch64-chipset:
1. then download the image:  public.ecr.aws/codebuild/local-builds:aarch64
2. For a full list of all possible images (with Go/Java/Python/..):   aws codebuild list-curated-environment-images

Problem Statements

Want to use Ubuntu as the O/S for CodeBuild projects, and use it to test/debug locally on my MacBook-M1.
Rather than wait 10+ minutes to find a stupid-mistake in the buildspec.yaml file, can I run buildspec.yaml file locally on my laptop, to test everything in it?
How to proactively ensure the CodeBuild project’s build will work on all chip-architectures - without having to scramble later? How to explicitly utilize x86_64/amd64/arm64/aarch64 architectures locally on laptop?
Company does Not allow Laptops to have AWS-Credentials (in ~/.aws/credentials file); My buildspec.yaml file uses Secrets, Parameter-Store, etc.. as well as AWS CLI commands. Even so, how can I EFFICIENTLY test/debug the buildspec.yaml file locally on my laptop?

Ready to use scripts

Please note: For Ubuntu based CodeBuild, read the sub-section (below) titled “CodeBuild projects using Ubuntu”.

I have bash-shell scripts that require just one CLI-argument (either AmazonLinux2 or UBUNTU).

To get started .. ..

git clone https://gitlab.com/tosarma/macbook-m1  
- You’ll notice 2 bash-scripts in AWS-CodeBuild/ sub-folder:
  - “LOCAL-aws-codebuild-runner.sh” and
  - “LOCAL-create-aws-codebuild-standard-image.sh” 
On your bash-terminal, run the 1st script above using the full-path to it.
- FYI: The 1st script will automatically run the 2nd one as needed.

FYI — Running the 1st script assumes you have a buildspec.yaml file in the current-folder. Or a “.yml” file.
Note: If you have a LOCAL-buildspec.yaml file in the current-folder, then it is used instead.
Re: this “LOCAL-” file, immediately please read the following sub-section “No AWS Credentials on your laptop?”.

Note: Only for “AmazonLinux2” cli-argument, the __ FIRST __ time you run the above script it’ll take up to 5-minutes to download about 30+/- images from AWS ECR-Repo.

No Bash? Want Python instead?

Just replace the “.sh” with “.py” — in the script names (above).
And, of course, you must insert “python3” at the very beginning of the CLI (this is a platform-independent advice).

To repeat:

“LOCAL-aws-codebuild-runner.py” will hang with NO output, for roughly 2-to-5 minutes (depending on how much CPU and MEMORY you have allocated to the Docker-Desktop, as well as speed of your internet connection)
“LOCAL-create-aws-codebuild-standard-image.py” will hang with NO output .. ..
- .. for AmazonLinux2 for roughly 2-to-5 minutes (depending on how much CPU and MEMORY you have allocated to the Docker-Desktop, as well as speed of your internet connection)
- .. for Ubuntu (see full details below) for a minimum of 2+ H O U R S !! (If you have minimal CPU allocated for Docker-Desktop, it will as long as 4+ HOURS)

Important - Note these:

I only tested using Python3.11; No other Python version tested.
PRE-REQUISITES:
- pip3 install docker
- pip3 install GitPython

Using Ubuntu 20.04

Warning: As of 2024-January, only ‘x86_64/amd64’ chipset-architecture supported (on MacBook-M1) for “Ubuntu” O/S.

Question: Who would use Ubuntu (instead of AmazonLinux) - for their CodeBuild projects?

Answer: Ubuntu has great trouble-free support for installing EXACT versions of software, whether Google-Chrome (for headless testing) or older versions of NodeJS or Python, etc.. ..

Warning: Only for “Ubuntu” O/S CodeBuild builds, you _ MUST _ turn-OFF (that is, un-check the checkbox) as shown in screenshot below.
Note: For fix many __ OTHER __ issues while running Docker-containers on MacBook-M1, you are REQUIRED to turn-ON this checkbox.
So, please pay attention to conflicting configurations (within Docker-Desktop on MacBook-M1)!

Warning: Only for “Ubuntu” O/S, the __ FIRST __ time you run the above script it’ll takes minimum 2 hours — to re-create Ubuntu Image runtime from scratch (after downloading the AWS Source-code).

Why? Because, unfortunately, AWS does Not offer these ready-to-use images for Ubuntu O/S to download.

To get started, here are the simple commands to run!

cd <Your-own-project>

export BUILDPLATFORM=linux/amd64
export DOCKER_DEFAULT_PLATFORM="${BUILDPLATFORM}"
export TARGETPLATFORM="${DOCKER_DEFAULT_PLATFORM}"

<Path-to-Git-Cloned-folder>/AWS-CodeBuild/LOCAL-aws-codebuild-runner.sh  UBUNTU

Let me know of any issues with the above.

Issues & Errors?

See Appendix, for resolving the errors.
Example: See Appendix sub-section titled “Docker-Desktop settings for Ubuntu-on-x86 images”.

Installing Chromium Headless on Ubuntu

See install-Chromium-latest-on-ubuntu20.04.sh
under https://gitlab.com/tosarma/macbook-m1/-/tree/main/AWS-CodeBuild/software-install-scripts?ref_type=heads

Installing Old Node.JS on Ubuntu

Edit and run install-NodeJS-latest-on-ubuntu20.04.sh
under https://gitlab.com/tosarma/macbook-m1/-/tree/main/AWS-CodeBuild/software-install-scripts?ref_type=heads

Installing Old Python on Ubuntu

Edit and run install-python-latest-on-ubuntu20.04.sh
under https://gitlab.com/tosarma/macbook-m1/-/tree/main/AWS-CodeBuild/software-install-scripts?ref_type=heads

Ubuntu 22.04

To switch to the newer Ubuntu 22.04 (released in the year 2022) ..

Edit script “./AWS-CodeBuild/LOCAL-create-aws-codebuild-standard-image.sh”.
UN-comment the line # 21 (to use “aws/codebuild/standard:7.0”)
Comment out the next line # 22. (Disable “~~5.0~~”)
Must run “docker system prune --all --force —volumes”.
Finally, follow the instructions in above section titled “CodeBuild projects using Ubuntu 20.04”

No AWS Credentials on your laptop?

For security-reasons, many companies are denying developers the AWS-credentials for AWS-CLI use.
If your buildspec.yaml file has AWS-Secrets (quite common!) or if it runs AWS-CLI inside (example: to get Stack-outputs), etc .. ..
.. then, you have a showstopper in locally testing/debugging your buildspec.yaml locally on your laptop.

My best-practice is to:

Create a 2nd (new) file named “LOCAL-buildspec.yaml” (as described below) +
Create a 3rd (new) file called “.env” file to along with it.

WARNING: Do _ NOT _ git-commit the “.env” file, as per global practices.
Immediately add the “.env” file to your “.gitignore” file.

This new “LOCAL-buildspec.yaml” file will:

Not have entries for Secrets and Parameter-Store entries
Not have all AWS-CLI commands
Not have any CDK or other commands.
Must “source” the .env file, in the “Install or Pre-Build” phases as:

. .env

NOTE: Re: the .env file:-

You will ensure the .env file sets all bash-variables that’ll contain the values for the Secrets, Parameter-Store entries, etc. ..
You will ensure .env file also provides all the values expected from the AWS-CLI commands (that were removed).

Finally, just run:

cd <your-own-project>

<Path-to-Git-Cloned-folder>/AWS-CodeBuild/LOCAL-aws-codebuild-runner.sh

Open Questions, Concerns and Challenges

WARNING: Do _ NOT _ git commit the .env file, as per global practices.
Immediately add the .env file to your .gitignore file.

If you mistakenly commit it, confess honestly and immediately to your corporate security team, and work on fixing the security vulnerability.

FYI - CodeBuild for Ubuntu O/S on aarch64 & amd64 chipsets are Not yet supported.
No timetables available.

Note: If you mistakenly or consciously run “docker system prune”, the Ubuntu O/S option will take 2+ hrs again (to re-create images from AWS official source-code)!
Warning: Do __ Not __ rely on “save” and “import” of the docker-image.
The Following failed for me.

 docker save                   --output ~/aws_codebuild_standard_5.0.tar   aws/codebuild/standard:5.0  
 docker import --platform linux/x86_64  ~/aws_codebuild_standard_5.0.tar   aws/codebuild/standard:5.0

FYI: Above “.tar” file is typically 10GB in size and takes about 2-3 minutes to be “saved”.

APPENDIX

Docker-Desktop settings for Ubuntu-on-x86 images

Why?
As of 2023, per https://github.com/moby/moby/issues/44578 Docker-Desktop’s containerd-integration can NOT interact with images that don't have the default platform.
And .. aarch64 is the default-platform on MacBook-M1, while we’re seeking amd64 for Ubuntu.

Docker-Desktop settings for aarch64-chipset

See screenshot in previous sub-section.
Turn _ ON _ the setting titled “Use containerd for pulling and storing images”!
Yup! Doing just the opposite!
I can’t help explain this crazy conflicting instructions.
As of 2023, this is a Docker-on-MacBook issue.

Running out of Disk-space or Memory?

Screenshot below shows the recommended “high” settings.
After building images, you can reduce:

“CPU” can be lowered to “2”.
“Memory” can be lowered to “4GB”.

FYI only - To run on a MacBook-M1, many amd64 emulated containers like Neo4j v4.x will frequently fail, unless you provide Docker with a minimum of 5+ cpus and 8GB of RAM!

Creating ARM64/Aarch64 Docker-Images on a MacBook-M1 - New Security-related Best-Practices

Sarma — Mon, 08 Jan 2024 03:12:11 +0000

by Udaybhaskar Sarma Seetamraju
ToSarma@gmail.com
Dec 31 2023

Highest-level Context

If you are into “Shift-Left” (whether re: Testing, Security, or Replicating-problems-on-developer-laptop, etc ..), then this article is for you.

Running AWS CodeBuild locally on MacBook-M1.
- Running Containers based on older Ubuntu 20.04 (released in the year 2020) as well as on the newer Ubuntu 22.04 (released in the year 2022)
- Running Containers based on arm64-based Linux
Running AWS Glue locally on MacBook-M1. Various scenarios covered like: you do Not have “aws credentials” on your Laptop (forcing you to mock all the AWS API calls like S3 GET, Glue-Catalog queries, etc..)
(This) New Security-related Best-Practices when creating arm64/aarch64 Docker-Images on a MacBook-M1.

Summary

You’ve been issued a MacBook-M1 laptop (by your own choice or otherwise). You may have prior experience working on Intel-based MacBooks; but, that is of little help once you start “local” Container-based development on it. Basic commands like “npm” and “maven/gradle” as well as “curl/wget” stop working for local Dockerfile builds, with the weirdest in-decipherable errors that leave you and the rest of your development simply cannot figure out.

You may ask: Why even bother with arm64/aarch64 docker images, especially when we can set the following ENV-Variables and successfully emulate x86/amd64 chipsets on MacBook-M1?

export DOCKERPLATFORM="linux/amd64"
export DOCKER_DEFAULT_PLATFORM="${DOCKERPLATFORM}"
export TARGETPLATFORM="${DOCKER_DEFAULT_PLATFORM}"

My response: Sooner or later you’ll hit a situation like this -> One single Neo4j ver4.x container in amd64-emulation mode takes up 50% of cpus + 8GB/50% of RAM on your MacBook-M1.

Benefit #1: If you’d like to have your software work on linux/arm64, which invariably is cheapest compute on cloud.

Benefit #2: In addition, this document hopefully will bring some significant changes to the way you look at “free software” (as mostly un-trustworthy). Hope this will help you to be in harmony with MacOS and thereby avoid fighting with the Corporate Security-team (so that you can efficiently continue with your software development).

Advanced-User: Short summary

Use your Laptop’s browser (Chrome, ideally) and download the entire “Cert-Chain” of SSL-Public-Certs for maven.apache.org and for npm.org or .. whatever url/website is failing on your MacBook-M1.
Store all these downloaded SSL-certs into a new folder in your project.
See also: “should you store these in Git?” in sub-section below (preceding the Appendix).
For Java, use “keytool -importcert” commands to import these (within the Dockerfile). For NodeJS/npm, use “npm config set cafile” command to import the “unified” CER file (within the Dockerfile).

Problem Statement: What you experience on your MacBook-M1

curl commands (within Dockerfile) fail during “docker build” commands on MacBook-M1.
“mvn” Maven commands (within Dockerfile) fail on MacBook-M1 — see sample error below.
“npm” commands (within Dockerfile) fail on MacBook-M1 — see sample error below.
“dockerfile-maven-plugin” uses a runtime based on x86 architecture and will NOT run on Apple M1 (aarch64).

Sample “mvn/maven” ERROR (for a spring-boot app):

Non-resolvable parent POM for groupId.subgroupId.myapp:myapp:0.0.1:
Could not transfer artifact org.springframework.boot:spring-boot-starter-parent:pom:3.0.2
from/to central (https://repo.maven.apache.org/maven2):
transfer failed for https://repo.maven.apache.org/maven2/org/springframework/boot/spring-boot-starter-parent/3.0.2/spring-boot-starter-parent-3.0.2.pom
and 'parent.relativePath' points at wrong local POM @ line __,
column 13: PKIX path building failed:
sun.security.provider.certpath.SunCertPathBuilderException:
unable to find valid certification path to requested target
..
Root Cause ERROR: UnsafeLegacyRenegotiation option for SSL connections
 is now disabled by default in OpenSSL 3.0
..

Sample “npm” error:

qemu: uncaught target signal 5 (Trace/breakpoint trap) - core dumped
Trace/breakpoint trap
ERROR: executor failed running [/bin/sh -c npm install --production]: exit code: 133

What is the underlying root-cause?

MacBook-M1-chip based laptops use the latest Apple MacOS.
Latest MacOS uses SSL v3.x.
Latest MacOS exclusively relies on SSL-v3.x (if you rely on Xcode’s tools and on home-brew).

FYI - SSL-v3 is officially renamed to “TLS v1.2”.
It is the latest and greatest and apparently, much much much more secure (than SSL-v2.x which is now considered legacy and a security-risk).
For those who are a tiny bit curious about why SSL-v3.0/TLS-v1.2 is a big-deal, take a quick look at Amazon’s decision to implement their own custom TINY-version of TLS - as published in “Continuous Formal Verification of Amazon s2n”

SSL v2.x has been around for so long, and .. “most” (in quotes) of its Ciphers it supports are considered legacy.
Translating that into developer language .. ..

AlpineLinux.org website uses “legacy” SSL-Cert CIPHERS.
maven.apache.org — same issue.
npm.org — same issue.
.. etcetera ..

Bottomline .. ..

Since Docker relies on the laptop’s O/S for all Cryptography-related activities, Docker is missing support for the legacy SSL-v2.x on the MacBook-M1.

Really BAD-idea(s) & Hit-or-Miss workaround(s)

(A) Choose “mirrors” that use latest SSL-Cert ciphers.

Example: switch to Alpine-MIRROR: https://mirror.math.princeton.edu/pub/alpinelinux/v3.17/

Unfortunately, it’s very hard to find “good” mirrors (that use SSL-v3.x/TLS-v1.2).

(B) Get rid of encryption-in-motion! ugh!

Example: replace “httpS” with plain “http” within “Dockerfile”.

~~https~~ curl http://archive.apache.org/dist/tomcat/tomcat-"$TOMCAT_MAJOR"/v"$TOMCAT_VERSION"/bin/apache-tomcat-"$TOMCAT_VERSION".tar.gz

Robust and Permanent Fixes

Step 1: Download the SSL-Certs - the complete “Hierarchy”

Since I have many easy-to-follow screenshots re: this, and since I want to keep this article “short” .. ..

.. Please jump to the only Appendix at the end of this article.

Step 2: For Java Maven

Inside Dockerfile, run the following commands.

Important for Java-only: 1st import the TOP-MOST CA-Cert;
Then, finally, import (bottommost in chain) web-site SSL-Cert.

The “if” condition (below) is just a suggestion - so that the keytool is ONLY run for MacBook-M1 chipset.
Keep it, if you’d like to have your software work on linux/arm64, which invariably is cheaper compute on AWS.

RUN if [ "${TARGETPLATFORM}" == "linux/aarch64" ] || [ "${TARGETPLATFORM}" == "linux/arm64" ]; then \
      keytool -importcert -noprompt -trustcacerts \
        -cacerts -storepass changeit   -keystore "${JAVA_HOME}/lib/security/cacerts"   \
        -alias ROOT-1A               -file "${APP_SRC_FOLDER}/etc/ROOT-1A.cer"; \
      keytool -importcert -noprompt -trustcacerts \
        -cacerts -storepass changeit   -keystore "${JAVA_HOME}/lib/security/cacerts"   \
        -alias SSLCA-1A.cer          -file "${APP_SRC_FOLDER}/etc/SSLCA-1A.cer";   \
      keytool -importcert -noprompt -trustcacerts \
                 -storepass changeit   -keystore "${JAVA_HOME}/lib/security/cacerts"   \
        -alias org.apache.maven.repo -file "${APP_SRC_FOLDER}/etc/org.apache.maven.repo.cer"; \
    fi

Java’s KeyTool command: https://docs.oracle.com/en/java/javase/17/docs/specs/man/keytool.htm

Step 2: For npm

ISSUE: We have multiple Cert-files. But, “npm” can only accept ONE SINGLE file.

SOLUTION:

Combine multiple Cert-files into a single file.
Here’s bash-commands for that.
Note: The files should be listed in the ORDER in which they were downloaded - as shown in this article.

FILES_IN_PROPER_SEQ=(~~nodejs.org.cer~~
~~Cloudflare-ECC-CA-3.cer~~
~~LEVEL2-ROOTCA-1A.cer~~ ~~TOPMOST-ROOT-1A.cer~~
)

TMPFILE="./etc/my-unified-certs.cer"
echo '' > "${TMPFILE11}"                  ### Empty out the file (if it exists)

for f in ${FILES_IN_PROPER_SEQ[@]}; do
    CERTFILE="${CERTS_FOLDER}/${f}"
    if [ ! -f "${CERTFILE}" ]; then
        echo "ERROR: File '${CERTFILE}' does not exist !"
        exit 9
    fi
    cat "${CERTFILE}" >> "${TMPFILE11}"
    echo '' >> "${TMPFILE11}"
done

NEXT: Inside Dockerfile, run the following commands.

COPY      /usr/local/share/ca-certificates/.

RUN  if [ "${TARGETPLATFORM}" == "linux/aarch64" ] || [ "${TARGETPLATFORM}" == "linux/arm64" ]; then \
        npm config set cafile   ${APP_SRC_FOLDER}/etc/my-unified-certs.cer  \
     fi

Step 3 (Optional) - `dockerfile-maven-plugin` won't run on Apple M1-chip

Try official “substitute” plugins like Eclipse JKube plugin: Kubernetes Maven Plugin or OpenShift Maven Plugin . Refer to Migration Guide for more details.

Open Questions, Concerns and Challenges

Should you store these SSL-Public-Certs in Git?

Risks as viewed by Enterprise Security Team(s) and my responses.

Risk: As a broad/generic statement, Certs and Secrets should Not be put into Git.
- Counter-Argument #1: These are PUBLIC-Key SSL-Certs. Meant for the whole world to know (or have a copy)!
- Counter-Argument #2: By checking them into Git, perhaps .. .. Security will know what Apps are impacted if the corresponding website is ever hijacked, or the CA-Root is compromised; Security can then alert the developers - to find a workaround (or, at least pause all Builds temporarily).
Risk: Not a good idea to download + store the CA-Root and the Intermediate-Root Certs.
- Counter-Argument - Benefit: So that we do Not have to “blindly” trust just one “standalone” certificate. This significantly reduces chances that Laptop will Not be downloading malware.
Challenge: This is a Laptop-issue and Not a CI/CD or DevOps issue. Nor related to application-design. So, why is this part of the codebase?
- Wham!
- KO’ed.
- Hard to have a counter-argument.
- But here’s a pathetic “excuse”!!
  - There are “developers” and then there are “developers”.
  - You know EXACTLY what I mean.
  - Putting this in the Git-repo makes life convenient for the leads and for the senior members of any development team.

APPENDIX

Step 1: Download the SSL-Certs - the complete “Hierarchy”

Note: The screenshots are for “maven.apache.org".
Note: Please repeat the following for the website/url that fails to download on your MacBook-M1.

Step 3 screenshot somehow not rendering on browsers. It shows how to go to the "Details" tab of "Certificate Viewer: *.apache.org"

Now repeat steps 4.a and 4.b (steps shown above) ..
.. for the two OTHER entries in the “Certificate Hierarchy” (as indicated below).

/ End of Article.

Sophisticated use-case of AWS CDK that leverages AWS SDK

Sarma — Wed, 01 Nov 2023 02:06:36 +0000

by Udaybhaskar Sarma Seetamraju (ToSarma gmail)
Oct 16 2023

One sentence summary

A real-world sophisticated use-case exists for leveraging AWS SDK within AWS CDK code, when deploying cloud-native Non-Kubernetes solutions, within Enterprise-environments where IP-address space is a major constraint.

One Paragraph Summary

For those of us, who do Not work for the leading-edge tech-companies (which is the overwhelming majority of the Fortune 1000), private IP addresses are a fact. That is, we face serious constraints in how many VPCs and AWS Accounts can exist. Also, frequently due to lack of an IPAM system, the ability to automatically create and dispose AWS Accounts is missing. In such Enterprise contexts, as shown in this article, using AWS SDK (within AWS CDK code) leads to far simpler and far more robust solutions, specifically when deploying cloud-native Non-Kubernetes solutions.

Focus Areas & Keywords

DevOps, Enterprise Environments, “Dirty” VPCs, AWS CDK, AWS SDK, CLI Profiles, “Everything is Disposable”.

The main article

Enterprise Context: A unwavering focus on Production

My Philosophical “True North”: Regarding AWS Resources, my perspective is: AWS SDK is for obtaining details on what already exists, whereas AWS CDK (incl. Cloudformation) is about what we need to create. In other words, I personally do Not use AWS SDK to create new AWS Resources, nor do I “mildly” manipulate any pre-existing AWS Resources. FWIW: CDK itself relies on SDK.

I belong to the group of people who do Not like Cloudformation in which each resource has a “Condition:”. The best practice (of maintainability / supportability) is to have Cloudformation-templates (CFT) that simply & clearly tell you what was deployed; It’s easy to predict/understand what will happen if you re-deploy. In many industries, Cloudformation-templates, Stack-drifts and Change-Sets are like god’s gifts, but that is out of scope for this article.

“Corporate Life”: Commonly enough within Enterprise environments, either there is a limited ability, or there is an outright ban on the ability to create “fresh new” VPCs and/or AWS Accounts. An even more important nuance is, this limitation/absence can be quite severe within Production environment/VPCs, even as developers may experience “freedom” within sandbox/PoC/Development/Integration environments.

Per the best practice for designing solutions, we must have all identical environments. So, developers should “copy/simulate” the constraints of Production into all the Non-prod environments.

Use-Case / Scenario: This article is specific to one specific use-case, where CDK-based deployment is done manually (via “cdk” CLI). Personally, I experience this use-case early in every project, when the deployment-architecture diagram (onto AWS) is constantly in flux. The iterative approach using the CI/CD pipeline to improve your CDK is too horribly slow, un-acceptably inefficient and error-prone.

For any of my projects, for 3 specific environments (namely sandbox, PoC and development environments), I rely on manual CDK-CLI based deployments.

The core security needs

CDK offers a great advantage within DevSecOps pipelines, something we should all take FULL advantage of. Via CDK-code, we can very easily ensure all IAM Roles and Network-security components are also covered robustly, for each component of overall solution.
The example in this article focuses solely on AWS Security Groups (SGs), since use-cases to share at least one SG (across various components of a solution) is a common enough. More details on this is throughout this article. Note: Never share IAM Roles across components of a solution.
In the early release of a medium/large application, it is very common to have an “Application-wide” Security Group (“AppWide-SG”). This is a good best-practice to remember, as it forces developers to start day-one, in strictly using SGs to limit access to every single component.

What is a Dirty VPC?

The concept of a “Dirty VPC” is simply an implication of the Enterprise constraint that “One is restricted in creating new VPCs within DevSecOps pipelines”
The dictionary word “Dirty” has a negative meaning. But, for this article, we will interpret that word to mean that the VPC already has one of more desired AWS resources; in this example, Security-Group(s).
I love the philosophical approach off "Everything is Disposable". I practice it a lot, including the use of cloud IDEs. But, “Dirty VPCs” are the proverbial “immovable object” within Enterprise environments.

Some foundational facts

FYI: Per AWS Official documentation, AWS SDK authenticates (a.k.a. AuthT) based on one of the following 3 mechanisms:
1. AWS_ACCESS_KEY_ID, AWS_SECRET_ACCESS_KEY, and AWS_SESSION_TOKEN environment variables.
2. Based on “--profile” CLI arguments.
3. Based on “default” AWS Profile in ~/.credentials file.
FYI: Similarly, AWS CDK relies on the above 3 for AuthT a.k.a. authen*T*ication.
FYI: Similarly, re: the “Region”, both AWS CDK & AWS SDK determine it either based on environment variables, or “--region” CLI-arg or on the region specified within “default” AWS Profile in ~/.credentials file.
Note: In an Enterprise work-environment, due to pervasive use of SSO (Single-SignOn), you are highly likely to be using the “--profile” cli-arg.

DRY (Don’t Repeat Yourself) Problem

Best Practice: For sandbox/PoC/development environments, exclusively for manual deployments, CDK best practices suggest that you use “--profile” CLI-arg (as well as the “—region” CLI-arg. In contrast, for all other environments, especially within DevSecOps pipelines, you should be exclusively rely on AWS_ACCESS_KEY_ID, AWS_SECRET_ACCESS_KEY, and AWS_SESSION_TOKEN environment variables. CDK’s & SDK’s best-practices suggests the use of “--profile” CLI-arg and the “--Region” CLI-arg (instead of using “default” AWS Profile).
Limitation: Per the AWS Support-case 13701110271, dated Sept 2023, AWS confirmed that CDK uses AWS SDK underneath, but, unfortunately, we can Not get “access” to the SDK. That implies that our code must explicitly initialize AWS SDK, by providing it the credentials.
The Issue: the CDK will Not share details on how it successfully-authenticated -- whether via Env-Vars, or via the “--profile” CLI-arg, or via the “default” AWS Profile.
The PROBLEM: We have a “DRY problem”, when we use “--profile” CLI-arg and the “--Region” CLI-arg (instead of “default” AWS Profile); AWS CDK gets that information, but Not the AWS SDK.  So, we need to “copy/clone” that information and send it to AWS SDK too (for it to AuthT too). See code-snippet #2 below, for the implementation.

The official Workaround: AWS-SDK AuthT

Within my CDK code, I mandate 2 additional CLI-arguments as: “--context AWSProfile=...” and  “--context AWSRegion=...” To enforce these additional cli-args, use code-snippet #1 (see below).
FYI: The above makes your command for manual CDK-deployment look like:-

cdk deploy|synth \
     --profile ${AWSPROFILE} \
     --region ${AWSREGION} \
     --context AWSProfile=${AWSPROFILE} \
     --context AWSRegion=${AWSREGION}

Per the AWS Support-case 13701110271, dated Sept 2023, AWS FYI: I got 2023-specific confirmation on what I’m recommending within this article: “What you are doing with the context file is the best approach to this.” Please do Not assume this remains true for 2025 & beyond.

Code Snippet #1 - “`cdk`” CLI’s mandatory CLI args

Within your CDK project, please copy these lines within the only file: “./bin/*.ts”, ideally preceding the lines starting with “new Stack(app, .. {”.

const myAWSProfile = scope.node.tryGetContext('AWSProfile');

if ( ! AWSProfile ) {
    console.error(`!! FATAL-ERROR: CLI-args are missing:-->      --context AWSProfile=....\n\nExiting ...`);
    process.exit(1);
}

const myAWSRegion = app.node.tryGetContext('AWSRegion');

if ( ! AWSRegion ) {
    console.error(`!! FATAL-ERROR: CLI-args are missing:-->      --context AWSRegion =....\n\nExiting ...`);
    process.exit(1);
}

// OPTIONAL CLI-arg
const isDirtyVPC_cliarg :string | undefined = scope.node.tryGetContext('dirtyVPC');
const isDirtyVPC :boolean = (isDirtyVPC_cliarg != undefined) && (isDirtyVPC_cliarg !== "false");
if ( isDirtyVPC ) {
    console.log(`Ok! Deploying to a "dirty" VPC that may already have certain SecurityGroups!`);
}

Code Snippet #2 - AWS-SDK: Authenticating into AWS-API

import * as EC2 from ‘aws-cdk-lib/aws-ec2';
..
..
  const aws_sdk_credentials = {
      credentials: fromIni({
          profile: myAWSProfile,
      }),
      // region: process.env.CDK_DEPLOY_REGION || process.env.CDK_DEFAULT_REGION,
      region: myAWSRegion,
  };

  const stsclient = new STSClient(aws_sdk_credentials);
  const stscommand = new GetCallerIdentityCommand({});
  const stsresponse = stsclient.send(stscommand).then((data) => {
     const AWSProfile_derived = data.Arn?.split('/')[1] || “UnknownProfile1";
     console.log(`Sanity-Check: AWSProfile in ARN = ${AWSProfile_derived}`);
  }

How to keep your CFT simple & highly maintainable

This article wouldn’t be complete without ready-to-use CDK code-snippets, to conditionally/optionally generate resources within your Cloudformation-Template (CFT), after AWS-SDK calls are used to determine those conditions.

In this example (code snippet #3), we use AWS-SDK calls to determine if an Security-Group exists; If missing, the creation of the SG (AWS Security Group) is included in the cloudformation generated by SDK.

Code Snippet #3 - Conditional Cloudformation

import * as EC2 from ‘aws-cdk-lib/aws-ec2';
import { EC2Client } from ‘@aws-sdk/client-ec2';
..
..
async function lookupSecurityGroup( securityGroupName: string, 
                                    aws_sdk_credentials: any, )
              : Promise<string | undefined>
{
    const ec2client = new EC2Client( aws_sdk_credentials );
  ..
  .. // see full version of THIS function -> at bottom of this article.
  ..
    const sgresponse = await ec2client.send( sgcommand );
    if ( !sgresponse.SecurityGroups || sgresponse.SecurityGroups!.length < 1) {
        throw .. ..
    } else {
      return new Promise((res) => res( sgresponse.SecurityGroups[0].GroupId );
    }
} // end of async function.

// now let’s utilize the above function !!

    await lookupSecurityGroup( SGNAME, aws_sdk_credentials )
    .then((sgid) => {
        myAppWideSecurityGroup = EC2.SecurityGroup.fromLookupByName( this,  SGNAME, SGNAME, defaultVpc );
        if ( ! isDirtyVPC ) {
           throw Error( `!! SAFETY-CHECK ERROR !! SG ${SGNAME} already exists.  You did NOT provide CLI-arg "--dirtyVPC".\nExiting with error-code ...` );
        }
     }).catch( (e) => {
        console.log( `Ok. SG Does Not exist. So, creating new SG named ${SGNAME} ...` );
        const mySGProps: EC2.SecurityGroupProps = {
                    vpc: defaultVpc,
                    securityGroupName: SGNAME,
                    description: SGDESC,
                    allowAllOutbound: true,
                    disableInlineRules: true,
        };
        myAppWideSG = new EC2.SecurityGroup( this, SGNAME, mySGProps );
        // commented out: myAppWideSG.addEgressRule( EC2.Peer.anyIpv4(), EC2.Port.allTcp(), "Any Port OUTbound" );

        const cfn_sg = myAppWideSG.node.defaultChild as EC2.CfnSecurityGroup;
        cfn_sg.overrideLogicalId( SGNAME.replace(/-/g, '') );

The last line of code removes all hyphens from SGNAME, so the resulting string can be used as Resource-Name within Cloudformation-template.

Note: I'm ignoring addEgressRule since 'allowAllOutbound' is set to true by default; To add customized rules, set allowAllOutbound=false.

Key Takeways

Even though this article focused on a very valid common real-world scenario w.r.t. AWS Security Groups, this article offers a simple clean way to combine AWS-SDK with AWS-CDK, to can design/write CDK code that cleanly checks for a pre-existing component (say, a Fargate ECS-Cluster), and skip the generation of Cloudformation accordingly.

APPENDIX: full-code

import { EC2Client, DescribeVpcsCommand, DescribeSecurityGroupsCommand } from '@aws-sdk/client-ec2';
import { SecurityGroup } from '@aws-sdk/client-ec2';

export async function lookupSecurityGroup( securityGroupName: string,  aws_sdk_credentials: any, ): Promise<string>
{
  const ec2client = new EC2Client( aws_sdk_credentials );
  const filter = {
      Filters: [ { Name: "isDefault",  Values: ["true"] } ],
      MaxResults: 5, // Number("int");    Attention! Minimum value is 5. Maximum value of 1000.
      DryRun: false,
      // VpcIds: [ "vpc-0123456", ],
  };
  const vpccommand = new DescribeVpcsCommand( filter );
  const vpcresponse = await ec2client.send(vpccommand);
  if ( ! vpcresponse.Vpcs || vpcresponse.Vpcs.length < 1 ) {
      throw new Error('No default VPC found');
  }
  const defaultVpcId = vpcresponse.Vpcs![0].VpcId!;
  // DescribeVpcsResult: https://docs.aws.amazon.com/AWSJavaScriptSDK/v3/latest/clients/client-ec2/interfaces/vpc-8.html
  // {  Vpcs: [{
  //       CidrBlock: "STRING_VALUE",
  //       DhcpOptionsId: "STRING_VALUE",
  //       VpcId: "STRING_VALUE",
  //       IsDefault: true || false,
  //       Tags: [ // TagList
  //       .. ..

  // ---------------------------------- 
  const sgfilter = { // DescribeSecurityGroupsRequest
      Filters: [
          { Name: 'vpc-id', Values: [defaultVpcId], },
          { Name: 'group-name', Values: [securityGroupName], },
               // NOTE: Simpler to use "GroupName" instead !!!
      ],
      DryRun: false,
      MaxResults: 5, // Number("int");    Attention!  Minimum value is 5. Maximum value of 1000.
      // GroupIds: [ "sg-12345678", ],
      // GroupNames: [  "STRING_VALUE", ],
      // NextToken: "STRING_VALUE",
  };
  const sgcommand = new DescribeSecurityGroupsCommand( sgfilter );
  const sgresponse = await ec2client.send(sgcommand);
  // { // DescribeSecurityGroupsResult
  //   SecurityGroups: [ // SecurityGroupList
  //     { // SecurityGroup
  //       Description: "STRING_VALUE",
  //       GroupName: "STRING_VALUE",
  if ( ! sgresponse.SecurityGroups || sgresponse.SecurityGroups!.length < 1 ) {
      throw new Error(`No SG found with name ${securityGroupName}`);
  }

  return sgresponse.SecurityGroups![0].GroupId!;
}

DEV Community: Sarma

Vector-Database: Qdrant-cluster on ECS-Fargate

When do you need this?

Short Summary

how do I .. ?

Key variables

Fargate Task Definition

Fargate Service

Container for Fargate-Task

SecurityGroup

Other related articles

Vector-Database: Qdrant-cluster (Fargate) - Dockerfile

Why on earth did I chose to create my own Dockerfile?

What if Qdrant's official docker-image is mandatory?

HOW-TO

Explaining the "entrypoint" Bash-script

Why?

bash-script

Dockerfile

APPENDIX

Snapshots & Data-RESTORE: Vector-Database: Qdrant-Cluster

(A) HOW-TO - creating snapshots

OPTION 1: in an AUTOMATED manner

Option 2: Manually

(B) HOW-TO:- viewing snapshots

(C) HOW-TO:- prep for VERIFYING the snapshots

(D) HOW-TO:- Restore/Recover

Snapshots stored locally inside Qdrant-cluster.

From s3-bucket

(E) HOW-TO:- Lambda to create AUTOMATED-Snapshots

APPENDIX

Design: Vector-Database: Qdrant-Cluster on Fargate

Summary of Challenges:

Must-have Requirements

Decisions

Stop using ANY Container-image

Summary

Instead (for 2026) use ✅

Why bother?

Alternative

Why should I trust the above recommendation?

Appendix - How to SIMPLY confirm that the replacement works?

Appendix -- For those who love mega-deep-dives

Running AWS Glue locally on a Windows Laptop

Highest-level Context

Quick Summary

Problem Statements

Get started!

To try out a sample ..

Ready to use it for your own Glue-Script?

Want to change the CLI-arguments?

Tips, Issues & Errors

Advanced User - Complex folder-hierarchies?

No AWS Credentials on your laptop?

Running as a plain python command

APPENDIX

Running out of Disk-space or Memory?

Running AWS Glue locally on a MacBook-M1

Highest-level Context

Quick Summary

Problem Statements

Get started!

To try out a sample ..

No Bash? Want Python instead?

Ready to use it for your own Glue-Script?

Want to change the CLI-arguments?

Tips, Issues & Errors

Advanced User - Complex folder-hierarchies?

No AWS Credentials on your laptop?

Running as a plain python command

APPENDIX

Docker-Desktop settings for aarch64-chipset

Running out of Disk-space or Memory?

Running AWS CodeBuild locally on a MacBook-M1

Highest-level Context

Summary

Very-Advanced User: Short summary

Problem Statements

Ready to use scripts

No Bash? Want Python instead?

Docker-Desktop settings for `aarch64`-chipset

Step 3 (Optional) - `dockerfile-maven-plugin` won't run on Apple M1-chip

Code Snippet #1 - “`cdk`” CLI’s mandatory CLI args