AI-Powered Serverless Image Processing Pipeline — Part 6: Containerizing with Docker & Implementing CI/CD

AI-Powered Serverless Image Processing Pipeline — Part 6: Containerizing with Docker & Implementing CI/CD

In the first five installments we set up the AI model (a lightweight PyTorch‑based super‑resolution net), wired it to AWS Lambda via aws‑lambda‑powertools, and connected the whole thing to an S3 trigger that stores the processed images in a separate bucket. Based on my technical understanding as a Lead Programmer Analyst, I’ll now show you how to make that solution reproducible, portable, and continuously deployable using Docker and modern CI/CD practices.

Why Docker Matters for a Serverless AI Workflow

• Reproducibility. A Docker image captures the exact OS, runtime, libraries, and model weights you tested locally.
• Speed. Layer caching (as highlighted by Octopus Deploy) means that only the parts of the image that actually change are rebuilt, shaving minutes off each pipeline run.
• Portability. Whether you run on AWS Lambda, Google Cloud Run, or an on‑premise Fargate cluster, the same image works everywhere.
• Security. By pinning base images and using multi‑stage builds you reduce the attack surface and keep your supply chain auditable.

In the context of a serverless AI pipeline, Docker also acts as the glue between the model training environment and the runtime environment. The AWS Prescriptive Guidance for CI/CD and automation in serverless AI stresses that a repeatable build‑test‑deploy loop is the only way to keep up with rapid model iteration without breaking downstream services.

Step 1: Organize the Project Layout

Before we write any Dockerfile, let’s adopt a clean directory structure that works well with both Docker and CI pipelines:

ai-image-pipeline/
├── src/
│   ├── handler.py          # Lambda entry point
│   ├── model/
│   │   ├── super_resnet.pt # Pre‑trained weights (git‑ignored)
│   │   └── model.py        # PyTorch model definition
│   └── utils/
│       └── image_ops.py
├── tests/
│   └── test_handler.py
├── Dockerfile
├── requirements.txt
├── .dockerignore
├── .github/
│   └── workflows/
│       └── ci-cd.yml       # GitHub Actions pipeline
└── README.md

The .dockerignore file keeps large data files (like the model checkpoint) out of the build context unless we explicitly copy them, which speeds up the build and respects repository size limits.

Step 2: Write a Multi‑Stage Dockerfile

We’ll use a two‑stage build: the first stage compiles the Python dependencies in a slim python:3.11‑slim image, the second stage copies only the runtime artefacts. This mirrors the best‑practice patterns described in the Coursera project “Build a CI/CD Pipeline with Docker”.

# ---------- Stage 1: Build ----------
FROM python:3.11-slim AS builder

# Install OS‑level build tools (required for torch & numpy wheels)
RUN apt-get update && \
    apt-get install -y --no-install-recommends gcc g++ make && \
    rm -rf /var/lib/apt/lists/*

# Set a deterministic working directory
WORKDIR /app

# Install Python dependencies into a virtual environment
ENV VIRTUAL_ENV=/opt/venv
RUN python -m venv $VIRTUAL_ENV
ENV PATH="$VIRTUAL_ENV/bin:$PATH"

# Copy only the requirements first to leverage Docker cache
COPY requirements.txt .
RUN pip install --upgrade pip && \
    pip install -r requirements.txt

# ---------- Stage 2: Runtime ----------
FROM python:3.11-slim AS runtime

# Create a non‑root user for security
RUN useradd --create-home appuser
WORKDIR /home/appuser

# Copy the virtual environment from the builder stage
COPY --from=builder /opt/venv /opt/venv
ENV PATH="/opt/venv/bin:$PATH"

# Copy source code (excluding .git, __pycache__, etc.)
COPY src/ ./src/
COPY tests/ ./tests/
COPY model/ ./model/

# Ensure the model checkpoint is present at runtime (you can also pull from S3)
# For demo purposes we assume the checkpoint is checked in .gitignore
# and will be added manually in CI before deployment.

# Set the Lambda handler entry point
ENV PYTHONPATH=/home/appuser/src
CMD ["handler.lambda_handler"]

Key points:

• Layer ordering. OS packages and the base image are installed first, then requirements.txt. Because they change rarely, Docker can reuse the cached layer across builds.
• Non‑root user. Security best‑practice (also recommended by AWS Lambda container images).
• Virtual environment. Keeps the runtime lightweight and isolates system packages.

Step 3: Verify the Image Locally

Run a quick sanity check before committing the Dockerfile to source control.

# Build the image
docker build -t ai-image-pipeline:dev .

# Run a container, mounting a test image from the host
docker run --rm \
    -e AWS_REGION=us-east-1 \
    -v $(pwd)/sample.jpg:/tmp/input.jpg \
    ai-image-pipeline:dev \
    python -c "
import src.handler as h;
print('Invoking lambda handler locally...');
event = {
    'Records': [{
        's3': {
            'bucket': {'name': 'dummy-bucket'},
            'object': {'key': 'sample.jpg'}
        }
    }]
}
context = type('obj', (object,), {'aws_request_id': 'local-test'})
h.lambda_handler(event, context)
"

If everything is wired correctly you should see the model loading (once) and the processed image saved to /tmp inside the container. This mirrors the local “Docker‑run” test that the Octopus Deploy guide recommends for early feedback.

Step 4: Choose a CI Platform – GitHub Actions

GitHub Actions provides a free, fully‑managed runner pool that integrates natively with the repository. For an enterprise setting you could swap to GitLab CI or Azure Pipelines without changing the core Docker steps.

Below is a complete .github/workflows/ci-cd.yml that implements the following stages:

• Checkout – pull the repository.
• Login to Amazon ECR – secure push destination.
• Build & Test – compile the Docker image, run unit tests inside the container, and push to ECR on success.
• Deploy – trigger an AWS CloudFormation stack update (or Serverless Framework) that references the new image tag.

name: CI/CD – Dockerized AI Image Pipeline

on:
  push:
    branches: [ main ]
  pull_request:
    branches: [ main ]

env:
  AWS_REGION: us-east-1
  ECR_REPOSITORY: 123456789012.dkr.ecr.${{ env.AWS_REGION }}.amazonaws.com/ai-image-pipeline
  IMAGE_TAG: ${{ github.sha }}

jobs:
  build-test-deploy:
    runs-on: ubuntu-latest

    permissions:
      contents: read
      id-token: write   # For OIDC auth to ECR

    steps:
      # 1️⃣ Checkout source
      - name: Checkout repository
        uses: actions/checkout@v4

      # 2️⃣ Set up Python (needed for unit tests)
      - name: Set up Python 3.11
        uses: actions/setup-python@v5
        with:
          python-version: '3.11'

      # 3️⃣ Cache Docker layers (speed up builds)
      - name: Cache Docker layers
        uses: actions/cache@v3
        with:
          path: /tmp/.buildx-cache
          key: ${{ runner.os }}-docker-${{ github.sha }}
          restore-keys: |
            ${{ runner.os }}-docker-

      # 4️⃣ Configure Docker BuildKit for caching
      - name: Set up Docker Buildx
        uses: docker/setup-buildx-action@v3

      # 5️⃣ Log in to Amazon ECR using OIDC (no static credentials)
      - name: Authenticate to Amazon ECR
        id: ecr-login
        uses: aws-actions/amazon-ecr-login@v2
        with:
          mask-password: 'true'

      # 6️⃣ Build the Docker image
      - name: Build Docker image
        run: |
          docker build \
            --cache-from type=local,src=/tmp/.buildx-cache \
            --cache-to type=local,dest=/tmp/.buildx-cache,mode=max \
            -t ${{ env.ECR_REPOSITORY }}:${{ env.IMAGE_TAG }} \
            .

      # 7️⃣ Run unit tests inside the image
      - name: Run tests
        run: |
          docker run --rm \
            -e PYTHONPATH=/home/appuser/src \
            ${{ env.ECR_REPOSITORY }}:${{ env.IMAGE_TAG }} \
            pytest /home/appuser/tests

      # 8️⃣ Push image to ECR
      - name: Push Docker image to ECR
        run: |
          docker push ${{ env.ECR_REPOSITORY }}:${{ env.IMAGE_TAG }}

      # 9️⃣ Deploy with AWS CloudFormation (or Serverless)
      - name: Deploy CloudFormation stack
        uses: aws-actions/aws-cloudformation-github-deploy@v1
        with:
          name: ai-image-pipeline-stack
          template: infra/cloudformation.yml
          parameter-overrides: |
            ImageUri=${{ env.ECR_REPOSITORY }}:${{ env.IMAGE_TAG }}
          no-fail-on-empty-changeset: true

Notice how the workflow mirrors the “repeatable workflow” concept from the Coursera project: the same Dockerfile is used for local dev, CI, and production, guaranteeing “code‑to‑deployment” fidelity.

Step 5: CloudFormation Template for Lambda Container Image

Below is a minimal infra/cloudformation.yml that creates a Lambda function using the image we just pushed. It also provisions the S3 buckets and the necessary IAM role. Feel free to replace this with a Serverless Framework serverless.yml if you prefer.

AWSTemplateFormatVersion: '2010-09-09'
Description: AI‑Powered Image Processing Lambda (container image)

Parameters:
  ImageUri:
    Type: String
    Description: ECR image URI for the Lambda function

Resources:
  ImageProcessingFunction:
    Type: AWS::Lambda::Function
    Properties:
      FunctionName: ai-image-processor
      PackageType: Image
      Code:
        ImageUri: !Ref ImageUri
      MemorySize: 1024            # Adjust based on model size
      Timeout: 30
      Role: !GetAtt LambdaExecutionRole.Arn
      Environment:
        Variables:
          LOG_LEVEL: INFO
          MODEL_PATH: /home/appuser/model/super_resnet.pt

  LambdaExecutionRole:
    Type: AWS::IAM::Role
    Properties:
      RoleName: ai-image-lambda-exec
      AssumeRolePolicyDocument:
        Version: '2012-10-17'
        Statement:
          - Effect: Allow
            Principal:
              Service: lambda.amazonaws.com
            Action: sts:AssumeRole
      ManagedPolicyArns:
        - arn:aws:iam::aws:policy/service-role/AWSLambdaBasicExecutionRole
        - arn:aws:iam::aws:policy/AmazonS3ReadOnlyAccess
        - arn:aws:iam::aws:policy/AmazonS3FullAccess   # For writing processed images

  SourceBucket:
    Type: AWS::S3::Bucket
    Properties:
      BucketName: ai-image-source-bucket
      NotificationConfiguration:
        LambdaConfigurations:
          - Event: s3:ObjectCreated:*
            Function: !GetAtt ImageProcessingFunction.Arn
            Filter:
              S3Key:
                Rules:
                  - Name: suffix
                    Value: .jpg

  DestinationBucket:
    Type: AWS::S3::Bucket
    Properties:
      BucketName: ai-image-processed-bucket

Outputs:
  LambdaArn:
    Description: ARN of the image‑processing Lambda
    Value: !GetAtt ImageProcessingFunction.Arn

When the CI/CD pipeline pushes a new image tag, the CloudFormation stack updates the ImageUri parameter, causing a seamless zero‑downtime rollout. This is precisely the “intelligent automation” that AI‑Powered DevOps talks about – the pipeline decides when to promote a new model version based on test results.

Step 6: Adding Model Versioning to the CI Process

In production you rarely want to overwrite the same model file. Instead, tag the model with a semantic version (e.g., v1.2.0) and store it in an S3 model/ prefix. The CI job can then:

• Download the latest model artifact from an S3 bucket.
• Validate the checksum against a stored hash.
• Inject the model into the Docker image using the --build-arg MODEL_VERSION=... flag.

Here’s a snippet that adds a build‑arg to the Docker build step and copies the model at build time:

# In the builder stage, after installing dependencies
ARG MODEL_VERSION=latest
ENV MODEL_VERSION=${MODEL_VERSION}
RUN mkdir -p /model && \
    curl -sSL "https://my-model-bucket.s3.amazonaws.com/${MODEL_VERSION}/super_resnet.pt" \
    -o /model/super_resnet.pt && \
    echo "Model ${MODEL_VERSION} downloaded"
COPY src/model/ /model/   # Fallback for local dev

Corresponding CI change:

# In the GitHub Actions build step
- name: Build Docker image with model version
  run: |
    MODEL_VER=${{ github.ref_name }}   # Use branch name as version for demo
    docker build \
      --build-arg MODEL_VERSION=$MODEL_VER \
      -t ${{ env.ECR_REPOSITORY }}:${{ env.IMAGE_TAG }} .

With this pattern you can later add an automated model‑registry check that only promotes a new version if performance metrics (e.g., PSNR improvement > 1 dB) meet a threshold—exactly the kind of feedback loop described in the AI‑Powered DevOps article.

Step 7: Running Integration Tests on AWS (Optional)

Unit tests are great, but for a serverless AI pipeline you also want to verify the end‑to‑end flow:

• Upload a sample JPEG to the source bucket.
• Invoke the Lambda (or let S3 trigger it).
• Check that the processed image appears in the destination bucket and meets quality expectations.

GitHub Actions can spin up a temporary AWS environment using aws-actions/configure-aws-credentials and localstack (or the real AWS account with a dedicated testing role). Below is a minimal integration test script written in Python that you can call from the workflow.

import boto3, time, os
import uuid

s3 = boto3.client('s3', region_name=os.getenv('AWS_REGION'))
source_bucket = 'ai-image-source-bucket'
dest_bucket   = 'ai-image-processed-bucket'

def upload_test_image():
    key = f'test/{uuid.uuid4()}.jpg'
    s3.upload_file('tests/fixtures/sample.jpg', source_bucket, key)
    return key

def wait_for_result(key, timeout=30):
    prefix = key.replace('test/', 'processed/')
    for _ in range(timeout):
        resp = s3.list_objects_v2(Bucket=dest_bucket, Prefix=prefix)
        if 'Contents' in resp:
            return resp['Contents'][0]['Key']
        time.sleep(1)
    raise TimeoutError('Processed image never appeared.')

def main():
    test_key = upload_test_image()
    print(f'Uploaded {test_key}')
    result_key = wait_for_result(test_key)
    print(f'Processed image stored at {result_key}')
    # Optional: download and run a quick PSNR check
    # ...

if __name__ == '__main__':
    main()

Add this step to the workflow after the Docker push:

- name: Run end‑to‑end integration test
  env:
    AWS_REGION: ${{ env.AWS_REGION }}
  run: |
    pip install boto3
    python tests/integration_test.py

Running the integration test against a real AWS account gives you confidence that the Docker image, the Lambda configuration, and the S3 event source are all wired correctly before you promote to production.

Step 8: Optimizing Docker Builds for Faster CI

Two tricks that have saved my teams minutes per build:

• Separate rarely‑changed layers. In the Dockerfile, place the apt‑get install and pip install -r requirements.txt steps before copying the source code. This way, changing a single line in handler.py doesn’t invalidate the whole dependency layer. Leverage BuildKit’s cache‑to/cache‑from. The GitHub Actions YAML already uses --cache-from and --cache-to with a local directory cache. For even more speed you

---

Originally published at https://artificial-inteligence.phptutorial.co.in