DEV Community: Henry

Implementing a Self-Healing Serverless CICD Pipeline with AWS Developer Tools.

Henry — Wed, 05 Nov 2025 14:13:58 +0000

In modern DevOps workflows, even the most mature CI/CD pipelines often fail for reasons that aren’t related to code transient network glitches, dependency timeouts, flaky tests, or resource limits. Each of these failures halts deployment and forces engineers to manually restart or troubleshoot pipelines, slowing release velocity and wasting valuable time.

This project introduces a Self-Healing Serverless CI/CD Pipeline built entirely on AWS Developer Tools. The solution intelligently detects pipeline or build failures in real time and automatically recovers from them without human intervention reducing downtime, improving reliability, and embodying the next step toward autonomous DevOps.

Core AWS Services Used
AWS CodePipeline — Orchestrates the CI/CD process (Source → Build → Deploy).
AWS CodeBuild — Builds and tests application code usingbuildspec.yml.
Amazon EventBridge — Detects CodePipeline or CodeBuild failure events instantly.
AWS Lambda (“Pipeline Doctor”) — Analyzes the failure and executes automated recovery actions.
Amazon DynamoDB — Stores retry counts and incident data to avoid infinite loops.
Amazon SNS — Sends notifications to developers if automatic healing fails.
Amazon CloudWatch — Captures logs and metrics for observability.
How It Works
CodePipeline triggers automatically when a commit is pushed to the source repository (CodeCommit or GitHub).
CodeBuild runs build and test commands.
If a build or pipeline fails, EventBridge immediately emits a failure event.
Lambda (“Pipeline Doctor”) receives the event, classifies the failure, and executes a healing playbook, such as: Retrying the failed build, Increasing build timeout, or Retrying the failed pipeline stage. The Doctor records the incident in DynamoDB, increments retry counts, and prevents repeated loops.
If the retry succeeds, CodePipeline resumes automatically and continues to deploy.
If the retry limit is reached, SNS sends a notification to engineers for manual review.

Step 1:
Create the Deploy-target Lambda

go to the AWS Console, search for Lambda then Create function using the parameters below

Name: SelfHealingDemoTarget

Runtime: Node.js 20.x then click Create.

Replace the default code with the contents of: lambda/demo/handler.js

code found https://github.com/Greynalytics/Self-healing-cicd.git

exports.handler = async (event) => {   
console.log("Deploy stage invoked with:", JSON.stringify(event));   
return { statusCode: 200, body: "Hello from Self-Healing Pipeline target!" }; 
};

Then Click Deploy. This is just a harmless endpoint your pipeline will “deploy” by invoking it during the Deploy stage.
**
Step 2:
Create the Code Build project**

Go to the Console, search for Code Build

then click Create project with details below

Name: self-healing-build

Source: https://github.com/Greynalytics/Self-healing-cicd.git

repo = self-healing-src,

branch main

Environment: Managed image → Amazon Linux 2 → Standard 7.0

Service role: create new (default)

Buildspec: Use a buildspec file (it’s already in the repo) the click Save.

My repo includes buildspec.yml that runs test.js ~40% of the time to simulate a flaky test, then zips an app/ folder into build.zip as an artifact.

**
Step 3
Create the CodePipeline **

on the console search for CodePipeline and build a custom pipeline

then click Create pipeline with parameters below

Name: SelfHealingPipeline

Source: github

Build: CodeBuild (select self-healing-cicdbuild)

Deploy stage: choose Add stage → + Add action group

Action provider: Lambda

Function name: SelfHealingcicdTarget

User parameters: { "msg": "deployed" } (optional)

Hit Save and then Create pipeline and let it run once (it may succeed or fail randomly due to the flaky test which is perfect for our demo).

Step 4:
Create the DynamoDB incidents table

go to your Console search for DynamoDB then Create table with parameters below;

Table name: SelfHealingIncidents

Partition key: incidentId (String)

Billing: On-demand (default)

then click Create.

Step 5
Create an SNS topic for alerts

on the console search for SNS then Topic and Create a Topic with the parameters below

Type: Standard

Name: SelfHealingAlerts

Create

then Add subscription → Email → enter your email → confirm via the email you receive.

Note, Copy the Topic ARN as seen above; you’ll need it in the Doctor’s env vars.

**
Step 6
Create the “Pipeline Doctor” Lambda **

to go the Console → Lambda → Create function

Name: PipelineDoctor

Runtime: Node.js 20.x

Create

Open the new function to go Code tab and replace the handler with the contents of:
lambda/doctor/index.js from the repository

Environment variables (Configuration → Environment variables):

TABLE = SelfHealingIncidents

TOPIC_ARN = your SNS topic ARN

MAX_RETRIES = 2

Permissions (Execution role):
Open the function’s Configuration → Permissions → Execution role:

Attach AmazonDynamoDBFullAccess (demo speed; tighten later).

Attach AmazonSNSFullAccess (demo speed; tighten later).

Add inline policy named Doctor-CodeBuild-Access:

{   "Version": "2012-10-17",   
    "Statement": [     { 
    "Effect": "Allow", 
    "Action": ["codebuild:BatchGetBuilds","codebuild:StartBuild","codebuild:UpdateProject"], 
    "Resource": "*" }   ] }

Add inline policy named Doctor-CodePipeline-Access:

{   "Version": "2012-10-17",   "Statement": [     { "Effect": "Allow", "Action": ["codepipeline:RetryStageExecution","codepipeline:GetPipelineExecution"], "Resource": "*" }   ] }

(For production, restrict "Resource": "*" to your specific project/pipeline ARNs.)

Click Deploy on the Lambda code.

**
Step 7
Wire Event-Bridge rules to the Doctor**
on the Console search Amazon EventBridge then click Rules and Create rule

**
Rule A — CodeBuild failures/timeouts**

Name: OnCodeBuildFail

Event pattern — Pre-defined pattern by service:

Service: CodeBuild,

Event type: CodeBuild Build State Change

Add pattern filter:

detail.build-status IN FAILED, TIMED_OUT

Target: Lambda function → PipelineDoctor

then Create Rule

Defining the Eventbridge Rule

Building the event patterns

Select target

Create CodeBuild failure triggers

// CodeBuild failures
{
  "source": ["aws.codebuild"],
  "detail-type": ["CodeBuild Build State Change"],
  "detail": { "build-status": ["FAILED","TIMED_OUT"] }
}

**
Rule B- Code Pipeline action failures**

Name: OnPipelineActionFail

Event pattern → Pre-defined pattern by service:

Service: CodePipeline, Event type: CodePipeline Action Execution State Change

Add pattern filter:

detail.state = FAILED

Target: Lambda function → PipelineDoctor

Create rule

// CodePipeline action failures
{
  "source": ["aws.codepipeline"],
  "detail-type": ["CodePipeline Action Execution State Change"],
  "detail": { "state": ["FAILED"] }
}

Step 8
Test the self-healing flow

Push a new commit to main in pipeline-src/.

Watch Code Pipeline run.

If Build fails (flaky test.js), EventBridge triggers Pipeline Doctor.

Doctor checks retries in DynamoDB, then retries the build (and may bump timeout).

If the retry passes, pipeline turns green which is self-healed.

If retries exhausted , you get an SNS email, and the incident is marked UNHEALED.

You can Open CloudWatch Logs for PipelineDoctor to see which playbook ran.

Simple and Seamless Continuous Integration workflow with AWS Code Build and Code Pipeline

Henry — Sun, 23 Mar 2025 18:40:38 +0000

This project leverages various tools and technologies to optimize Continuous Integration (CI) using AWS Code Build and AWS Code Pipeline:

AWS Code Build: A fully managed build service that compiles source code, runs tests, and generates software packages, automating the entire build process.
AWS Code Pipeline: A CI/CD service that automates software release workflows, managing tasks such as source retrieval, building, testing, and deployment.
GitHub: A version control repository used to store and manage source code, triggering automated builds in AWS Code Build whenever code is committed.
Docker: A containerization platform that packages applications and their dependencies into isolated environments, ensuring consistent performance across different systems.
Docker Hub: A cloud-based repository for storing Docker container images utilized by the application.
AWS Systems Manager: Uses Parameter Store to securely store and manage Docker Hub credentials, ensuring sensitive data remains centralized and protected.

Steps to Get Started:

Fork the GitHub Repository: Clone the repository used for this project. Click here to see GitHub repo to be cloned
Create an IAM Role and Assign Permissions
*Configure AWS Systems Manager Parameter Store *: Aws parameter store offers a robust solution that helps secure our docker credential
Set up AWS Code Build
Setup AWS Code Pipeline
Create IAM Role and Assign Permission

Open the IAM console by searching for “IAM” in the AWS services menu and configuring the necessary permissions.

In the left-hand menu, click on Roles under Access Management.

Click on the Create role button.

Select the service that will use this role, and choose AWS service. Then under use case, select the drop-down button and select AWS code build and hit next below.

Search for and select the following policies to attach to the role: AmazonEC2FullAccess and AmazonSSMFullAccess. Then click on the next button.

Provide a name and description for the role. Optionally, add tags to your IAM role for better organization and management.

Review everything then click on Create role.

Configure AWS Systems Manager Parameter Store
To securely manage and access Docker credentials (such as username, password, and registry URL) within AWS environments, the AWS Systems Manager Parameter Store offers a robust solution. By storing these sensitive values as Secure String parameters, AWS ensures they are encrypted using AWS Key Management Service (KMS), safeguarding the confidentiality and integrity of your credentials.

This setup is essential for integrating Docker-based workflows with AWS services like CodeBuild and CodePipeline, enabling seamless and secure Continuous Integration and Continuous Delivery (CI/CD) pipelines.

Navigate to the AWS Systems Manager Console.

Click on Parameter Store in the left-hand menu.

Click on the Create parameter button.

Enter Parameter Details:

Name: Enter a name for your parameter. Let’s start with the Docker registry URL. Use the following name: /myapp/docker-registry/url.

Description (Optional): Optionally, describe to document the purpose of the parameter.

Tier: For the tier, select Standard.

Type: For the type, select String.

Data Type: Under Data Type, select Text.

Value: For the value, enter docker.io. Finally, click Create Parameter.

Now, let’s create the parameter for the Docker registry password:

Name: Enter /myapp/docker-credentials/password as the name.

Type: Choose Secure String as the parameter type. This ensures that the parameter value is encrypted using an AWS Key Management Service (KMS) key for secure storage.

KMS Key (Optional): Optionally, you can select a specific AWS KMS key to encrypt the parameter value. If none is specified, AWS will use the default KMS key.

Value: Enter your Docker registry password. Finally, click Create Parameter to save.

Use the same process to create the parameter for the Docker registry username:

Name: Enter /myapp/docker-credentials/username.

Type: For the type, select Text.

Finally, click Create Parameter.

You should now have three parameters in total as shown below

/myapp/docker-registry/url

/myapp/docker-credentials/password

/myapp/docker-credentials/username

These Docker credentials are securely stored in AWS Systems Manager Parameter Store.

AWS CodeBuild
Navigate to the AWS CodeBuild Console.

Navigate to build project on the left side of the console and click on Create Project.

Enter a name for your CodeBuild project.

Choose the Source Provider where your build will fetch the code. This could be one of the following:

AWS CodeCommit, GitHub, Bitbucket, and Amazon S3 Next, specify the branch, tag, or revision ID that represents the source code version you want to build.

Define the Build Environment by specifying the following details:

Choose the operating system for your build environment (e.g., Amazon Linux, Ubuntu,).Docker Configuration (if applicable): If your build requires Docker, configure it by selecting the appropriate Docker image or enabling Docker in your environment.

Choose an Existing Service Role: From the dropdown menu, select the IAM role you created in the previous steps.

This role should have the necessary permissions, such as SSM Full Access, to access the Docker credentials stored in AWS Systems Manager, as well as any other required resources.

Enable Privileged Mode: Ensure you check the Privileged box to allow the build environment to run Docker commands or perform elevated tasks.

Choose between the following options for your build specification:

Buildspec File in Source Repository: Select this option if your buildspec.yml file is stored within your source code repository. The build process will reference this file directly.

Alternatively, you can provide the buildspec directly in the console by entering the required build

commands and configurations manually.

paste in the following build command in the buildspec editor section.

version: 0.2
env:
  parameter-store:
    DOCKER_REGISTRY_USERNAME: /myapp/docker-credentials/username
    DOCKER_REGISTRY_PASSWORD: /myapp/docker-credentials/password
    DOCKER_REGISTRY_URL: /myapp/docker-registry/url
phases:
  install:
    runtime-versions:
      python: 3.11
  pre_build:
    commands:
      - echo "Installing dependencies..."
      - pip install -r day-14/simple-python-app/requirements.txt
  build:
    commands:
      - echo "Running tests..."
      - cd day-14/simple-python-app/
      - echo "Building Docker image..."
      - echo "$DOCKER_REGISTRY_PASSWORD" | docker login -u "$DOCKER_REGISTRY_USERNAME" --password-stdin "$DOCKER_REGISTRY_URL"
      - docker build -t "$DOCKER_REGISTRY_URL/$DOCKER_REGISTRY_USERNAME/simple-python-flask-app:latest" .
      - docker push "$DOCKER_REGISTRY_URL/$DOCKER_REGISTRY_USERNAME/simple-python-flask-app:latest"
  post_build:
    commands:
      - echo "Build completed successfully!"
artifacts:
  files:
    - '**/*'
  base-directory: day-14/simple-python-app

Configure where build logs should be stored for monitoring and troubleshooting.

Specify the build artifacts to be generated and where they should be stored (e.g., S3 bucket).

Review the configuration details to ensure accuracy. Click on Create Build Project to create the CodeBuild project.

The Code Build project has now been created and is ready to execute builds based on the specified configuration.

Click Start Build to initiate the build process.

Use the Build Details section to monitor the status of the build.

By following these steps, you can create an AWS CodeBuild project that integrates with your source code repository and is configured to build Docker images or execute other build tasks according to your project requirements.

Verify the Image Was Pushed to the Docker Hub

Navigate to Repository: Go to the repository where you expected the image to be pushed.

Verify Image: Check that the image is present, has the correct tag, and was pushed recently.

By following these steps, you can confirm that your Docker image was successfully pushed to Docker Hub, ensuring that the build process is completed correctly

AWS Code pipeline

In this step, we will set up an AWS CodePipeline to automate the continuous integration (CI) process. AWS CodePipeline is a CI/CD service that automates the steps required to release software changes. It orchestrates various stages, including source retrieval, building, testing, and deployment.

For this article, we will focus on the CI aspect, ensuring that every code change is automatically built and tested.

Open the AWS Management Console.

Navigate to the AWS Code Pipeline service.

Click on Create Pipeline to start the pipeline creation wizard.

For creation options select Build Custom Pipeline then click the next button.

Give your pipeline a name.

New Service Role: Select this option to allow AWS to create a new IAM role for the pipeline.

Existing Service Role: Choose this option if you have a preconfigured IAM role that you want to use for the pipeline

Click Next below

Source Provider: Select the source provider for your code repository (e.g., AWS CodeCommit, GitHub, Bitbucket).

Repository Name: Enter the name of your repository. Specify the branch you want to use (e.g., main or master).

Choose AWS CodeBuild: Select AWS CodeBuild as the build provider.

Choose the region where your CodeBuild project is located.

Select the CodeBuild project you created earlier. Add any necessary environment variables required for the build process.

Skip Code Deploy: Since this guide focuses on Continuous Integration (CI), you may not need a deploy stage at this time. Review all the stages and configurations to ensure everything is set up correctly.

Scroll down and click on Create Pipelineto to finalize and create your CodePipeline.

Trigger the Pipeline: Manually trigger the pipeline by clicking Release Change, or make code changes to your GitHub repository to automatically initiate the pipeline. Monitor the progress of the pipeline in the CodePipeline console. Each stage (Source, Build) will display its status.

Our CodePipeline has been successfully set up, eliminating the need for manual build triggers. Now, the entire process is orchestrated by CodePipeline, which automatically detects changes in our GitHub repository. Once changes are detected, CodePipeline triggers the CodeBuild service to compile and build the code. Additionally, if you check Docker Hub, you’ll find that the latest image has been automatically pushed to our repository

Conclusion

In this project, we successfully set up a Continuous Integration (CI) pipeline using AWS CodePipeline and AWS CodeBuild, with secure Docker credential management handled through AWS Systems Manager. We leveraged AWS developer tools, including CodeBuild and CodePipeline, to streamline the process.

Stay tuned for the second part of this project.