DEV Community

Cover image for βœ… Setting Up Cloud-based CI-CD Pipeline πŸ”„
Madhurima Rawat
Madhurima Rawat

Posted on • Originally published at github.com

βœ… Setting Up Cloud-based CI-CD Pipeline πŸ”„

#cicd #github #githubactions #aws

πŸ›°οΈ Hello Cloud Sentinels!!

May all your logs be clean and your dashboards insightful! πŸ“ˆβ˜οΈ

πŸ“š All code, docs, and resources are available in my GitHub repository:

GitHub logo madhurimarawat / Cloud-Computing

This repository focuses on cloud computing and demonstrates how to set up virtual machines, S3, and other services using LocalStack. It provides a comprehensive guide to simulating AWS services locally for development and testing purposes.

Cloud-Computing

This repository focuses on cloud computing and demonstrates how to set up virtual machines, S3, and other services using LocalStack. It provides a comprehensive guide to simulating AWS services locally for development and testing purposes.

Repo Size GitHub Stars GitHub Forks GitHub Issues Closed Issues Open Pull Requests Closed Pull Requests GitHub Discussions GitHub Contributors Top Language License Last Commit Repository Age Workflow Status GitHub Watchers

Tools and Technologies βš™οΈπŸ’»

1. AWS CLI

AWS Command Line Interface (CLI) is a powerful tool that allows users to interact with AWS services directly from the terminal. It simplifies managing cloud resources by providing commands for a wide range of AWS services, enabling tasks such as provisioning, managing, and automating workflows with ease.

2. LocalStack

LocalStack is a fully functional, local testing environment for AWS services. It enables developers to simulate AWS services on their local machines, facilitating the development and testing of cloud-based applications without needing access to an actual AWS account.

3. Docker

Docker is a containerization platform that allows developers to build, share, and run applications in isolated environments called…


View on GitHub


In the last post,
madhurima_rawat

πŸ“Š Implementing Cloud Monitoring and Logging πŸ› οΈ

Madhurima Rawat ・ May 19

#aws #cloud #cloudcomputing #tutorial

we previously explored monitoring and logging, and how logs are managed within the cloud.

Today, we’re diving into something equally exciting and essential:
πŸš€ Cloud-based CI/CD Pipelines!

First, I’ll walk you through what CI/CD really is, why it’s a game-changer in modern development, and then we’ll move on to its hands-on implementation β€” from seamless code integration to fully automated deployment.

In this experiment, we’ll set up a Continuous Integration/Continuous Deployment (CI/CD) pipeline using GitHub Actions, Docker, and LocalStack. You'll gain practical insights into automating cloud deployments with AWS services, particularly leveraging the AWS CLI and S3 for efficient resource management.

Let’s build pipelines that flow effortlessly β€” like water through perfectly engineered pipes! β˜οΈπŸ”„πŸš€πŸ› οΈ

πŸš€ How It Works

1️⃣ GitHub Actions:

  • Automates build, test, and deployment processes directly from GitHub.
  • Triggers workflows on code commits, pull requests, or scheduled intervals.

2️⃣ Docker:

  • Creates containerized environments for running applications.
  • Ensures that the pipeline runs consistently across different systems.

3️⃣ LocalStack:

  • Simulates AWS cloud services locally (S3, Lambda, DynamoDB, etc.).
  • Allows developers to test AWS-related workflows without real AWS costs.

4️⃣ AWS CLI & S3:

  • AWS CLI automates interactions with AWS services.
  • S3 (Simple Storage Service) acts as a storage bucket for deployment artifacts.

πŸ”Ή Use Cases

βœ… Automated Deployment Pipelines

  • Code is automatically tested, built, and deployed to cloud environments.
  • Reduces manual intervention, ensuring faster release cycles.

βœ… Simulating AWS Services Locally

  • Developers can test AWS-dependent applications without incurring AWS costs.
  • Ideal for offline development or local testing of cloud-native applications.

βœ… Cloud-Based Workflow Testing

  • Ensures infrastructure as code (IaC) principles by defining cloud setups in version control.
  • Useful for DevOps teams deploying applications on AWS.

βœ… Disaster Recovery & Backup Automation

  • CI/CD can automate the creation of S3 backups and deployment rollbacks.
  • Helps maintain data integrity and business continuity.

βœ… Microservices & Serverless Development

  • Supports Lambda function deployment, API Gateway integration, and event-driven applications.
  • Helps teams working on serverless computing streamline their workflow.

🌍 Real-Life Examples

πŸ“Œ E-commerce Platforms

  • Deploy new features to AWS-hosted websites seamlessly without downtime.
  • Test changes in a LocalStack AWS simulation before pushing them live.

πŸ“Œ Financial Services

  • Automate deployment of fraud detection algorithms in a secure pipeline.
  • Ensure compliance by testing AWS interactions locally before deploying.

πŸ“Œ Mobile App Backend Development

  • Automatically deploy backend APIs (hosted on AWS Lambda) after each successful commit.
  • Use LocalStack to test S3 storage operations without using real AWS resources.

πŸ“Œ AI/ML Model Deployment

  • Automate pushing trained ML models to S3 for cloud inference.
  • Use GitHub Actions to validate the model before deployment.

🎯 Key Benefits

πŸ”Ή Faster Development Cycles β†’ Reduces manual deployment efforts.

πŸ”Ή Cost-Efficient Testing β†’ Simulates AWS without incurring costs.

πŸ”Ή Reliable Cloud Automation β†’ Ensures seamless integration & deployment.

πŸ”Ή Enhanced Security β†’ Controlled CI/CD workflow reduces human errors.

πŸ”Ή Scalability β†’ Easily extendable for various AWS services.

πŸ“· Visual Representation

πŸ”— Version Control in CI/CD

πŸ”— CI/CD Pipeline Execution

πŸ–ΌοΈ About the Cover Image:
The visual begins with a cloud icon surrounded by a πŸ”„ CI/CD loop, symbolizing continuous integration and deployment within the cloud environment.

From the cloud, the flow branches out into three development lines, each representing different GitHub branches. These branches visually merge into one, signifying the integration into the main branch β€” a core concept of CI/CD pipelines.

Next, the image shows three connected user icons, representing seamless team collaboration through GitHub and CI/CD processes. It highlights how developers work together, push changes, and build confidently in sync.

🎨 The entire theme follows a black and grey palette, inspired by the GitHub aesthetic.
βœ… The icons appear in green, reinforcing the success-driven nature of CI/CD pipelines β€” where green indicates passing builds, clean merges, and successful deployments.

CI/CD Pipelines with GitHub Actions, Docker, and LocalStack

1. Creating an S3 Bucket

Command: 

aws --endpoint-url=http://localhost:4566 s3 mb s3://my-ci-cd-artifacts
Enter fullscreen mode Exit fullscreen mode

Explanation:

  • aws s3 mb β†’ Creates a new S3 bucket.
  • s3://my-ci-cd-artifacts β†’ The name of the bucket being created.
  • --endpoint-url=http://localhost:4566 β†’ Uses LocalStack to simulate AWS services.

Output: 

make_bucket: my-ci-cd-artifacts
Enter fullscreen mode Exit fullscreen mode

2. Attempting to Create a CodeCommit Repository

Command: 

aws --endpoint-url=http://localhost:4566 codecommit create-
repository --repository-name my-repo
Enter fullscreen mode Exit fullscreen mode

Explanation:

  • aws codecommit create-repository β†’ Creates a new AWS CodeCommit repository.
  • --repository-name my-repo β†’ Assigns the repository name as my-repo.
  • --endpoint-url=http://localhost:4566 β†’ Uses LocalStack.

Error Output: 

An error occurred (InternalFailure) when calling the
CreateRepository operation: API for service 'codecommit'
not yet implemented or pro feature - please check
https://docs.localstack.cloud/references/coverage/ for further information
Enter fullscreen mode Exit fullscreen mode

3. Initializing a Git Repository

Command: 

git init
Enter fullscreen mode Exit fullscreen mode

Explanation:

  • git init β†’ Initializes a new Git repository in the current directory.

Output: 

Initialized empty Git repository in C:/Users/rawat/Documents/8
SEMESTER/Cloud Computing/Lab/Experiment 10/Codes/.git/
Enter fullscreen mode Exit fullscreen mode

4. Staging and Committing Files

Commands: 

git add .
git commit -m "Initial commit"
Enter fullscreen mode Exit fullscreen mode

Explanation:

  • git add . β†’ Stages all files for commit.
  • git commit -m "Initial commit" β†’ Commits the staged files with a message.

Output: 

[master (root-commit) 2dfb5b6] Initial commit
 3 files changed, 1153 insertions(+)
 create mode 100644 Command Prompt Input and Output Explanation.md
 create mode 100644 Command Prompt Input and Output Explanation.pdf
 create mode 100644 Command Prompt Input and Output.txt
Enter fullscreen mode Exit fullscreen mode

5. Uploading a ZIP File to S3

Command: 

aws --endpoint-url=http://localhost:4566 s3 cp my-code.zip
s3://my-ci-cd-artifacts/
Enter fullscreen mode Exit fullscreen mode

Explanation:

  • aws s3 cp β†’ Copies a file to S3.
  • my-code.zip β†’ The file being uploaded.
  • s3://my-ci-cd-artifacts/ β†’ Destination bucket in S3.
  • --endpoint-url=http://localhost:4566 β†’ Uses LocalStack.

Error Output: 

The user-provided path my-code.zip does not exist.
Enter fullscreen mode Exit fullscreen mode

6. Creating a ZIP Archive

Command: 

powershell Compress-Archive -Path * -DestinationPath my-code.zip
Enter fullscreen mode Exit fullscreen mode

Explanation:

  • Compress-Archive -Path * -DestinationPath my-code.zip β†’ Creates a ZIP archive of all files in the directory.

7. Uploading the ZIP File Again

Command: 

aws --endpoint-url=http://localhost:4566 s3 cp my-code.zip
s3://my-ci-cd-artifacts/
Enter fullscreen mode Exit fullscreen mode

Output: 

upload: .\my-code.zip to s3://my-ci-cd-artifacts/my-code.zip
Enter fullscreen mode Exit fullscreen mode

8. Listing the Uploaded Files in S3

Command: 

aws --endpoint-url=http://localhost:4566 s3 ls s3://my-ci-cd-artifacts/
Enter fullscreen mode Exit fullscreen mode

Output: 

2025-03-08 10:32:42     289415 my-code.zip
Enter fullscreen mode Exit fullscreen mode

9. Setting Up a Remote Git Repository

Commands: 

git remote add origin https://github.com/madhurimarawat/Cloud-
Computing.git
git branch -M main
git push -u origin main
Enter fullscreen mode Exit fullscreen mode

Explanation:

  • git remote add origin <repo-url> β†’ Links the local repository to GitHub.
  • git branch -M main β†’ Renames the current branch to main.
  • git push -u origin main β†’ Pushes the code to GitHub.

Error Output: 

To https://github.com/madhurimarawat/Cloud-Computing.git
 ! [rejected]        main -> main (fetch first)
error: failed to push some refs to
'https://github.com/madhurimarawat/Cloud-Computing.git'
hint: Updates were rejected because the remote contains
work that you do not
hint: have locally. This is usually caused by another
repository pushing to
hint: the same ref. If you want to integrate the remote changes, use
hint: 'git pull' before pushing again.
Enter fullscreen mode Exit fullscreen mode

Fix:

To resolve this issue, run:

git pull origin main --rebase
git push -u origin main
Enter fullscreen mode Exit fullscreen mode

10. Pulling the Latest Changes from GitHub

Command: 

git pull origin main --rebase
Enter fullscreen mode Exit fullscreen mode

Explanation:

  • Fetches changes from the remote repository and applies them using rebase instead of a merge.
  • Ensures a linear commit history by reapplying local changes on top of the latest remote changes.

Output: 

remote: Enumerating objects: 240, done.
remote: Counting objects: 100% (240/240), done.
remote: Compressing objects: 100% (212/212), done.
remote: Total 240 (delta 100), reused 43 (delta 21), pack-reused 0
Receiving objects: 100% (240/240), 9.22 MiB | 1.11 MiB/s, done.
Resolving deltas: 100% (100/100), done.
From https://github.com/madhurimarawat/Cloud-Computing
 * branch            main       -> FETCH_HEAD
 * [new branch]      main       -> origin/main
Successfully rebased and updated refs/heads/main.
Enter fullscreen mode Exit fullscreen mode

11. Staging All Changes

Command: 

git add .
Enter fullscreen mode Exit fullscreen mode

Explanation:

  • Stages all modified and newly created files in the current directory for the next commit.

12. Checking for an Ongoing Rebase

Command: 

git rebase --continue
Enter fullscreen mode Exit fullscreen mode

Explanation:

  • Used to continue an ongoing rebase operation if there are conflicts.
  • In this case, the error means there was no ongoing rebase, so this step was unnecessary.

Output: 

fatal: no rebase in progress
Enter fullscreen mode Exit fullscreen mode

13. Pushing Changes to GitHub

Command: 

git push -u origin main
Enter fullscreen mode Exit fullscreen mode

Explanation:

  • Pushes local changes to the remote repository (origin), setting main as the upstream branch.
  • This makes future git push commands simpler by automatically pushing to origin main.

Output: 

Enumerating objects: 6, done.
Counting objects: 100% (6/6), done.
Delta compression using up to 8 threads
Compressing objects: 100% (5/5), done.
Writing objects: 100% (5/5), 282.43 KiB | 31.38 MiB/s, done.
Total 5 (delta 1), reused 1 (delta 0), pack-reused 0
remote: Resolving deltas: 100% (1/1), completed with 1 local object.
To https://github.com/madhurimarawat/Cloud-Computing.git
   b201b02..eb4faf7  main -> main
branch 'main' set up to track 'origin/main'.
Enter fullscreen mode Exit fullscreen mode

Output Breakdown:

  • Delta compression β†’ Reduces the size of transmitted data.
  • Objects written successfully β†’ Confirms the push was successful.
  • Tracking branch set up β†’ Future git push commands will default to origin main.

14. Viewing the YAML Deployment Workflow

Link:

View the deployment YAML file

Purpose:

This GitHub Actions workflow automates a manual deployment process by performing the following steps:

Workflow Breakdown:

  • Triggering the Workflow

    • The workflow is manually triggered using workflow_dispatch, meaning it does not run automatically on commits or merges.

  • Job Execution

    • A single job named deploy is executed on Ubuntu-latest, the default GitHub-hosted runner.

  • Steps in the Workflow

  1. Checkout Repository
 - Uses `actions/checkout@v4` to fetch the repository contents into the GitHub Actions runner.
Enter fullscreen mode Exit fullscreen mode
  1. (Optional) Install AWS CLI 
 - This step is commented out but would install the AWS CLI if needed.
Enter fullscreen mode Exit fullscreen mode
  1. (Optional) Zip the Repository 
 - Another commented-out step that creates a ZIP archive of the repository.
Enter fullscreen mode Exit fullscreen mode
  1. (Optional) Upload to LocalStack S3 
 - Demonstrates an attempt to upload the ZIP file to a **LocalStack S3 bucket**.
 - **⚠️ This step does NOT work in GitHub Actions**, since LocalStack would need to be running on the same machine.
Enter fullscreen mode Exit fullscreen mode
  1. Print Success Message
    • Simply prints "Successfully run!" to indicate that the workflow has been executed.

Key Considerations:

  • This workflow is primarily a template for deploying to LocalStack S3.
  • Since GitHub Actions runs on cloud-hosted VMs, it cannot access LocalStack running locally.
  • We can modify this workflow to deploy to a real AWS S3 bucket by configuring proper AWS credentials.

πŸ“„ Want to see how everything came together step by step? Check it out here:
πŸ”— Experiment 10 Output (PDF)

🧠 Curious about the exact commands and how they function? Explore the detailed input-output flow:
πŸ–₯️ CI/CD Setup Input-Output Flow (PDF)

πŸŽ‰ And that’s a wrap on Cloud-based CI/CD Pipelines!

πŸ“š Got awesome CI/CD tips, resources, or cheat sheets? Feel free to drop them in the comments β€” I’d love to explore and share them!

πŸ“˜ Want a compact version of all experiments with explanations and outputs in one place?
Grab the complete PDF here:
πŸ“‚ Cloud Computing Lab Manual (PDF)

πŸ’– This is the final article in this series.
Thank you all so much for sticking with me and exploring the world of Cloud Computing through each of these posts! It’s been an incredible journey, and I hope you found each article insightful and practical.

πŸ’¬ I’d love to hear from you:
✨ Which article was your favorite?
✨ How did you like the series overall?
✨ And most importantly β€” what should I explore next?

Here are a few exciting ideas I’m considering for the next series:

  • πŸ“Š Big Data Analytics using Hadoop
  • 🏒 Data Warehousing using MySQL
  • 🐍🧹 Data Wrangling using Python and MySQL
  • πŸ’» A community-powered series based on my repo:

    GitHub logo madhurimarawat / CodeCulture-Daily

    A daily programming challenge repository where fun meets learning! With 39 challenges over 39 days, it helps coders enhance skills through practical tasks and interview prep. While maintenance is paused, contributions are welcome, and future expansions are planned to keep learning ongoing.

    πŸš€ CodeCulture-Daily

    A daily programming challenge repository where fun meets learning! Solve challenges, learn new tools and technologies, and get featured as a top contributor. Join now to sharpen your coding skills!

    β€œIt's not about ideas. It's about making ideas happen.” – Scott Belsky

    CodeCulture-Daily Banner

    Illustration 1

    Welcome to CodeCulture-Daily, your go-to spot for daily programming and tech challenges! πŸŽ‰

    By practicing daily, you'll not only sharpen your problem-solving skills but also learn and master various tools, technologies, algorithms, and much more. Whether you're starting as a beginner or pushing through advanced levels, CodeCulture-Daily is the perfect platform to grow your knowledge. πŸ’‘

    🌟 What to Expect

    • πŸ“… Daily Challenges: Every day at 7 PM, a new challenge will be posted in this repository.
    • πŸ† Three Levels: Each challenge is categorized by difficulty
      • 🟒 Beginner
      • 🟑 Intermediate
      • πŸ”΄ Advanced
    • πŸ“‚ Domain Categories
      • Tech πŸ’»
        • πŸ–₯️ Programming Languages…


    View on GitHub

    πŸ“Œ In this one, I’ll post one question daily β€” and we’ll solve it together in the comments. How fun would that be?

πŸ‘‰ Let me know in the comments which one you'd love to dive into next!

Until the next article β€” goodbye for now! 🌟
Let’s meet again in the next series. πŸ™ŒπŸ’»πŸš€

Top comments (0)