DEV Community: DevCloud Ninjas Scrolls

End to End CI/CD pipeline using GitHub Actions for Android Application

Lionel♾️☁️ — Fri, 13 Dec 2024 18:39:19 +0000

In this article, you will get a brief idea about how to create an End to End CI/CD Pipeline using GitHub Actions for an Android Application.

Project Source Code : LINK

Here, we will be covering some use cases like how to trigger one workflow from another workflow, how to run 2 jobs which depends upon each other, how to add public ip of GitHub Actions in security groups of Jfrog which is running in an EC2 , on port 8082 — so that GitHub Actions can access Jfrog to upload the .apk file into repository, how to integrate SonarQube , Teams with GitHub Actions , how to create a cron job in workflow, how to delete artifacts which are created during the workflow in GitHub Actions, how to clean caches in GitHub Actions which gets created every time your run a workflow.

Step by Step Process :

To create a workflow, go to Actions in your GitHub Repository and choose a template yml file or click “set up a workflow yourself” — for me I have chosen Android CI as my application is android application.

This basic template you will get to do CI part of Android Application.

Let me explain you some of the terms used in workflow file:

name : The name of the workflow as it will appear in the “Actions” tab of the GitHub repository. Like here it is “Android CI”

on: Specifies the trigger for this workflow. So here the workflow will be triggered when there is a push event in “main” branch and pull_request event in “main” branch

jobs: A workflow job is a set of steps that execute on the same runner. We can have multiple jobs in a single workflow yml file. Groups together all the jobs that run in the Android CI workflow. Here, in the example there is a single job whose name is build

runs-on: Configures the job to run on the latest version of an Ubuntu Linux runner. This means that the job will execute on a fresh virtual machine hosted by GitHub. You can use windows and macOS runner too.

steps: Groups together all the steps that run in the build job. Each item nested under this section is a separate action or shell script.

uses: actions/checkout@v3 : The uses keyword specifies that this step will run v3 of the actions/checkout action. This is an action that checks out your repository onto the runner, allowing you to run scripts or other actions against your code (such as build and test tools). You should use the checkout action any time your workflow will run against the repository's code.

uses: actions/setup-java@v3 : This step uses the actions/setup-java@v3 action to install the specified version of the JDK (this example uses v11) of distribution: 'temurin'

run: chmod +x gradlew: The run keyword tells the job to execute a command on the runner. In this case, you are granting execute permission for gradlew

run: ./gradlew build: In this case you are building the code using gradle

After this , Click on Start Commit and add comment and click Commit. This will create a basic android CI workflow in GitHub Action.

To create a secret in GitHub Actions  Go to Settings and then click on Secrets and then to Actions and create different secrets .

Let’s modify the android.yml file : Full Workflow

Let’s discuss part by part

Here, the workflow is getting triggered whenever we push into the branches “main” or “qa” or “develop” and whenever we pull request into the branches “main” or “qa”.

You can use environment variables to store information that you want to reference in your workflow. You reference environment variables within a workflow step or an action, and the variables are interpolated on the runner machine that runs your workflow. Commands that run in actions or workflow steps can create, read, and modify environment variables.

You can define environment variables that are scoped for:

The entire workflow, by using env at the top level of the workflow file.
The contents of a job within a workflow, by using jobs.<job_id>.env.
A specific step within a job, by using jobs.<job_id>.steps[*].env.

Here in the workflow we can have created a env at the top level with variable name as “AWS_DEFAULT_REGION” and assigned value as “ap-south-1”

uses: actions/setup-java@v3 : This step uses the actions/setup-java@v3 action to install the specified version of the JDK (this example uses v11) of distribution: 'temurin'

run: chmod +x gradlew: The run keyword tells the job to execute a command on the runner. In this case, you are granting execute permission for gradlew

run: ./gradlew clean: Gradle clean will delete a build directory if already present.

run: ./gradlew lint: It will detect poorly structured code that can impact the reliability and efficiency of your Android apps and make your code harder to maintain

run: ./gradlew build: A process of building a Gradle project

run: ./gradlew jacocoTest: The JacocoReport task can be used to generate code coverage reports in different formats

In this step we are integrating GitHub Actions with SonarQube. To make your workflows faster and more efficient, you can create and use caches for dependencies and other commonly reused files.

In the Cache SonarQube Package Step and Cache Gradle Package we are caching SonarQube packages and Gradle Packages with given path where runner stores the cache. The new cache will use the key you provided and contains the files you specify in path and alternative restore key used if no cache hit occurs for key, these restore keys are used sequentially in the order provided to find and restore a cache.

In the next step we are doing code analysis using SonarQube. We have added environment variables whose values are as secrets which you can use in your workflows as environment variables.

The secrets in GitHub Actions are defined as {{ secrets.secret_name }}. Here we have added SonarQube Token and SonarQube URL as secrets. Then we are running the command in runner as “./gradlew sonarqube” and passing the projectkey.

In build.gradle we have added the plugins for SonarQube

We can see that SonarQube analysis is passed and coverage report is greater than 80%

In the next step “Date and Time” , we have evaluated date and time using Linux command and created outputs in the step by writing to stdout in the format of ::set-output name=<name>::<value>. A step can have multiple outputs. Steps that create outputs must have unique ids.

current_date_time::$(date +”%d-%m-%Y-%H-%M-%S”)  Here current_date_time variable is in the format of %d : Day of the month , %m : Month , %Y : Year , %H : Hour, %M : Minutes, %S : Seconds

Here, the output name is “current_date_time” and id of the step of “Date and Time” is “date” which is unique name in the workflow.

To use this parameter in the job we can use in the way {{ steps.<step-id>.outputs.<output-name> }} . Here in the example it is {{ steps.date.outputs.current_date_time }}

In the next step “Copy APK files to a directory” we are creating a directory structure to store the Debug and Release APK files in the format

apk-files > debug > app-debug-11–11–2022–09–09–12–36.apk

apk-files > release> app-release-unsigned-11–11–2022–09–09–12–36.apk

In the last step “Upload apk-files” Directory we are uploading artifactory — APK Files which we created so that in the deploy “job” we can download this artifactory as it will be using a new ubuntu runner. Here {{ github.workspace }} is default path for the checkout action. The path which we want to upload is the apk-files directory and if-no-files is present in the path , then ignore this step.

In this step we are integrating teams with GitHub Actions. This action take your GitHub token and the webhook URL which is generated during the configuration part .

Create a teams channel add people who should be notified for workflow success and fail steps . Click Connectors in the channel and then choose “Incoming Webhook” and add and then configure and then add a name and copy the URL and paste it as a secret in your GitHub Secrets and use it in your workflow.

If you see the last part, we are creating an outputs variable name as “CURRENT_DATE_TIME” and passing the date time value . Since this variable we want to use in another job “deploy” . To pass any variables from 2 different jobs we need to create output values like this.

Then we are creating another job named as “deploy” where we are adding our CD part of the Workflow.

Here “needs” : build means after successfully executing the “build” job only this “deploy” job will run.

Then the if condition is telling that if branches are “qa” or “master” then only this steps will run inside the runner.

Then again in the steps checkout repository is happening and in the next step “Download apk-files Artifactory” , we are downloading the artifact which we just uploaded in “build” job. We have mentioned the path where we need to download the artifact

Then in “Display structure of downloaded files” step we are checking the directory structure of downloaded artifacts.

In the next step “Public IP of GitHub Hosted Runner”  we are generating the Public IP of the GitHub Hosted Runner by using the action haythem/public-ip@1.3

Then in the next step we are adding the Public IP to Security Group in which Jfrog is running in the EC2 Instance so that GitHub Actions can access the Jfrog Page at 8082 using AWS CLI Commands “authorize-security-group-ingress”. For this we need to create a user with having EC2 Full Access permission and give programmatic access to get access_key_id and secret_access_key.

In the first step we are downloading the Jfrog CLI with latest version. Add the JF_URL which is the URL of the artifactory where we are storing the .apk files and access token which we can create in Admin → User Management → Access Token .

Also set the “Password Encryption Policy” to Unsupported for demo purpose.

Then in the next step we are creating folders for QA and Master Branch. Where in script we are using if and else condition , that if GitHub branch is qa then we will create a directory QA in apk-files directory else master. Now the directory structure will be like:

apk-files > qa > debug > app-debug-11–11–2022–09–09–12–36.apk

apk-files > qa> release> app-release-unsigned-11–11–2022–09–09–12–36.apk

apk-files > master > debug > app-debug-11–11–2022–09–09–12–36.apk

apk-files > master > release> app-release-unsigned-11–11–2022–09–09–12–36.apk

In the step “Upload APK files to Jfrog” we are using Jfrog CLI commands to upload apk files from Ubuntu Runner to Jfrog Artifactory –> “android-artifact”

jf rt u — url ${{ secrets.JF_URL }} — user ${{ secrets.JF_USER }} — password ${{ secrets.JF_PASSWORD }} apk-files/qa/debug/app-debug-${{ needs.build.outputs.CURRENT_DATE_TIME }}.apk android-artifact/

Here, u means upload , — url means Artifactory Repository URL (android-artifact) one, — user means Username of Jfrog UI and –password means Password of Jfrog UI , <path of the file want to upload> <artifact-repo-name>/

This URL to file is the JF_URL which we need to add a secret in GitHub Actions.

Then in the next step , we are removing the GitHub Actions Public IP from the security group of Jfrog EC2 Instance . if: always() make sure that this step runs always although if any steps fails.

Then in the last step we are sending notifications to teams. You can see all these information will go to teams.

Use Case : If we want to delete the caches which are formed after running each workflow

Here, we are using workflow_run command  It allows you to execute a workflow based on execution or completion of another workflow. So, here we are telling that “Clear Cache” Workflow will run only when workflow “Android CI and CD” workflow will complete (type of activity) successfully. Then we add permissions as write in a top-level key, to apply to all jobs in the workflow. When you add the permissions key within a specific job, all actions and run commands within that job that use the GITHUB_TOKEN gain the access rights you specify.

After that we run a script where we are first listing all Caches using JavaScript command and then we are deleting caches using their ID.

You can see all the caches got deleted which got created in the previous workflow.

Use Case : If we want to delete the artifacts which are formed when we uploaded the artifacts (apk-files) so that we can pass that directory from build job to deploy job using cron job — Every HOUR

Here, we are running a cron job which will run every hour and delete all artifacts which got created . Here we are passing the GitHub token in purge-artifacts action.

Use Case : How to create Self Hosted Runner and how to configure the self-hosted runner application as a service.

To create self-hosted runner , Go to Settings  Actions  Runner and click create and select the type of OS you have . For me I am choosing Linux OS.

Run the commands which you get here.

Add all the values when asking about Runner Registration.

To connect to the Runner we need to start the run.sh file

We can see that the runner is up and running now.

If you want to configure the self-hosted runner application as a service so that the runner is up and running if your Linux machine is up and running

Run these commands :

Installing the service

1. Stop the self-hosted runner application if it is currently running.

2. Install the service with the following command:

sudo ./svc.sh install

3. Alternatively, the command takes an optional user argument to install the service as a different user.

./svc.sh install USERNAME

Starting the service

Start the service with the following command:

sudo ./svc.sh start

Checking the status of the service

Check the status of the service with the following command:

sudo ./svc.sh status

If you want test with Android Application and want all codes : Check this out

Buy me a coffee :) ← — — If you like my articles

DevOps from 0 to Hero - for Freshers

Lionel♾️☁️ — Sun, 20 Oct 2024 08:32:44 +0000

Introduction

DevOps is a transformative culture and set of practices that bring together software development (Dev) and IT operations (Ops). It aims to shorten the development lifecycle, deliver continuous integration and continuous delivery (CI/CD), and ensure high software quality. If you're a fresher with zero knowledge in DevOps, this guide will help you get started on your journey to becoming a proficient DevOps engineer.

📚 Step-by-Step Guide

1. Understand the Basics

1.1 What is DevOps?

DevOps is a set of practices that combines software development and IT operations. It emphasizes collaboration, communication, and integration between developers and IT operations teams. DevOps aims to automate and streamline the processes of building, testing, and deploying software.

1.2 Core DevOps Principles

Continuous Integration (CI): Regularly merging code changes into a central repository to detect and fix integration issues early.
Continuous Delivery (CD): Automating the process of deploying code changes to production after passing rigorous automated tests.
Infrastructure as Code (IaC): Managing and provisioning infrastructure through code, enabling version control and automated deployment of infrastructure.
Microservices Architecture: Breaking down applications into smaller, independently deployable services for improved scalability and maintainability.
Monitoring and Logging: Implementing robust systems to track application performance and quickly identify issues.

2. 🔧 Learn the Foundation Skills

2.1 Basic Programming

Learning a programming language is essential for automating tasks and writing scripts. Some widely used languages in DevOps are:

Python: Known for its simplicity and readability, Python is great for scripting and automation.
- Learn Python
Go: Gaining popularity in DevOps for its performance and concurrency features.
- Learn Go
JavaScript: Often used in web development and automation tasks.
- Learn JavaScript

2.2 Operating Systems

Understanding operating systems, especially Linux, is crucial as most DevOps tools and environments run on Linux. Learn basic commands, file systems, process management, and networking in Linux.

Linux Tutorial

2.3 Networking Basics

Understanding networking fundamentals is important for configuring and managing servers, containers, and applications. Learn about IP addresses, DNS, HTTP/HTTPS, firewalls, and load balancers.

Networking Basics

3. 🌿 Dive into Version Control

3.1 Git

Git is a version control system that tracks changes in source code, allowing multiple developers to work on a project simultaneously without conflicts. Learn the basics of Git commands and workflows.

3.2 GitHub

GitHub is a platform for hosting Git repositories, providing tools for collaborative development, code review, and project management.

4. 🔄 Master Continuous Integration and Continuous Delivery (CI/CD)

4.1 Jenkins

Jenkins is an open-source automation server that helps automate parts of the software development process, including building, testing, and deploying code.

4.2 GitLab CI/CD

GitLab CI/CD is a powerful tool integrated with GitLab for automating the entire DevOps lifecycle. Learn how to create and manage CI/CD pipelines.

5. Explore Configuration Management

5.1 Ansible

Ansible is an open-source automation tool used for configuration management, application deployment, and task automation. It uses simple, human-readable YAML templates to define automation jobs.

5.2 Puppet

Puppet is a configuration management tool that helps automate the provisioning and management of infrastructure.

6. 🐳 Understand Containerization and Orchestration

6.1 Docker

Docker is a platform for developing, shipping, and running applications inside containers. Containers are lightweight, portable, and consistent environments that ensure applications run the same way regardless of where they are deployed.

6.2 Kubernetes

Kubernetes is an open-source container orchestration platform that automates the deployment, scaling, and management of containerized applications. It helps manage containerized applications in a clustered environment.

7. ☁️ Explore Cloud Platforms

7.1 AWS (Amazon Web Services)

AWS is a comprehensive cloud computing platform offering a wide range of services, including compute, storage, and databases. Learn the basics of AWS services such as EC2 (virtual servers), S3 (object storage), RDS (relational databases), and Lambda (serverless computing).

7.2 Azure

Azure is Microsoft's cloud computing platform that provides a variety of cloud services, including those for compute, analytics, storage, and networking. Familiarize yourself with Azure's offerings and capabilities.

7.3 Google Cloud Platform (GCP)

GCP is Google's cloud computing service, offering a range of services such as compute, storage, and machine learning. Learn about GCP's infrastructure and services.

8. Learn Infrastructure as Code (IaC)

8.1 Terraform

Terraform is an open-source tool for building, changing, and versioning infrastructure safely and efficiently. It allows you to define and provision infrastructure using a high-level configuration language.

9. 📊 Implement Monitoring and Logging

9.1 Prometheus

Prometheus is an open-source monitoring system and time-series database that is well-suited for monitoring containerized applications.

9.2 ELK Stack (Elasticsearch, Logstash, Kibana)

The ELK Stack is a powerful set of tools for searching, analyzing, and visualizing log data. Elasticsearch stores and indexes log data, Logstash processes it, and Kibana visualizes it.

10. Get Hands-On Experience

10.1 Build Projects

Apply what you've learned by working on real projects. Here are some ideas:

Build a complete CI/CD pipeline for a sample application
Deploy a microservices architecture on Kubernetes
Implement a multi-cloud disaster recovery solution

Practical experience is crucial for mastering DevOps.

11. 🤝 Join the Community

Participate in DevOps communities, forums, and meetups to learn from others, share your experiences, and stay updated on industry trends.

12. 📚 Continuous Learning

12.1 Online Courses

Enroll in online courses to deepen your understanding and keep your skills up-to-date. Many platforms offer comprehensive DevOps courses taught by industry experts.

12.2 Books

Read books on DevOps practices, tools, and methodologies. Some highly recommended books are:

The Phoenix Project by Gene Kim, Kevin Behr, and George Spafford: A novel about IT, DevOps, and helping your business win.
The DevOps Handbook by Gene Kim, Jez Humble, Patrick Debois, and John Willis: How to create world-class agility, reliability, and security in technology organizations.

DevOps is not just about tools, but also about fostering a culture of collaboration, continuous improvement, and shared responsibility. Embrace the DevOps mindset in your work and interactions with team members.

📚 Security in DevOps (DevSecOps)

As you progress in your DevOps journey, don't forget to integrate security practices into your workflows. DevSecOps emphasizes the importance of building security into every stage of the development process.

Career Paths in DevOps

DevOps offers various career paths and specializations. Some roles you might consider as you progress in your career include:

DevOps Engineer
Site Reliability Engineer (SRE)
Cloud Architect
Automation Specialist
DevSecOps Engineer

DevOps Podcasts and YouTube Channels

Stay up-to-date with the latest in DevOps through these popular podcasts and YouTube channels:

Podcasts:

YouTube Channels:

A Day in the Life of a DevOps Engineer

To give you a practical perspective of what it's like to work as a DevOps engineer, here's a typical day:

Time	Activity
8:00 AM	Start the day by checking monitoring dashboards for any overnight issues
9:00 AM	Attend the daily stand-up meeting with the development team to discuss ongoing projects and potential blockers
10:00 AM	Work on automating a deployment process using Jenkins and Ansible
12:00 PM	Lunch break and catch up on the latest DevOps news and trends
1:00 PM	Troubleshoot a production issue reported by the operations team
3:00 PM	Collaborate with developers to optimize a Docker container for a new microservice
4:00 PM	Review and merge pull requests for infrastructure-as-code changes
5:00 PM	Document the day's work and plan for tomorrow's tasks

This schedule can vary greatly depending on the organization and current projects, but it gives you an idea of the diverse tasks a DevOps engineer might handle in a day.

Conclusion

Learning DevOps from scratch may seem daunting, but with the right approach and resources, you can master the essential skills and become a proficient DevOps engineer. Follow this step-by-step guide, practice consistently, and engage with the DevOps community to accelerate your learning journey.

Stay curious, and embrace the DevOps mindset to drive innovation and efficiency in software development and operations! 🎉

Thank you for reading our blog …:)