Sri for AWS Community Builders

Posted on Dec 31, 2022

How to set up Jenkins and a Pipeline on AWS

#aws #jenkins #tutorial

Introduction
Create an IAM User
Create a Key pair
Create a Security Group
Create an EC2 instance
Install and Configure Jenkins
Create a Pipeline
Clean Up
Summary
Referrals

Introduction

In this blog we are going to set up Jenkins on an EC2 instance and then set up a pipeline to copy a file from S3.
We are going to start something very basic and enhance it as we move along.

What is Jenkins?

Jenkins offers a simple way to set up a continuous integration or continuous delivery (CI/CD) environment for almost any combination of languages and source code repositories using pipelines, as well as automating other routine development tasks. While Jenkins doesn’t eliminate the need to create scripts for individual steps, it does give you a faster and more robust way to integrate your entire chain of build, test, and deployment tools than you can easily build yourself.

What is Jenkins Pipeline?

Jenkins Pipeline is a suite of plugins which supports implementing and integrating continuous delivery pipelines into Jenkins.

Pipeline provides an extensible set of tools for modeling simple-to-complex delivery pipelines as code via the Pipeline domain-specific language (DSL) syntax.

Demo

Let's get started with the demo.

Step 1. Create an IAM User

Navigate to IAM > Users > Click on Add users
Enter User Name as JenkinsUser
Select Access key - Programmatic access and Click Next: Permissions
Click Attach existing policies directly and then Create policy
In the Create policy window, Click on JSON and add the following JenkinsEC2Policy and save the policy as JenkinsEC2Policy
Select AmazonS3ReadOnlyAccess and JenkinsEC2Policy under Attach existing policies directly
Click Next:Tags, Next:Review and then Create user

Download the credentials .csv file as we will need this during Jenkins configuration at Step 6.

JenkinsEC2Policy

{
"Version": "2012-10-17",
"Statement": [
    {
        "Sid": "Stmt1312295543082",
        "Action": [
            "ec2:DescribeSpotInstanceRequests",
            "ec2:CancelSpotInstanceRequests",
            "ec2:GetConsoleOutput",
            "ec2:RequestSpotInstances",
            "ec2:RunInstances",
            "ec2:StartInstances",
            "ec2:StopInstances",
            "ec2:TerminateInstances",
            "ec2:CreateTags",
            "ec2:DeleteTags",
            "ec2:DescribeInstances",
            "ec2:DescribeInstanceTypes",
            "ec2:DescribeKeyPairs",
            "ec2:DescribeRegions",
            "ec2:DescribeImages",
            "ec2:DescribeAvailabilityZones",
            "ec2:DescribeSecurityGroups",
            "ec2:DescribeSubnets",
            "iam:ListInstanceProfilesForRole",
            "iam:PassRole",
            "ec2:GetPasswordData"
        ],
        "Effect": "Allow",
        "Resource": "*"
    }
]
}

IAM_JenkinsUser1

IAM_JenkinsUser2

IAM_JenkinsUser3

Step 2. Create a key pair

Navigate to EC2 > Key Pairs (under Network & Security).
Click Create key pair.
Navigate to the folder where the key pair is downloaded and run.

   chmod 400 <key_pair_name>.pem

Jenkins_Keypair

Step 3. Create a Security Group

We are going to create a Security Group for SSH and Jenkins web access.
Navigate to EC2 > Security Groups > Create a new security group for your ALB, and set the following values:
- Name: JenkinsSG.
- Add an Inbound rule to allow SSH (TCP 22) traffic from My IP.
- Add another Inbound rule to allow Custom (TCP 8080) traffic from My IP.

Jenkins_SG

Step 4. Create an EC2 instance

Navigate to EC2 > EC2 Dashboard > Click on Launch instance.
Launch an instance with the following values as shown in the screenshots.
When the instance state is Running, select the instance and click on Connect and then copy the connections details.

Connect to the instance.

ssh -i "Jenkins-Keypair.pem" ec2-user@ec2-54-211-70-130.compute-1.amazonaws.com

Jenkins_EC1

Jenkins_EC2

Step 5. Install and Configure Jenkins

Install Jenkins

Ensure that the software packages are up to date on the instance by executing the following command:
```
[ec2-user ~]$ sudo yum update –y
```

Add the Jenkins repo using the following command:

[ec2-user ~]$ sudo wget -O /etc/yum.repos.d/jenkins.repo https://pkg.jenkins.io/redhat-stable/jenkins.repo

Import a key file from Jenkins-CI to enable installation from the package:

[ec2-user ~]$ sudo rpm --import https://pkg.jenkins.io/redhat-stable/jenkins.io.key

[ec2-user ~]$ sudo yum upgrade

Install Java:

[ec2-user ~]$ sudo amazon-linux-extras install java-openjdk11 -y

Install Jenkins:

[ec2-user ~]$ sudo yum install jenkins -y

Enable Jenkins service to auto start at boot:

[ec2-user ~]$ sudo systemctl enable jenkins

Start Jenkins as a service:

[ec2-user ~]$ sudo systemctl start jenkins

You can check the status of the Jenkins service using the command:
```
[ec2-user ~]$ sudo systemctl status jenkins
```
Just in case we ever need to restart Jenkins during this setup/configuration.
```
[ec2-user ~]$ sudo systemctl restart jenkins
```

Configure Jenkins

Copy the Public IPv4 DNS of EC2 instance and paste the URL as following in the browser For Example: http://ec2-54-211-70-130.compute-1.amazonaws.com:8080

Enter the initialAdminPassword from /var/lib/jenkins/secrets/initialAdminPassword and Click Continue

[ec2-user@ip-172-31-89-157 ~]$ sudo cat /var/lib/jenkins/secrets/initialAdminPassword
    671x0x5x3xxx46xxxxxx099x1xf0149x

Select Install suggested plugins.
Once the installation is complete, Create First Admin User will open. Enter your information, and then select Save and Continue.
Click Dashboard, select Manage Jenkins, and then select Manage Plugins.
Click Available plugins, Search and Select Amazon EC2 and then **Install without restart*
Once the installation is complete, Navigate back to Dashboard, select Manage Jenkins, select Manage nodes and clouds and then click on Configure Clouds.
Select Add a new cloud, and select Amazon EC2. A new pop up window opens with more fields.
Click Add under Amazon EC2 Credentials and Select Jenkins
- From the Jenkins Credentials Provider: Jenkins, select AWS Credentials as the Kind.
- Enter Access Key ID, Secret Access Key from the key pair and Click Add.
Scroll down to Region and select your Region.
Click Add under EC2 Key Pair's Private Key and Select Jenkins.
- From the Jenkins Credentials Provider: Jenkins, select SSH Username with private key as the Kind and set the Username to ec2-user.
- Select Enter Directly under Private Key, then select Add.
- Open the private key pair you created in the creating a key pair step and paste in the contents from -----BEGIN RSA PRIVATE KEY----- to -----END RSA PRIVATE KEY-----. Select Add when completed.
Scroll down to Test Connection and ensure it states Success and then Click Save.

Jenkins_Configure1

Jenkins_Configure2

Jenkins_Configure3

Jenkins_Configure4

Jenkins_Configure5

Jenkins_Configure6

Jenkins_Configure7

Jenkins_Configure8

Jenkins_Configure9

Jenkins_Configure10

Jenkins_Configure10.1

Jenkins_Configure10.2

Jenkins_Configure11

Jenkins_Configure11.1

Jenkins_Configure11.2

Jenkins_Configure12

Step 6. Create a Pipeline

Navigate to Dashboard > Manage Jenkins and copy the ID of IAM User created in Step 1, which we need to replace xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx with the ID.
Navigate to Dashboard > Manage Jenkins > Plugin Manager > Click on Available plugins
Search for AWS Steps and Install without restart.
Navigate to Dashboard > New Job > Enter download-a-file-from-s3 > Select Pipeline and Click OK.
Scroll down to Pipeline, add the following Pipeline script and Click Save
Create an S3 bucket and copy some files to the bucket.
Replace the s3bucket with your bucket name and filename with one of the files in your S3 bucket.
Navigate to Dashboard > download-a-file-from-s3 and Click on Build Now.

Navigate to the latest Build History link and check out the Console Output.

pipeline 
{
agent any
    stages
    {
    stage('S3download') 
        {      
            steps {
                withAWS(region:'us-east-1',credentials:'xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx')\
                {
                    s3Download(file: "filename", bucket: 's3bucket', path: '')
                }
            }
        }
    }
}

Jenkins_Pipeline1

Jenkins_Pipeline2

Jenkins_Pipeline3

Jenkins_Pipeline4

The following is the console log, which shows the pipeline has been successful. ✅

Started by user Sri
[Pipeline] Start of Pipeline
[Pipeline] node
Running on Jenkins in /var/lib/jenkins/workspace/download-a-file-from-s3
[Pipeline] {
[Pipeline] stage
[Pipeline] { (S3download)
[Pipeline] withAWS
Constructing AWS CredentialsSetting AWS region us-east-1 
[Pipeline] {
[Pipeline] s3Download
Downloading s3://s3bucket/ to file:/var/lib/jenkins/workspace/download-a-file-from-s3/receiveMessages.sh 
Finished: Downloading from s3bucket/
Download complete
[Pipeline] }
[Pipeline] // withAWS
[Pipeline] }
[Pipeline] // stage
[Pipeline] }
[Pipeline] // node
[Pipeline] End of Pipeline
Finished: SUCCESS

The file that was copied from S3 to the EC2 instance. ✅

[ec2-user@ip-172-31-89-157 ~]$ ls -lrt /var/lib/jenkins/workspace/download-a-file-from-s3/
total 0
drwxr-xr-x 2 jenkins jenkins 55 Dec 31 03:45 receiveMessages.sh

Let's run the build again, this time the build has failed with the following error ❓
The error message suggests to use set force=true.

Started by user Sri
[Pipeline] Start of Pipeline
[Pipeline] node
Running on Jenkins in /var/lib/jenkins/workspace/download-a-file-from-s3
[Pipeline] {
[Pipeline] stage
[Pipeline] { (S3download)
[Pipeline] withAWS
Constructing AWS CredentialsSetting AWS region us-east-1 
[Pipeline] {
[Pipeline] s3Download
Downloading s3://s3bucket/ to file:/var/lib/jenkins/workspace/download-a-file-from-s3/receiveMessages.sh/ 
Download failed due to existing target file; set force=true to overwrite target file
[Pipeline] }
[Pipeline] // withAWS
[Pipeline] }
[Pipeline] // stage
[Pipeline] }
[Pipeline] // node
[Pipeline] End of Pipeline
java.lang.RuntimeException: Target exists: file:/var/lib/jenkins/workspace/download-a-file-from-s3/receiveMessages.sh/
    at de.taimos.pipeline.aws.S3DownloadStep$Execution.run(S3DownloadStep.java:146)
    at de.taimos.pipeline.aws.S3DownloadStep$Execution.run(S3DownloadStep.java:113)
    at org.jenkinsci.plugins.workflow.steps.SynchronousNonBlockingStepExecution.lambda$start$0(SynchronousNonBlockingStepExecution.java:47)
    at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515)
    at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264)
    at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
    at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
    at java.base/java.lang.Thread.run(Thread.java:829)
Finished: FAILURE

So we need to add force:true as per the Pipeline: AWS Steps documentation

pipeline 
{
agent any
    stages
    {
    stage('S3download') 
        {      
            steps {
                withAWS(region:'us-east-1',credentials:'xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx')\
                {
                    s3Download(file: "filename", bucket: 's3bucket', path: '', force:true)
                }
            }
        }
    }
}

The build succeeded after adding force:true and now we can run the build multiple times.

Now let's parameterise the destination foldername.

Navigate to Dashboard > download-a-file-from-s3 and click on Configure
Select This project is parameterised and then Add String parameter
Enter Name as foldername, Default Valu as foldername and Save
Click Build with Parameters
Navigate to the latest Build History link and check out the Console Output.

pipeline 
{
agent any
    stages
    {
    stage('S3download') 
        {      
            steps {
                withAWS(region:'us-east-1',credentials:'xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx')\
                {
                    echo "${foldername}" 
                    s3Download(file: "${foldername}", bucket: 's3bucket', path: '', force:true)
                }
            }
        }
    }
}

Jenkins_Pipeline_Parameter

Jenkins_Pipeline_Build-with-Parameters

The following is the console log, which shows the pipeline has been successful. ✅

Started by user Sri
[Pipeline] Start of Pipeline
[Pipeline] node
Running on Jenkins in /var/lib/jenkins/workspace/download-a-file-from-s3
[Pipeline] {
[Pipeline] stage
[Pipeline] { (S3download)
[Pipeline] withAWS
Constructing AWS CredentialsSetting AWS region us-east-1 
[Pipeline] {
[Pipeline] echo
s3files
[Pipeline] s3Download
Downloading s3://s3bucket/ to file:/var/lib/jenkins/workspace/download-a-file-from-s3/s3files 
Finished: Downloading from sqssri/
Download complete
[Pipeline] }
[Pipeline] // withAWS
[Pipeline] }
[Pipeline] // stage
[Pipeline] }
[Pipeline] // node
[Pipeline] End of Pipeline
Finished: SUCCESS

The files were copied to s3files directory on the EC2 instance. ✅

[ec2-user@ip-172-31-89-157 ~]$ ls -lrt /var/lib/jenkins/workspace/download-a-file-from-s3/s3files 
total 8
-rw-r--r-- 1 jenkins jenkins 952 Dec 31 03:40 receiveMessages.sh
-rw-r--r-- 1 jenkins jenkins 646 Dec 31 03:40 sendMessages.sh

Clean Up

Delete the EC2 instance.
Delete the JenkinsUser.
Delete the key pair.
Delete the Security Group JenkinsSG.

Summary

We learned how to install and configure Jenkins.
We also learned about setting up the pipeline.

Referrals

Top comments (9)

Anu Sachdeva • Jan 4 '23

Thank you for the descriptive article. I have a few questions please 1) Why was the S3 bucket not made public and no bucket policy attached to let Ec2 access this s3.
2) why was no role attached to the ec2 to access the S3 bucket?
I had to follow some more additional steps to complete this setup. I had to do below coz I was getting 403 access denied error. I am not sure which one step exactly worked because I feel it is cumulation of all these steps.
1) Made my s3 bucket public 2) Attached below policy to my S3 bucket . 3) Attached an IAM role to my Jenkins ec2 so that it can access s3(not sure if this was actually needed or not but I tired everything to resolve the 403 error I was getting) . After following these steps my 403 error message went away and the job build was success
{
"Version": "2012-10-17",
"Id": "Policy1672831391552",
"Statement": [
{
"Sid": "Stmt1672830972647",
"Effect": "Allow",
"Principal": "",
"Action": "s3:",
"Resource": "arn:aws:s3:::XXXXX/"
},
{
"Sid": "Stmt1672831390540",
"Effect": "Allow",
"Principal": "",
"Action": "s3:*",
"Resource": "arn:aws:s3:::XXXXX"
}
]
}

Sri • Jan 4 '23

@anusachdeva4321 ,

you don't need S3 bucket to be public, actually you should never make your bucket public without a proper bucket policy.

We create a user with programmatic access and we use the use at Step 9 of Configure Jenkins. which will be used when you run the pipeline.

PLUS

Step 6. Create a Pipeline
Navigate to Dashboard > Manage Jenkins and copy the ID of IAM User created in Step 1, which we need to replace xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx with the ID.

There is an option to perform this with EC2 Instance role but that option has not been covered in this blog post.

hope I have answered your questions,
Sri

Anu Sachdeva • Jan 4 '23

I am trying to understand why I got the 403 , I will try this again by following this tutorial and share the error if I get it again.

John🪙🇬🇧 • Jan 2 '23

I followed the steps through and was able to get it done. Thanks for the time you spent on making this post.

Sri • Jan 2 '23

Happy that my post helped you @mlops_engineer.

Anu Sachdeva • Jan 4 '23

When you mention ID instead of xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx, is it the fully qualified ARN "arn:aws:iam::AWS_ACT:user/JenkinsUser" or is it only "JenkinsUser" ?

Sri • Jan 4 '23 • Edited

The id of the user created in step 1, which will be in the Jenkins console.
I haven’t provided the screenshot for security reasons.

Matthew Schacht • Oct 17

Hello. Thank you for this article, as it's been very helpful in setting up my own Jenkins controller in an EC2 instance.
I am having an issue getting builds to run, using the dynamically allocated cloud nodes (i.e., spinning up other EC2 instances as nodes for Jenkins to run builds). I have followed all the instructions on this guide up to step 6 (I am going to setup a different pipeline for my repo), other than attached a more permissive S3 permission to the IAM role so it can write to buckets as well, and I chose the Amazon Linux 2023 AMI because the one listed in these instructions is deprecated.
I have disabled the built-in node because I want to dynamically allocate nodes (and the instance does not have enough space in the /tmp in order to run said node anyway, and it seems difficult to change that). I setup the Cloud nodes as instructed here, but when I start a build, the build just hangs and says its waiting for an available node. Is there some step I'm missing, or perhaps something that was default when these instructions were written that must now be done manually?

987K views • Dec 16

the node gets terminated by Jenkins saying the /tmp is very low, this is becuase Jenkins marks the node as unhealthy and shuts it down., you have to go to node setting page and change the maximum available space from from 1GB to 100Kb and start the node again, by reducing this threshold value Jenkins will treat that the node is healthy and the job will get executed from next run.

DEV Community

How to set up Jenkins and a Pipeline on AWS

Table of Contents

Introduction

What is Jenkins?

What is Jenkins Pipeline?

Demo

Step 1. Create an IAM User

Step 2. Create a key pair

Step 3. Create a Security Group

Step 4. Create an EC2 instance

Step 5. Install and Configure Jenkins

Install Jenkins

Configure Jenkins

Step 6. Create a Pipeline

Now let's parameterise the destination foldername.

Clean Up

Summary

Referrals

Top comments (9)

Read next

Matt Garman keynote takeways

Day 11: Docker Compose

Full-Stack Web Application with AWS Amplify: AWS Project

Mastering CI/CD: How to Build a Robust Pipeline with GitHub Actions, Docker, and ECS