DEV Community: Justin

Jenkins - Build a Private GitHub Repo

Justin — Fri, 18 Dec 2020 19:12:46 +0000

Objective

This guide shows you how to configure an existing Jenkins server to communicate with a private GitHub repo using GitHub Deploy Keys.

My intent is to just be quick and to the point with minimal explanations.

Prerequisites

This guide assumes you have:

A GitHub account with a private GitHub repo containing some form of source code or file(s).
An AWS account containing an EC2 instance running with an installation of Jenkins. See my Jenkins - Setup a Build Environment in AWS guide if you don't have this.
Basic knowledge of Jenkins.
Basic knowledge of GitHub.

I use macOS, but this guide does cater to Linux and Windows as well.

Overview

Jenkins - Build a Private GitHub Repo

Create a Simple Jenkins Job
Generating an SSH Key on the Jenkins Server
Add the SSH Key as a Jenkins Credential
Add the SSH Key as a GitHub Deploy Key
Add the Credential to the Jenkins Job

Jenkins - Build a Private GitHub Repo

Before beginning, make sure your Jenkins EC2 instance is running, log into your Jenkins dashboard, and your GitHub account.

Create a Simple Jenkins Job

Open the Jenkins dashboard and select New Item from the menu on the left.
Give the item a name (this can be anything you want but it's representing the job itself)
Select Freestyle project from the list of options and then click OK
Once the job configuration view loads, locate the Source Code Management section and tick Git
Add in the SSH Clone URL of your desired private GitHub repo
You should see an error appear that reads something like: "Failed to connect to repository...". This is fine for now; it's because we haven't yet set up the SSH keys.
Scroll down to the Build section, click Add build step and then select *Execute shell from the list.
In the text area that appears, enter: echo "Hello, world!"
Scroll to the bottom and click Save
You'll now be brought to your job overview; click Build Now from the left menu to trigger a build
This should immediately fail, but that's intended at this point. Click the red dot next to the build number to view the console output.
Looking into the console output, you should see references relating to: "Permission denied" or "Could not read from remote repository" - this is what we are aiming to solve.
Return to the Jenkins dashboard by clicking the Dashboard breadcrumb at the top and then continue onto the next section.

Generating an SSH Key on the Jenkins Server

Open a terminal and SSH into your EC2 Jenkins server: ssh -i /path/to/<your_key_pair.pem> ec2-user@<your_public_dns> (replacing the <angles> with your info)
Sudo to the jenkins user: sudo su -s /bin/bash jenkins.

This guide assumes you are using Linux or macOS, but if you are using Windows, you can reference the AWS guide for
Connecting to your Linux instance from Windows using PuTTY

Generate an SSH key: ssh-keygen.
Accept the defaults from the prompts and leave the password blank.
Leave the terminal open and continue onto the next section.

Add the SSH Key as a Jenkins Credential

From the Jenkins dashboard select Manage Jenkins from the menu on the left.
Under the Security section, select Manage Credentials.
In the table labeled Stores scoped to Jenkins, click the Jenkins store.
In the table labeled System, click the Global credentials domain.
From the menu on the left, click Add Credentials.
In the form that appears, apply the following:
- Kind: SSH Username with private key
- Scope: Global
- ID: (blank)
- Description: (blank)
- Username: gh-deploy
- Private Key: tick Enter Directly and then click Add
- Back in your Jenkins terminal session, copy the private key to your clipboard: cat ~/.ssh/id_rsa
- Paste this key into the form making sure not to include any additional preceding/trailing spaces
- Passphrase: (blank)
- Click OK
We'll apply this credential to your new job in later steps.
Keep your terminal session open and continue onto the next section.

Add the SSH Key as a GitHub Deploy Key

Open GitHub and select the repo you specified in your job configuration
From within the repo view, select the Settings tab
From the menu on the left, select Deploy keys and then click the button on the right that says Add deploy key
Give the deploy key a Title (makes sense to match it with the name given to the jenkins credential: gh-deploy)
Back in your Jenkins terminal session, copy the public key to your clipboard: cat ~/.ssh/id_rsa.pub
Paste this key into the form making sure not to include any additional preceding/trailing spaces
Leave Allow write access unchecked
Click Add Key
The SSH key is now attached to this specific GitHub repo and is being used as a Deploy Key. For more info on Deploy Keys, checkout the GitHub documentation

Add the Credential to the Jenkins Job

Return to the Jenkins dashboard and select your new job that we created in the first section.
From the job overview; click Configure from the left menu
Scroll to the Source Code Management section and select your new credential from the Credentials select list
You should now see the errors resolve
Scroll to the bottom and click Save
You'll now be brought to your job overview; click Build Now from the left menu to trigger a build
The job should now succeed
Click the blue dot next to the build number to view the console output.
Looking into the console output, you should see the Hello, world! from the shell command we added and a status of Finished: SUCCESS - the job is now pulling from the private GitHub Repo!

Next Steps

Per usual, there are multiple ways that this connection can be configured. I've just shown one here. In a team setting, you might encounter a setup where Jenkins has its own GitHub account with SSH keys configured much like you'd do for yourself. That user can then be added as a collaborator on any given repo and will have access to it. Check out this guide if you're interested in that.

But from here, you can work on configuring your particular build for your project. In this example, I showed a Freestyle project, but I really recommend using Pipeline projects where the build specifics live with the source code in a Jenkinsfile. Check out Using a Jenkinsfile from the official Jenkins documentation to get started.

I'm continuing to work on new content showcasing my own Jenkins pipeline, so I'll make sure to add a reference here as it's completed.

Resources

Jenkins - Setup a Build Environment in AWS

Justin — Fri, 18 Dec 2020 19:08:55 +0000

Objective

This guide is simply an updated regurgitation of the AWS guide: Set Up a Jenkins Build Server. I mainly wanted to have this for my own reference, but I hope it can help someone else!

My intent is to just be quick and to the point with minimal explanations.

Prerequisites

This guide assumes you have:

Basic knowledge of AWS, an AWS account and familiarity with the AWS Console and the EC2 service.

All resources used are Free Tier eligible.

I use macOS, but this guide does cater to Linux and Windows as well.

Overview

Setting up a Jenkins Environment in AWS

Creating a Security Group
Creating an EC2 Instance
Installing Packages on your EC2 Instance
Configuring Jenkins for the First Time

Setting up a Jenkins Environment in AWS

Before beginning, log into the AWS console and verify that you are using the region that is closest to you.

Creating a Security Group

From the AWS console search for and select the EC2 service
From the left menu, select Security Groups and then click Create Security Group
Give the Security Group a name (this can be anything you want but usually reflects the underlying permissions: WebServerSecurity)
Add a description
Choose your VPC from the list (you can accept the default if you don't have a specific VPC you're targeting)
On the Inbound tab, click Add Rule and add the following rules:

Type	Protocol	Port Range	Source	Description
SSH	TCP	22	Custom (YOUR IPv4)/32	--
HTTP	TCP	80	0.0.0.0/0, ::/0	--
Custom TCP Rule	TCP	8080	0.0.0.0/0, ::/0	--

Click Create
Verify that your security group appears in the list and continue on to the next section.

Creating an EC2 Instance

From the left menu in the EC2 service, select Instances and then click Launch Instance
From the Choose an Amazon Machine Image page, select Free tier only and then select the Amazon Linux AMI 2 (HVM)
From the Choose an Instance Type page, select the t2.micro instance and click Next: Configure Instance Details
From the Configure Instance Details page, complete the items below (the rest can be left as the default):
- Network - select your VPC (same as the previous section).
- Subnet - No preference .
- Auto-assign Public IP - Enable.
- Click Review and Launch.
From the Review Instance Launch page click Edit security groups.
From the Configure Security Group page:
- Tick Select an existing security group.
- Choose the security group that you created in the previous section (WebServerSecurity).
- Click Review and Launch.
On the Review Instance Launch page, click Launch.
When prompted to Select an existing key pair or create a new key pair:
- Select Create a new key pair and then give your key a name (this can be anything you want but usually reflects the use case: JenkinsSSH).
- Click Download Key Pair and save it to a location that you can easily access.
- With the key pair saved, check the I acknowledge and then click Launch Instances.
From the Launch Status page, click View Instances to review the status of the instance you've just launched.
Once the Instance State is green and running and the Status Checks shows 2/2 checks, it is ready for use. (This may take a few minutes).
Once these statuses update, continue onto the next section.

Installing Packages on your EC2 Instance

From the Instances view in the EC2 service, select your instance and copy the Public IPv4 DNS which should be located under the Details tab (In the older UI this field is called Public DNS and is located under the Description tab).
Open a terminal and SSH into your EC2 instance: ssh -i /path/to/<your_key_pair.pem> ec2-user@<your_public_dns> (replacing the <angles> with your info)

This guide assumes you are using Linux or macOS, but if you are using Windows, you can reference the AWS guide for
Connecting to your Linux instance from Windows using PuTTY

Type yes if prompted: "Are you sure you want to continue connecting?"
Once you're logged in, enter:
- sudo yum update -y to update your instance with the latest software packages.
- sudo yum install java-1.8.0 -y
- sudo yum install git -y
- sudo wget -O /etc/yum.repos.d/jenkins.repo http://pkg.jenkins-ci.org/redhat/jenkins.repo to add the Jenkins repo
- sudo rpm --import https://pkg.jenkins.io/redhat/jenkins.io.key to import a key file from Jenkins-CI enabling the installation.
- sudo yum install jenkins -y to finally install the package
- Once the installation completes, start the Jenkins service with sudo service jenkins start.
Verify the service started with an OK and continue on to the next section.

Configuring Jenkins for the First Time

With the Jenkins service running on your EC2 instance, open a browser and navigate to http://<your_public_dns>:8080 replacing the <angles> with your info.
From the Unlock Jenkins page, enter the following in your SSH session to find the administrator password: sudo cat /var/lib/jenkins/secrets/initialAdminPassword. Copy and enter the password into the prompt and click Continue.
From the Customize Jenkins page, click Install suggested plugins
Once the installation completes, complete the form to create a secondary user that you will use in the future to access this Jenkins instance. Click Save Credentials and then click Start Using Jenkins
Verify that you can now see the Jenkins Dashboard and continue on to the next section.

Next Steps

Your Jenkins instance is now ready for you to explore and use. If you're new to Jenkins, try Creating your first Pipeline from the official Jenkins documentation.

If you're familiar with Jenkins, try Jenkins - Build a Private GitHub Repo where I show you how to configure a build to pull from a private GitHub repository.

Resources

Cypress and Angular in a Jenkins Pipeline via Docker

Justin — Wed, 25 Mar 2020 21:51:25 +0000

What's the objective?

In this post I'm going to show you how to run Cypress in a Jenkins pipeline (via Docker) to test an Angular application.

The main focus of this tutorial is not on these individual technologies. Instead, it's how to pull them all together to give you an example of how continuous integration could look for your project.

What's needed before starting?

This tutorial assumes that you have...

An Angular application (built with the CLI) with Cypress installed and at least one test written.
Docker installed & configured on your machine
Experience working with these technologies before

This example was developed using macOS

Jenkins Initial Setup, Startup and Configuration

Start by creating some volumes for your Jenkins Docker container:

# from the terminal
docker volume create jenkins-data 
docker volume create npm-cache
docker volume create cypress-cache

Next, use the docker run command to pull down the Jenkins Docker image, configure the volumes you've created and run the container serving the Jenkins app on localhost:8080:

# from the terminal
docker run -u root -d --name jenkins-tut -p 8080:8080 --ipc=host \
-v jenkins-data:/var/jenkins_home \
-v npm-cache:/root/.npm \
-v cypress-cache:/root/.cache \
-v /var/run/docker.sock:/var/run/docker.sock jenkinsci/blueocean:latest

Once that completes, open a browser and navigate to: http://localhost:8080 and wait until you see the 'Unlock Jenkins' page appear.

This page should prompt you for an administrator password. To retrieve it, peek at the Docker logs for the jenkins-tut container you have running:

# from the terminal
docker logs jenkins-tut

The password should appear in the log under the 'Jenkins initial setup is required...' statement. It's a long alphanumeric string - you can't miss it! It looks like this: 2f064d3663814887964b682940572567.

Copy and paste that puppy in and select 'Continue'!

After you make it past the 'Unlock Jenkins' page, you will be prompted to create a user and login for returning sessions. Create your desired user and continue on.

Next, you'll be prompted with the 'Customize Jenkins' page. Select 'Install Suggested Plugins' and continue on.

You should now see the Jenkins Dashboard!

Prepping Angular/Cypress for Deployment

Now hop over to your Angular project (that has Cypress installed and configured with at least one running test) and wire up the build process.

First, we need to install 2 packages that will allow Angular and Cypress to run together on the Jenkins node.

Inside the root directory of your Angular project, run:

# from the terminal
npm install angular-http-server --save-dev
npm install start-server-and-test --save-dev

angular-http-server allows the built Angular code (in /dist) to be served on the Jenkins node during the Pipeline. We want it running so that Cypress can call it.

start-server-and-test enforces a sequential start up between the Angular app and the Cypress tests. This prevents the issue of Cypress trying to call the app before it's running.

Next we'll need to add some scripts to the package.json file. These scripts will make use of the new packages and orchestrate the startup of the Angular app and the Cypress test suite.

In the package.json:

...
"scripts": {
    ...
    "cy:run": "cypress run",
    "cy:verify": "cypress verify",
    "ci:start-ng": "angular-http-server --path ./dist/YOUR_APP_NAME -p 4200",
    "ci:cy-run": "start-server-and-test ci:start-ng http://localhost:4200 cy:run",
    ...
}
...

If you're using Angular 9 you may want to add this line as well to the package.json scripts section:

    # for Angular 9 only
    "postinstall": "ngcc --properties es2015 browser module main --first-only --create-ivy-entry-points",

This works with v9's Ivy view engine and will speed up compilation time when running ng build.

Lastly, well create a Jenkinsfile at the root of the project that Jenkins will read from and define some stages for the build to go through.

Create a Jenkinsfile

pipeline {
    agent {
        docker {
            image 'cypress/base:12.16.1' 
            args '-p 3000:3000' 
        }
    }
    stages {
        stage('Install Dependencies') { 
            steps {
                sh 'npm ci'
                sh 'npm run cy:verify'
            }
        }
        stage('Build') { 
            steps {
                sh 'npm run build'
            }
        }
        stage('Test') { 
            steps {
                sh 'npm run ci:cy-run'
            }
        }
    }
}

The agent section will allow our Jenkins container to pull in a Cypress Docker image (with Node 12.16.1) for the duration of the build. The steps that follow will actually be run in that environment.

We also add in 3 steps that will install the projects dependencies, verify a successful Cypress installation, build the Angular project and then run the compiled Angular code with Cypress using the sequential start provided by the angular-http-server and start-server-and-test packages.

With all that added, save and commit your changes!

Create the Jenkins Pipeline Project

Now head back to your Jenkins instance...

From the dashboard view, click 'Create New Jobs' in the middle or 'New Item' at the top of the left menu.
Enter a name for your item.
Select 'Pipeline' as the type and click 'OK' at the bottom of the page.
Add an optional description for your pipeline.
Click the 'pipeline' tab at the top of the page to bring the 'pipeline' section into view.
In the 'Definition' field, select: 'Pipeline script from SCM'
In the 'SCM' field, select: 'Git'
In the 'Repository URL' field, enter the path to your project on your local system translated to the /home directory of the Jenkins container. For example, if your project is at /Users/YOUR_NAME/Development/YOUR_PROJECT then you would enter /home/Development/YOUR_PROJECT. Jenkins will perform some validation against your entry to let you know if it can reach the project.
Lastly, 'Save' your configuration!

Run the Build!

With your new pipeline created, it's time to run a build!

From the Jenkins dashboard view, you should see your new pipeline listed. Select the project name and then in the details view select 'Build Now'.

Once you've started the build, you should see a new entry in the 'Build History' menu to the left. If you click on the flashing dot next to the build number, you can view the log of the build.

Verify success and celebrate!

Resources

This tutorial came from a collection of resources. I encourage you to further explore these technologies and improve upon this process!

https://jenkins.io/doc/tutorials/build-a-node-js-and-react-app-with-npm/

https://www.cypress.io/blog/2019/08/02/guest-post-angular-adding-cypress-ui-tests-to-your-devops-pipeline/

https://github.com/cypress-io/cypress-example-kitchensink/blob/master/Jenkinsfile

Cover photo by JJ Ying on Unsplash