DEV Community

Faruq2991
Faruq2991

Posted on

Setting Up Jenkins SSH Build Agents: A Complete Troubleshooting Guide

#cloud #cloudcomputing #devops #beginners

Introduction

In modern DevOps practices, distributed builds are essential for scalability and efficiency. Jenkins, one of the most popular CI/CD tools, supports this through its agent (formerly called slave) architecture. This guide walks through the complete process of configuring multiple Linux servers as SSH build agents in Jenkins, including all the troubleshooting steps for common issues you'll likely encounter.

What Problem Does This Solve?

The Challenge

When running Jenkins in a production environment, you often face these limitations:

  1. Resource Constraints: The Jenkins controller (master) server has limited CPU, memory, and disk space
  2. Build Isolation: Running all builds on the controller can cause conflicts between different projects
  3. Scalability: As your organization grows, a single server cannot handle increased build loads
  4. Platform Diversity: Different projects may require different operating systems or tools

The Solution: Distributed Build Architecture

By configuring multiple servers as Jenkins agents, you achieve:

  • Horizontal Scaling: Distribute builds across multiple machines
  • Resource Optimization: Execute builds on dedicated servers with appropriate resources
  • Build Isolation: Each agent can have specific tools and configurations
  • Parallel Execution: Run multiple builds simultaneously across different agents
  • High Availability: If one agent fails, others continue working

My Setup Requirements

For this implementation, I needed to configure three application servers as Jenkins SSH build agents:

Agent Name Server Hostname User Remote Directory Label
App_server_1 stapp01 tony /home/tony/jenkins stapp01
App_server_2 stapp02 steve /home/steve/jenkins stapp02
App_server_3 stapp03 banner /home/banner/jenkins stapp03

Prerequisites

Before starting, ensure you have:

  1. Jenkins Controller: A running Jenkins instance with admin access
  2. Agent Servers: Linux servers accessible via SSH
  3. Network Connectivity: Jenkins controller can reach agent servers on port 22
  4. User Accounts: Valid user accounts on each agent server with appropriate permissions
  5. SSH Credentials: SSH keys or passwords for authentication

Step-by-Step Implementation

Step 1: Install SSH Build Agents Plugin

The first step is ensuring Jenkins has the necessary plugin to manage SSH-based agents.

  1. Log into Jenkins with admin credentials (in my case: username admin, password Adm!n321)
  2. Navigate to Manage Jenkins → Manage Plugins
  3. Click the Available tab
  4. Search for "SSH Build Agents" plugin
  5. Select the plugin and click Install without restart
  6. Check "Restart Jenkins when installation is complete and no jobs are running"
  7. Wait for Jenkins to restart (this may take 1-2 minutes)

Important Note: If the Jenkins UI appears stuck after restart, simply refresh your browser page.

Step 2: Configure SSH Credentials

Before adding agents, you need to set up SSH credentials that Jenkins will use to connect to each server.

  1. Go to Manage Jenkins → Manage Credentials
  2. Click on (global) domain
  3. Click Add Credentials
  4. Configure as follows:
    • Kind: SSH Username with private key
    • Scope: Global
    • ID: Give it a meaningful name (e.g., tony-ssh-key)
    • Username: Enter the SSH username (e.g., tony)
    • Private Key: Select "Enter directly" and paste your private key, or select "From a file on Jenkins controller"
  5. Click OK
  6. Repeat for each user (steve and banner)

Step 3: Add the First Agent (App_server_1)

Now let's configure the first agent node:

  1. Navigate to Manage Jenkins → Manage Nodes and Clouds
  2. Click New Node
  3. Enter node name: App_server_1
  4. Select Permanent Agent
  5. Click Create

Configure the node with these settings:

  • Name: App_server_1
  • Description: App Server 1 Build Agent (optional)
  • Number of executors: 2 (adjust based on server capacity)
  • Remote root directory: /home/tony/jenkins
  • Labels: stapp01 (important for targeting builds to specific agents)
  • Usage: Use this node as much as possible
  • Launch method: Launch agents via SSH
    • Host: stapp01 (or IP address if DNS not configured)
    • Credentials: Select the credentials you created for tony
    • Host Key Verification Strategy: Non verifying Verification Strategy (for initial setup)
    • Port: 22
  • Availability: Keep this agent online as much as possible

Click Save.

Step 4: Repeat for Additional Agents

Follow the same process for App_server_2 and App_server_3:

For App_server_2:

  • Name: App_server_2
  • Remote root directory: /home/steve/jenkins
  • Labels: stapp02
  • Host: stapp02
  • Credentials: steve's credentials

For App_server_3:

  • Name: App_server_3
  • Remote root directory: /home/banner/jenkins
  • Labels: stapp03
  • Host: stapp03
  • Credentials: banner's credentials

Troubleshooting: Problems Encountered and Solutions

Problem 1: Java Not Found on Agent

Error Message:

[SSH] Starting agent process: cd "/home/tony/jenkins" && java -jar remoting.jar...
bash: line 1: java: command not found
Agent JVM has terminated. Exit code=127
Enter fullscreen mode Exit fullscreen mode

Root Cause: Java Runtime Environment (JRE) is not installed on the agent server.

Solution:

SSH into each agent server and install Java:

# SSH into the server
ssh tony@stapp01

# Switch to root user
sudo su -

# Install Java 17 (see Problem 2 for why version 17)
yum install -y java-17-openjdk java-17-openjdk-devel

# Verify installation
java -version

# Exit root
exit
Enter fullscreen mode Exit fullscreen mode

Repeat for all agent servers (stapp02 with user steve, stapp03 with user banner).

Problem 2: Java Version Mismatch (Critical Issue)

This was the most challenging issue I encountered.

Error Message:

java.lang.UnsupportedClassVersionError: hudson/slaves/SlaveComputer$SlaveVersion 
has been compiled by a more recent version of the Java Runtime 
(class file version 61.0), this version of the Java Runtime only 
recognizes class file versions up to 55.0
Enter fullscreen mode Exit fullscreen mode

Root Cause Analysis:

This error indicates a Java version compatibility issue:

  • Class file version 61.0 = Compiled with Java 17
  • Class file version 55.0 = Running with Java 11

The Jenkins controller was running on Java 17, but I initially installed Java 11 on the agent servers. Java bytecode is forward-compatible but not backward-compatible, meaning code compiled with Java 17 cannot run on Java 11.

Understanding Java Version Numbers:

  • Java 8 = class version 52.0
  • Java 11 = class version 55.0
  • Java 17 = class version 61.0
  • Java 21 = class version 65.0

Solution:

Upgrade all agent servers to Java 17:

# SSH into each agent server
ssh tony@stapp01
sudo su -

# Remove old Java 11 installation
yum remove -y java-11-openjdk java-11-openjdk-devel

# Install Java 17
yum install -y java-17-openjdk java-17-openjdk-devel

# Verify the correct version is installed
java -version
# Should output: openjdk version "17.x.x"

# Optional: Set JAVA_HOME environment variable
echo 'export JAVA_HOME=/usr/lib/jvm/java-17-openjdk' >> ~/.bashrc
source ~/.bashrc

exit
Enter fullscreen mode Exit fullscreen mode

Important Rule: Always ensure agent Java version matches or exceeds the Jenkins controller Java version.

Problem 3: Missing Jenkins Working Directory

Error Symptoms: Agent connects but fails to execute builds with permission errors.

Root Cause: The remote root directory specified in the agent configuration doesn't exist or has incorrect permissions.

Solution:

Create the directories on each agent server:

# On App Server 1
ssh tony@stapp01
mkdir -p /home/tony/jenkins
chmod 755 /home/tony/jenkins
ls -la /home/tony/jenkins

# On App Server 2
ssh steve@stapp02
mkdir -p /home/steve/jenkins
chmod 755 /home/steve/jenkins

# On App Server 3
ssh banner@stapp03
mkdir -p /home/banner/jenkins
chmod 755 /home/banner/jenkins
Enter fullscreen mode Exit fullscreen mode

Permissions Explanation:

  • 755 means: owner can read/write/execute, group and others can read/execute
  • This is necessary for Jenkins to create subdirectories and store build artifacts

Problem 4: SSH Connection Issues

Potential Errors:

  • Connection timeout
  • Authentication failed
  • Host key verification failed

Troubleshooting Steps:

  1. Test SSH connectivity manually: 
ssh tony@stapp01
# If this fails, the issue is not Jenkins-specific
Enter fullscreen mode Exit fullscreen mode
  1. Check firewall rules: 
# On the agent server
sudo firewall-cmd --list-all
# Ensure port 22 is open

# If needed, add the rule
sudo firewall-cmd --permanent --add-service=ssh
sudo firewall-cmd --reload
Enter fullscreen mode Exit fullscreen mode
  1. Verify SSH service is running: 
sudo systemctl status sshd
sudo systemctl start sshd
sudo systemctl enable sshd
Enter fullscreen mode Exit fullscreen mode
  1. Check SSH key permissions: 
# Private key on Jenkins controller should be 600
chmod 600 ~/.ssh/id_rsa

# Public key on agent server should be in authorized_keys
cat ~/.ssh/id_rsa.pub >> ~/.ssh/authorized_keys
chmod 600 ~/.ssh/authorized_keys
chmod 700 ~/.ssh
Enter fullscreen mode Exit fullscreen mode

Verification and Testing

Step 1: Check Agent Status

After configuration, verify all agents are online:

  1. Go to Manage Jenkins → Manage Nodes and Clouds
  2. All three agents should show a green checkmark indicating they're online
  3. Click on each agent to view the system information and log

Step 2: Review Agent Logs

Click on each agent name and check the logs for:

Successful Connection Log:

[SSH] Opening SSH connection to stapp01:22.
[SSH] Authentication successful.
[SSH] Starting agent process: cd "/home/tony/jenkins" && java -jar remoting.jar...
INFO: Using /home/tony/jenkins/remoting as a remoting work directory
<===[JENKINS REMOTING CAPACITY]===>channel started
Remoting version: 3.xx
Agent successfully connected and online
Enter fullscreen mode Exit fullscreen mode

Step 3: Test Build Execution

Create a simple test job to verify agents work correctly:

  1. Create a new Freestyle project
  2. In General section, check Restrict where this project can be run
  3. Enter label expression: stapp01 (to test first agent)
  4. In Build section, add Execute shell build step: 
echo "Testing on $(hostname)"
java -version
whoami
pwd
Enter fullscreen mode Exit fullscreen mode
  1. Save and click Build Now
  2. Check the console output to verify it ran on the correct agent

Repeat this test for each label (stapp02, stapp03).

Best Practices and Recommendations

Security

  1. Use SSH Keys: Prefer SSH key authentication over passwords
  2. Host Key Verification: In production, use "Known hosts file Verification Strategy" instead of non-verifying
  3. Principle of Least Privilege: Agent user accounts should have minimal necessary permissions
  4. Network Segmentation: Place Jenkins agents in a secure network zone

Performance

  1. Executor Configuration: Set number of executors based on CPU cores (typically 1-2 per core)
  2. Disk Space: Ensure adequate disk space in remote root directory (workspaces can grow large)
  3. Memory: Monitor agent memory usage and adjust Java heap size if needed using JVM Options in node configuration

Monitoring

  1. Agent Health: Regularly check agent status in Jenkins UI
  2. Log Monitoring: Review agent logs for warnings or errors
  3. Resource Usage: Monitor CPU, memory, and disk usage on agent servers
  4. Build Queue: Watch for builds waiting for available executors

Maintenance

  1. Java Updates: Keep Java versions updated on both controller and agents
  2. Jenkins Updates: Update Jenkins and plugins regularly
  3. Agent Cleanup: Periodically clean up old workspaces to free disk space
  4. Backup: Include agent configurations in Jenkins backup strategy

Using Labels Effectively

Labels are powerful for routing builds to appropriate agents:

Example Use Cases:

  1. By Application: 
pipeline {
    agent { label 'stapp01' }  // Run on specific app server
}
Enter fullscreen mode Exit fullscreen mode
  1. Multiple Labels: 
# In agent configuration, labels: "linux docker maven"
# In pipeline, agent { label 'linux && maven' }
Enter fullscreen mode Exit fullscreen mode
  1. Load Balancing: 
# Don't specify label to use any available agent
agent any
Enter fullscreen mode Exit fullscreen mode

Common Pitfalls to Avoid

  1. Version Mismatches: Always match Java versions between controller and agents
  2. Insufficient Resources: Don't overload agents with too many executors
  3. Ignoring Logs: Agent logs provide valuable troubleshooting information
  4. Hardcoding Hostnames: Use labels instead of hardcoding specific node names in jobs
  5. Poor Credential Management: Rotate SSH keys regularly and use Jenkins credential store

Automated Setup Script

For future deployments, here's a script to automate agent preparation:


bash
#!/bin/bash
# prepare_jenkins_agent.sh
# Run this script on each agent server

set -e

JENKINS_USER=$1
JENKINS_DIR="/home/${JENKINS_USER}/jenkins"

echo "=== Preparing Jenkins Agent for user: ${JENKINS_USER} ==="

# Install Java 17
echo "Installing Java 17..."
sudo yum remove -y java-11-openjdk java-11-openjdk-devel 2>/dev/null || true
sudo yum install -y java-17-openjdk java-17-openjdk-devel

# Verify Java installation
echo "Verifying Java installation..."
java -version

# Create Jenkins directory
echo "Creating Jenkins working directory..."
mkdir -p "${JENKINS_DIR}"
chmod 755 "${JENKINS_DIR}"

# Set JAVA_HOME
echo "Setting JAVA_HOME..."
echo 'export JAVA_HOME=/usr/lib/jvm/java-17-openjdk'
Enter fullscreen mode Exit fullscreen mode

Top comments (0)