DEV Community

Janak Shrestha
Janak Shrestha

Posted on

Day 73: Jenkins Scheduled Jobs

#devops #jenkins

The devops team of xFusionCorp Industries is working on to setup centralised logging management system to maintain and analyse server logs easily. Since it will take some time to implement, they wanted to gather some server logs on a regular basis. At least one of the app servers is having issues with the Apache server. The team needs Apache logs so that they can identify and troubleshoot the issues easily if they arise. So they decided to create a Jenkins job to collect logs from the server. Please create/configure a Jenkins job as per details mentioned below:

Click on the Jenkins button on the top bar to access the Jenkins UI. Login using username admin and password Adm!n321

  1. Create a Jenkins jobs named copy-logs.
  2. Configure it to periodically build every 8 minutes to copy the Apache logs (both access_log and error_log) from App Server 1 (stapp01) from the default logs location to location /usr/src/finance on the Storage Server.
  3. Build the job at least once so that the logs are copied and can be verified.

Note:

  1. You might need to install some plugins and restart Jenkins. We recommend selecting Restart Jenkins when installation is complete and no jobs are running in the update centre. Refresh the page if the UI gets stuck after a restart.
  2. Define the cron expression as required (e.g. */10 * * * * to run every 10 minutes).
  3. For scenarios that require web UI changes, take screenshots or record your work (e.g. using loom.com) so you can share it for review if the task is marked incomplete.

Understanding the Architecture

Components Overview

Component Specification
Job Name copy-logs
Schedule Every 8 minutes (*/8 * * * *)
Source Apache logs from App Server 1 (stapp01)
Logs access_log and error_log
Destination /usr/src/finance on Storage Server (ststor01)

Architecture Diagram 

┌─────────────────────────────────────────────────────────────────────────────┐
│                         Jenkins Server                                     │
│                                                                              │
│  ┌────────────────────────────────────────────────────────────────────────┐ │
│  │  Jenkins Job: copy-logs                                               │ │
│  │  Schedule: */8 * * * * (Every 8 minutes)                             │ │
│  │                                                                       │ │
│  │  Build Steps:                                                         │ │
│  │  1. SSH to App Server 1 → Copy access_log and error_log              │ │
│  │  2. SCP to Storage Server → /usr/src/finance                         │ │
│  └────────────────────────────────────────────────────────────────────────┘ │
│                                    │                                         │
│                                    ▼                                         │
│  ┌────────────────────────────────────────────────────────────────────────┐ │
│  │  App Server 1 (stapp01)                                               │ │
│  │  ┌──────────────────────────────────────────────────────────────────┐ │ │
│  │  │  Apache Logs: /var/log/httpd/                                    │ │ │
│  │  │  - access_log                                                    │ │ │
│  │  │  - error_log                                                     │ │ │
│  │  └──────────────────────────────────────────────────────────────────┘ │ │
│  └────────────────────────────────────────────────────────────────────────┘ │
│                                    │                                         │
│                                    ▼                                         │
│  ┌────────────────────────────────────────────────────────────────────────┐ │
│  │  Storage Server (ststor01)                                            │ │
│  │  ┌──────────────────────────────────────────────────────────────────┐ │ │
│  │  │  Destination: /usr/src/finance/                                  │ │ │
│  │  │  - access_log                                                    │ │ │
│  │  │  - error_log                                                     │ │ │
│  │  └──────────────────────────────────────────────────────────────────┘ │ │
│  └────────────────────────────────────────────────────────────────────────┘ │
└─────────────────────────────────────────────────────────────────────────────┘
Enter fullscreen mode Exit fullscreen mode

Step-by-Step Implementation

Step 1: Access Jenkins UI

  1. Click the Jenkins button on the top bar
  2. Login with:
    • Username: admin
    • Password: Adm!n321

Step 2: Install Required Plugins (If Needed)

  1. Navigate to Manage Jenkins → Plugins
  2. Click on Available plugins
  3. Search for and install:
    • Publish Over SSH
    • SSH plugin
  4. Restart Jenkins if prompted

Note: When installing plugins, select Restart Jenkins when installation is complete and no jobs are running.

Step 3: Configure SSH Server for Storage Server

  1. Go to Manage Jenkins → Configure System
  2. Scroll to Publish over SSH section
  3. Click Add next to SSH Servers
  4. Configure the following:
    • Name: storage-server
    • Hostname: ststor01.stratos.xfusioncorp.com
    • Username: natasha
    • Remote Directory: /usr/src/finance
  5. Under Advanced, check Use password authentication
  6. Enter the password: Bl@kW
  7. Click Test Configuration to verify the connection
  8. Click Save

Step 4: SSH to Jenkins Server (For Key-Based Configuration) 

ssh jenkins@jenkins
# Password: j@rv!s
Enter fullscreen mode Exit fullscreen mode

Step 5: Generate SSH Key for Jenkins User 

# Check if SSH key exists
ls -la ~/.ssh/id_rsa

# If not, generate it
ssh-keygen -t rsa -b 4096 -N "" -f ~/.ssh/id_rsa
Enter fullscreen mode Exit fullscreen mode

Output:

Generating public/private rsa key pair.
Your identification has been saved in /var/lib/jenkins/.ssh/id_rsa
Your public key has been saved in /var/lib/jenkins/.ssh/id_rsa.pub
Enter fullscreen mode Exit fullscreen mode

Step 6: Copy SSH Key to App Server 1 

# Copy public key to stapp01
ssh-copy-id -o StrictHostKeyChecking=no tony@stapp01
# Password: Ir0nM@n

# Verify passwordless access
ssh -o StrictHostKeyChecking=no tony@stapp01 hostname
Enter fullscreen mode Exit fullscreen mode

Output:

stapp01
Enter fullscreen mode Exit fullscreen mode

Step 7: Copy SSH Key to Storage Server 

# Copy public key to ststor01
ssh-copy-id -o StrictHostKeyChecking=no natasha@ststor01
# Password: Bl@kW

# Verify passwordless access
ssh -o StrictHostKeyChecking=no natasha@ststor01 hostname
Enter fullscreen mode Exit fullscreen mode

Output:

ststor01
Enter fullscreen mode Exit fullscreen mode

Step 8: Create Destination Directory on Storage Server 

# Create and set permissions for destination directory
ssh -o StrictHostKeyChecking=no natasha@ststor01 "sudo mkdir -p /usr/src/finance && sudo chown natasha:natasha /usr/src/finance"
Enter fullscreen mode Exit fullscreen mode

Step 9: Verify Apache Logs Location on App Server 1 

# Check Apache log location
ssh -o StrictHostKeyChecking=no tony@stapp01 "ls -la /var/log/httpd/"
Enter fullscreen mode Exit fullscreen mode

Output:

-rw-r--r-- 1 root root access_log
-rw-r--r-- 1 root root error_log
Enter fullscreen mode Exit fullscreen mode

Step 10: Create the Jenkins Job

  1. From Jenkins dashboard, click New Item
  2. Enter name: copy-logs
  3. Select Freestyle project
  4. Click OK

Step 11: Configure Build Trigger

  1. Under Build Triggers, check Build periodically
  2. Enter cron expression: 
   */8 * * * *
Enter fullscreen mode Exit fullscreen mode

This runs the job every 8 minutes.

Step 12: Configure Build Step

  1. Under Build, click Add build step → Execute shell
  2. Enter the following script: 
#!/bin/bash

# Copy Apache logs from App Server 1 to Storage Server
scp -o StrictHostKeyChecking=no tony@stapp01:/var/log/httpd/access_log natasha@ststor01:/usr/src/finance/
scp -o StrictHostKeyChecking=no tony@stapp01:/var/log/httpd/error_log natasha@ststor01:/usr/src/finance/
Enter fullscreen mode Exit fullscreen mode

Step 13: Save and Build

  1. Click Save
  2. Click Build Now
  3. Check Console Output

Successful Console Output:

Started by user admin
Running as SYSTEM
Building in workspace /var/lib/jenkins/workspace/copy-logs
[copy-logs] $ /bin/bash /tmp/jenkins1234567890.sh
+ scp -o StrictHostKeyChecking=no tony@stapp01:/var/log/httpd/access_log natasha@ststor01:/usr/src/finance/
+ scp -o StrictHostKeyChecking=no tony@stapp01:/var/log/httpd/error_log natasha@ststor01:/usr/src/finance/
Finished: SUCCESS
Enter fullscreen mode Exit fullscreen mode

Verification

Check Copied Logs on Storage Server 

ssh natasha@ststor01 "ls -la /usr/src/finance/"
Enter fullscreen mode Exit fullscreen mode

Expected Output:

-rw-r--r-- 1 natasha natasha ... access_log
-rw-r--r-- 1 natasha natasha ... error_log
Enter fullscreen mode Exit fullscreen mode

Check Log Content 

ssh natasha@ststor01 "head -20 /usr/src/finance/access_log"
ssh natasha@ststor01 "head -20 /usr/src/finance/error_log"
Enter fullscreen mode Exit fullscreen mode

Understanding the Script

SCP Command 

scp -o StrictHostKeyChecking=no tony@stapp01:/var/log/httpd/access_log natasha@ststor01:/usr/src/finance/
Enter fullscreen mode Exit fullscreen mode
Part Purpose
scp Secure Copy Protocol
-o StrictHostKeyChecking=no Skip host key verification
tony@stapp01:/var/log/httpd/access_log Source file
natasha@ststor01:/usr/src/finance/ Destination directory

Log Files

File Purpose
access_log Records all HTTP requests to the server
error_log Records server errors and warnings

Cron Expression Reference

Expression Meaning
*/8 * * * * Every 8 minutes
*/10 * * * * Every 10 minutes
*/5 * * * * Every 5 minutes
0 * * * * Every hour at minute 0
0 0 * * * Every day at midnight

Important: Use */8 not H/8 for exact 8-minute intervals.

Troubleshooting

Issue 1: SSH Connection Failed

Error: Permission denied (publickey)

Solution: Ensure SSH keys are configured correctly:

# Test SSH connectivity
ssh -o StrictHostKeyChecking=no tony@stapp01 hostname
ssh -o StrictHostKeyChecking=no natasha@ststor01 hostname

# If fails, re-copy the key
ssh-copy-id -o StrictHostKeyChecking=no tony@stapp01
Enter fullscreen mode Exit fullscreen mode

Issue 2: Permission Denied on Destination

Error: Permission denied

Solution: Ensure the destination directory exists and is writable:

ssh natasha@ststor01 "sudo mkdir -p /usr/src/finance && sudo chown natasha:natasha /usr/src/finance"
Enter fullscreen mode Exit fullscreen mode

Issue 3: Apache Logs Not Found

Error: No such file or directory

Solution: Verify the Apache log location:

ssh tony@stapp01 "ls -la /var/log/httpd/"
Enter fullscreen mode Exit fullscreen mode

If logs are in a different location (e.g., /var/log/apache2/), adjust the paths accordingly.

Issue 4: Jenkins UI Gets Stuck

Solution: Refresh the UI page. If Jenkins service restarts, it may take a moment to come back online.

Best Practices

1. Log Rotation

Implement log rotation to prevent logs from consuming too much disk space:

# Configure logrotate for Apache logs
sudo vi /etc/logrotate.d/httpd
Enter fullscreen mode Exit fullscreen mode

2. Compression

Compress large log files to save storage space:

gzip /usr/src/finance/access_log
Enter fullscreen mode Exit fullscreen mode

3. Retention Policy

Implement a retention policy to delete old logs:

find /usr/src/finance -name "*.log" -mtime +30 -delete
Enter fullscreen mode Exit fullscreen mode

4. Monitoring

Set up notifications for job failures:

  • Configure email notifications in Jenkins
  • Use monitoring tools to track log collection status

5. Security

Ensure log files are stored securely with appropriate permissions:

chmod 640 /usr/src/finance/*.log
Enter fullscreen mode Exit fullscreen mode

Useful Commands Reference

Command Purpose
ssh-keygen -t rsa -b 4096 -N "" -f ~/.ssh/id_rsa Generate SSH key
ssh-copy-id -o StrictHostKeyChecking=no user@host Copy SSH key to remote host
ssh -o StrictHostKeyChecking=no user@host hostname Verify passwordless SSH
scp -o StrictHostKeyChecking=no source destination Copy files between hosts
ssh user@host "mkdir -p /path" Create directory on remote host
find /path -name "*.log" -mtime +30 -delete Delete old logs
gzip /path/to/logfile Compress log file

Summary

In this challenge, we successfully:

  1. Accessed the Jenkins UI and logged in as admin
  2. Installed the Publish Over SSH plugin
  3. Configured SSH server for the Storage Server
  4. Generated SSH keys on the Jenkins server
  5. Copied SSH keys to App Server 1 and Storage Server
  6. Created the destination directory on the Storage Server
  7. Created a Jenkins job named copy-logs
  8. Configured the job to copy Apache logs from App Server 1
  9. Set up the job to run every 8 minutes
  10. Verified the logs were copied successfully

This automated log collection solution ensures that Apache logs are regularly collected and stored on the Storage Server. The DevOps team can then analyze these logs to troubleshoot issues, monitor traffic, and identify security threats.

Top comments (0)