100 Days of DevOps: Day 73

Automated Apache Log Collection for xFusionCorp Industries

I successfully implemented the copy-logs Jenkins job to automatically collect Apache access and error logs. This provides critical log data for immediate troubleshooting, bridging the gap until the full centralized logging system is complete.

1. Initial Access and Plugin Installation

The first step involves accessing the Jenkins UI and ensuring the necessary remote execution plugin is installed.

1. Access Jenkins: Log in using username admin and password Adm!n321.
2. Install Plugin:
   • Navigate to Manage Jenkins then Manage Plugins.
   • Go to the Available tab and search for Publish Over SSH.
   • Select the plugin and click Install without restart.
   • After installation, click Restart Jenkins (if necessary and no jobs are running).

---

2. Configure Global SSH Server Details

After the plugin is installed, the target servers must be configured in Jenkins' global settings.

1. Navigate to Manage Jenkins then Configure System.
2. Scroll down to the Publish over SSH section and click Add.

Server	Name	Hostname/IP	Username	Password
App Server 1	app-server-1	172.16.238.10	tony	Ir0nN@M
Storage Server	(Not strictly required for Jenkins connection, but credentials are needed for scp)	172.16.238.15	natasha	Bl@kW

1. Click Test Configuration to verify connectivity (for app-server-1).
2. Click Save at the bottom of the page.

---

3. Configure SSH Key Authentication (Prerequisite)

To ensure the non-interactive scp command works, SSH keys must be set up manually between the execution user (tony on stapp01) and the destination user (natasha on ststor01).

1. SSH into App Server 1 (stapp01) as user tony.

2. Generate Key Pair: If no key exists, generate one (use default file location and no passphrase for automation):
   

   ssh-keygen -t rsa
   

3. Copy Public Key: Copy the key to the Storage Server's authorized keys file. You will be prompted for natasha's password (Bl@kW) for this one-time setup:
   

   ssh-copy-id natasha@172.16.238.15
   

---

4. Create and Configure the Jenkins Job (copy-logs)

The final step is to build the Jenkins job using the established connectivity.

1. Create Job: On the Jenkins dashboard, click New Item, name it copy-logs, and select Freestyle project.

2. Schedule Build:

• Go to the Build Triggers section.
• Check Build periodically.
• Enter the schedule: */5 * * * *

3.Configure Build Step:

• Go to the Build section.
• Click Add build step then Execute shell script on remote host using SSH.
• SSH Server: Select app-server-1.

• Command: Enter the following script to execute the file transfer:

  # Define variables
  LOG_DIR="/var/log/httpd" # Confirmed Apache log path on CentOS
  DEST_USER="natasha"
  DEST_HOST="172.16.238.15"
  DEST_PATH="/usr/src/data/"
  
  # Executes scp using the previously set up SSH keys
  scp ${LOG_DIR}/access_log ${LOG_DIR}/error_log ${DEST_USER}@${DEST_HOST}:${DEST_PATH}
  

4.Save: Click Save to finalize the job configuration.

The job is now running every five minutes, securely transferring the required Apache logs and achieving a SUCCESS status.