DEV Community: Bhargavi Chiluka

Jenkins Upgrade from 2.1x to 2.4x

Bhargavi Chiluka — Tue, 20 Feb 2024 06:35:14 +0000

This Article speaks about the Jenkins upgrade from 2.1X to 2.4x Due to vulnerabilities observed in the Jenkins on 24th Jan 2024.

References
For more information on CVE-2024-23897, please refer to the following sources:]

Vulnerable versions

Jenkins 2.441 and earlier, LTS 2.426.2 and earlier.

Temporary mitigation

Access to the CLI needs to be disabled. Both of the following steps must be taken:

Remove the CLI HTTP endpoint.
Disable the SSH Port

Both steps can be performed by executing the below script in script console of Jenkins UI(jenkins-->mange jenkins-->script-->console)

def removal = { lst ->
  lst.each { x -> if (x.getClass().getName()?.contains("CLIAction")) lst.remove(x) }
}
def j = jenkins.model.Jenkins.get();
removal(j.getExtensionList(hudson.cli.CLIAction.class))
removal(j.getExtensionList(hudson.ExtensionPoint.class))
removal(j.getExtensionList(hudson.model.Action.class))
removal(j.getExtensionList(hudson.model.ModelObject.class))
removal(j.getExtensionList(hudson.model.RootAction.class))
removal(j.getExtensionList(hudson.model.UnprotectedRootAction.class))
removal(j.getExtensionList(java.lang.Object.class))
removal(j.getExtensionList(org.kohsuke.stapler.StaplerProxy.class))
removal(j.actions)

println "Done!"

if (j.getPlugin('sshd')) {
  hudson.ExtensionList.lookupSingleton(org.jenkinsci.main.modules.sshd.SSHD.class).setPort(-1)
}

Permanent solution/mitigation:

Permanent mitigation can be done by upgrading the Jenkins to latest version.
As per our current Jenkins setup automatic upgrade/migration is not possible and we have to replace the source file(i.e. Jenkins.war)
What is Jenkins.War: The Jenkins Web application Archive (WAR) file bundles Winstone, a Jetty servlet container wrapper, and can be started on any operating system or platform with a version of Java supported by Jenkins

please install openjdk-17(install jdk not jre) with yum repositories before starting the upgrade which is required for 2.444 version(Latest version with fixed vulnerability at the time of writing this article

The command to install openjdk-17 is

yum install java-17-devel

and please don’t create any symbolic links if you have other version of jdk in your system instead please set the config to take the java 17 by below command.

alternative --config java

this command will prompt for the versions which are available in the system

Upgrade implementation steps on Linux:

Step 1: Stop the Jenkins service

sudo su -
service jenkins stop

Step 2: If the Jenkins is running in background, please kill the PID of the Jenkins by checking respective Jenkins port 8080

ps -ef | grep 8080
kill -9 PID

Step 3: Take backup of Jenkins home directory by zipping the file and move to temporary path. in my case the paths are given below,but it might different from system to system

cd /var/lib
tar -cvzf jenkins_date.tar.gz jenkins/
mv jenkins_date.tar.gz to /tmp path

Step 4: Take the backup of Jenkins current version binary(jenkins.war) using following commands.

cd /usr/lib/jenkins/
mv jenkins.war jenkins_old.war

Step 5: The webroot folder which is /var/cache/Jenkins has to be empty, when we are starting with new jenkins.war file.
so take the backup of war folder(mv war war_old)and empty the folder (so that it will extract new configuration in war cache folder)

mv war war_old
rm -rf war/*
chown Jenkins:Jenkins war
chmod 755 war/

Step 6: Download The New Jenkins Version and We can check the downloaded war file of SHA by

wget https://updates.jenkins-ci.org/latest/jenkins.war
sha256sum jenkins.war

Note: if the server is behind the proxy please execute the http and https proxy commands before downloading it.

step 7: Start The Jenkins Service

service jenkins start

There are several difficulties has been faced during this since it is a major version upgrade

Unable to start the Jenkins service.

it is due to incompatible init.d/jenkins file where the --daemon is not supported anymore
and comment the handlermaxCount, handlerMaxIdle (https://www.jenkins.io/doc/upgrade-guide/2.387/#de-duplicate-logging-implementations)

The Jenkins will start in the background but still the service shows as failed.

To resolve this completely please take the Jenkins command which is used to start and create Jenkins.service file like below.

service Jenkins status

Note: the starting command can be shown while checking the status, so please take the command and convert to below service file.

Create jenkis.service file in /etc/systemd/system with the following content(from the above copied command)

ExecStart command should match with the above copied command

[Unit]
Description=Jenkins Service
After=network.target

[Service]
Type=simple
User=jenkins
Group=jenkins
ExecStart=/etc/alternatives/java -Djava.awt.headless=true -DJENKINS_HOME=/var/lib/jenkins -jar /usr/lib/jenkins/jenkins.war --logfile=/var/log/jenkins/jenkins.log --webroot=/var/cache/jenkins/war --httpPort=8080
Restart=always

[Install]
WantedBy=multi-user.target

Please enable the service after creating the jenkins.service file

systemctl enable Jenkins.service

And start with the below command

systemctl start Jenkins.service

Note: please delete the init.d/Jenkins file if it is present in server , that might cause multiple instance starts
this could be different from server to server please check your server config)

var/lib/Jenkins—JENKINS_HOME
/usr/lib/Jenkins—Jenkins.war
/var/cache/Jenkins/war—webroot war extraction location
Inti.d/jenkins—manually written service file location
/etc/system/system/--- systemctl controlled services configuration locations

Elastic Search 8.x : ELK Setup over TLS/SSL

Bhargavi Chiluka — Tue, 06 Feb 2024 15:25:19 +0000

Starting from ElasticSearch V8.x, all the configurations with security runs on the self-signed certificates that are auto-generated from the installation pack itself. Hence unless there is a requirement for the private certs or public signed certs of organisation it’s recommended to use the self-signed certs for best practice.

Table of Contents:

Elastic Search
1.1. Cluster(Master Node) Installations
1.1.1.Import the Elastic search GPG Key
1.1.2.Installing from the RPM repository
1.1.3.Elastic-Search Installation output with security enabled
1.1.4.Configuration for Master Node.
1.1.5.Starting the Elasticsearch
1.1.6.Reset the password for required users.
1.2.Other (data) Node installation
1.2.1.Installation Process
1.2.2.Configuration and Connection establishment with Cluster.
1.2.3.Starting the data node and confirming the connection with cluster
1.2.4.Common Errors in connection establishment between remote systems
Kibana
2.1.Import the Elasticsearch GPG Key
2.2.Installing from the RPM repository
2.3.Configuring Kibana with Cluster Node Connection
2.4.Starting Kibana and confirming connection with Cluster Configure kibana as service using
2.5 Kibana SSL Certificates for Browser Traffic

3.Logstash
3.1.Import the Elasticsearch GPG Key
3.2.Installing from the RPM repository
3.3.Configuring Logstash with Cluster Node Connection

4.Filebeat Agent
4.1 Download Filebeat

For Kibana Web Application, the public signed certificate can be used for browser/HTTPS traffic, while self-signed certificates can be still used to establish the connection between the kibana and elastic-search cluster.

This whole document focuses on installation and configuration happening through RedHat Linux OS, which has an RPM package manager available. Installation for the Ubuntu/Debian system may slightly differ and suggest to follow the public documentation. But the configuration and connection establishment remains the same.

1. Elastic Search:

RPM Installation Ref Doc

1.1. Cluster(Master Node) Installations

1.1.1.Import the Elastic search GPG Key

rpm --import https://artifacts.elastic.co/GPG-KEY-elasticsearch

1.1.2.Installing from the RPM repository

Create a file called elasticsearch.repo in the /etc/yum.repos.d/ directory for RedHat based distributions.

[elasticsearch]
name=Elasticsearch repository for 8.x packages
baseurl=https://artifacts.elastic.co/packages/8.x/yum
gpgcheck=1
gpgkey=https://artifacts.elastic.co/GPG-KEY-elasticsearch
enabled=0
autorefresh=1
type=rpm-md

Please add the proxy=example.com if you have any proxy setup
Install package

sudo yum install --enablerepo=elasticsearch elasticsearch

If there is any error in the above installation or trouble in downloading and installing automatically, please refer to this manual installation guide from the docs.

1.1.3.Elastic-Search Installation output with security enabled

When installing Elastic-search, security features are enabled and configured by default. When you install Elastic-search, the following security configuration occurs automatically:

Authentication and authorization are enabled, and a password is generated for the elastic built-in superuser.
Certificates and keys for TLS are generated for the transport and HTTP layer, and TLS is enabled and configured with these keys and certificates.

Please save the output generated here as it includes some password and important commands.
Ex:

-------Security autoconfiguration information-------

Authentication and authorization are enabled.
TLS for the transport and HTTP layers is enabled and configured.

The generated password for the elastic built-in superuser is : <password>

If this node should join an existing cluster, you can reconfigure this with
'/usr/share/elasticsearch/bin/elasticsearch-reconfigure-node --enrollment-token <token-here>'
after creating an enrollment token on your existing cluster.

You can complete the following actions at any time:

Reset the password of the elastic built-in superuser with
'/usr/share/elasticsearch/bin/elasticsearch-reset-password -u elastic'.

Generate an enrollment token for Kibana instances with
 '/usr/share/elasticsearch/bin/elasticsearch-create-enrollment-token -s kibana'.

Generate an enrollment token for Elasticsearch nodes with
'/usr/share/elasticsearch/bin/elasticsearch-create-enrollment-token -s node'.

1.1.4.Configuration for Master Node.

Please enable the following details in the master node which is also a cluster to work parallel.

Open 9200 port for HTTP Communication. From other data/master nodes over http
Open 9300 port for Transport communication from nodes.
Update the configuration of below params in /etc/elasticsearch/elastricsearch.yml

cluster.name:<name-for-cluster>
node.name:<unique-name-for-node-in-cluster>
http.port:9200
network.host:<DNS of system>
http.host:<DNS of system>
transport.host:<DNS of System>
cluster.initial_master_nodes: ["<node name given above>"]

Here by default http.host and transport.host are commented out or either enabled with 0.0.0.0 or 127.0.0.1, please convert them to system DNS as explained in above.

In above, transport.host is optional but if any issues raised in connecting the cluster please

DO NOT modify any other parameters including security/certs. All are configured as required by default. Any modification may lead to issues.

1.1.5.Starting the Elasticsearch

Running the elasticsearch as service is a suggested approach and rpm installation, by default has halfway configuration.
Running following will tie the service to systemctl

sudo systemctl daemon-reload
sudo systemctl enable elasticsearch.service

Start / Stop the service

sudo systemctl start elasticsearch.service
sudo systemctl stop elasticsearch.service

Logs / systemctl logs

If the service failed to start, the initial logs for the service can be found in

sudo journalctl --u elasticsearch

ElasticSearch Package Logs

less /var/logs/elasticsearch/elasticsearch.log

Status Check with cURL
ElasticSearch by default runs with the security enabled and plaintext/non-secure calls are ignored by default. Requests must be hit with a certificate.
And do not hit localhost with curl as server(s) may be behind proxy and result will not be piped.

curl --cacert /etc/elasticsearch/certs/http_ca.crt -u elastic https://<DNS of System>:9200

Above command will ask for “elastic” user password. Either enter the password collected from above installation output or reset the password using the next section in the document.
Expected Sample Output from the cluster

{
  "name" : "Cp8oag6",
  "cluster_name" : <cluster name>,
  "cluster_uuid" : "AT69_T_DTp-1qgIJlatQqA",
  "version" : {
    "number" : "8.8.2",
    "build_type" : "tar",
    "build_hash" : "f27399d",
    "build_flavor" : "default",
    "build_date" : "DATE",
    "build_snapshot" : false,
    "lucene_version" : "9.6.0",
    "minimum_wire_compatibility_version" : "1.2.3",
    "minimum_index_compatibility_version" : "1.2.3"
  },
  "tagline" : "You Know, for Search"
}

1.1.6.Reset the password for required users.

There are different users required for different purposes. But majorly,

“elastic”: root user
“kibana” : for kibana ui
“logstash_system”: for logstash metrics etc.

All of these username(s) password can be set/ reset by the

/usr/share/elasticsearch/bin/elasticsearch-reset-password -i -u <username>

-i means its an interactive terminal session
-u is to define the user-name.

1.2.Other (data) Node installation:

1.2.1.Installation Process

Please follow the steps from Import the Elasticsearch GPG Key to Elasticsearch Installation output with security enabled steps.

1.2.2.Configuration and Connection establishment with Cluster.

This process involves token generation from and token configuration in the current node (which has to connect to the cluster).

Generate the node token in the elasticsearch cluster that we set previously.
Execute the following one in the cluster system terminal

Note:Make sure the elastic-search service is running in cluster while executing this

/usr/share/elasticsearch/bin/elasticsearch-create-enrollment-token -s node

Copy the token generated in the above and run the following in new node.

Note: Till the execution of the below script, DO NOT start the elastic-search service in the new node. If done, the following script will fail and it requires a fresh install of elasticsearch in the new node.

/usr/share/elasticsearch/bin/elasticsearch-reconfigure-node --enrollment-token <enrollment-token>

The above script prompts some warnings of certs dir deletion and config override. Approve the same.
If the above script failed to run with any error, check Common Errors in connection establishment between remote systems
If the above script ran successfully, please proceed in configuring the following parameters in new node, /etc/elasticsearch/elastricsearch.yml

cluster.name:<cluster name given in cluster elasticsearch.yml file>
node.name:<unique-name-for-node-in-cluster>
http.port:9200
network.host:<DNS of system>

Confirm the following things

http.host:<this parameter must be uncommented, else uncomment. But don’t edit the value>
transport.host:<this parameter must be uncommented, else uncomment. But don’t edit the value>
discovery.seed_hosts:<this value must be array with cluster DNS/IP [“”] format>

1.2.3.Starting the data node and confirming the connection with cluster

Running the elasticsearch as service is a suggested approach and rpm installation, by default has halfway configuration.
Running following will tie the service to systemctl

sudo systemctl daemon-reload
sudo systemctl enable elasticsearch.service

Start / Stop the service

sudo systemctl start elasticsearch.service
sudo systemctl stop elasticsearch.service

Logs/ systemctl logs

If the service failed to start, the initial logs for the service can be found in

sudo journalctl --u elasticsearch

ElasticSearch Package Logs

less /var/logs/elasticsearch/elasticsearch.log

Status Node Connection status Check with Cluster cURL
This command must be executed in the cluster.

curl --cacert /etc/elasticsearch/certs/http_ca.crt -u elastic https://<DNS of Cluster System>:9200/_cluster/health?pretty

Above command will ask for “elastic” user password. Either enter the password collected from above installation output or reset the password using the next section in the document. (Reset the password for required users.)
Expected Sample Output from the cluster<>node configuration is,
Please check the number_of_nodes param, it should show the count of connected nodes. (Where cluster itself is also node)
If anything is not expected please new node logs first and cluster logs next.

{
  "cluster_name" : "cluster-name",
  "status" : "yellow",
  "timed_out" : false,
  "number_of_nodes" : 2, ⇐⇐⇐⇐⇐⇐⇐⇐⇐ Pay attention here, the count must increase.
  "number_of_data_nodes" : 1,
  "active_primary_shards" : 1,
  "active_shards" : 1,
  "relocating_shards" : 0,
  "initializing_shards" : 0,
  "unassigned_shards" : 1,
  "delayed_unassigned_shards": 0,
  "number_of_pending_tasks" : 0,
  "number_of_in_flight_fetch": 0,
  "task_max_waiting_in_queue_millis": 0,
  "active_shards_percent_as_number": 50.0
}

1.2.4.Common Errors in connection establishment between remote systems

Cluster elasticseach.yml doesn’t have http.host not set to DNS or it may be commented.
9200 & 9300 of Cluster are not opened, please raise a FW
Cluster /etc/elasticsearch/certs folder is modified or security config in yml modified.
Cluster is not running as a service.

2. Kibana

Kibana RPM Docs

2.1.Import the Elasticsearch GPG Key

rpm --import https://artifacts.elastic.co/GPG-KEY-elasticsearch

2.2.Installing from the RPM repository

create a file called kibana.repo in the /etc/yum.repos.d/ directory for RedHat based distributions

[kibana-8.x]
name=Kibana repository for 8.x packages
baseurl=https://artifacts.elastic.co/packages/8.x/yum
gpgcheck=1
gpgkey=https://artifacts.elastic.co/GPG-KEY-elasticsearch
enabled=1
autorefresh=1
type=rpm-md

Please add the proxy=example.com if you have any proxy setup
Install package

sudo yum install kibana

If there is any error in the above installation or trouble in downloading and installing automatically, please refer to this manual installation guide from the docs.

2.3.Configuring Kibana with Cluster Node Connection

As explained in above details, from 8.X all elastic-search installation comes with self-signed certificates with TLS/SSL configured by default. Hence running the following scripts, before starting the Kibana would help adopt most of the settings.

Open 5601 FW for kibana UI to work on
< FW Details to be updated.>
Generate the Kibana token in the cluster system. Run the following the Cluster terminal

/usr/share/elasticsearch/bin/elasticsearch-create-enrollment-token -s kibana

Copy the above token and execute the following in the kibana system.

Note: Kibana must not be started before this command. Else we may need a fresh install.

/usr/share/kibana/bin/kibana-setup –enrollment-token <token>

Above command will configure and generate the security certs for kibana<>cluster communication with access-token (not username/password) and updates the
Token
Elasticsearch host
Certs config

In the kibana yml.

If any errors are raised in the above config, rectify the issue using (Common Errors in connection establishment between remote systems) if not solved, go for a fresh install of kibana.
If the above script ran successfully, please proceed in configuring the following parameters in new kibana, /etc/kibana/kibana.yml

server.port:5601
server.name: "kibana-host-name-can-be-anything"
server.publicBaseUrl:”DNS of kibana system with port sample.com:5601” <= This can be optional
server.host:”<kibana system DNS>”

Observe the following things, they must be auto-set with above script

elasticsearch.hosts:<this-must-be-auto-configured-with-above-script as [“cluster-ip/dns”]>
elasticsearch.serviceAccountToken:<this-must-be-auto-set-with-above-script-token>

If kibana web-application has to serve over https and needs TLS certificates setup, please update them by storing .crt and .pem files in a dir near to kiba na and update as

server.ssl.enabled: true
server.ssl.certificate: /path/to/your/server.crt ⇐⇐⇐⇐⇐ CA cert comes here
server.ssl.key: /path/to/your/server.key  ⇐⇐⇐⇐⇐ PEM file comes here

2.4.Starting Kibana and confirming connection with Cluster Configure kibana as service using

sudo systemctl daemon-reload
sudo systemctl enable kibana.service

Start/ Stop service using

sudo systemctl start kibana.service
sudo systemctl stop kibana.service

Logs/systemctl logs If the kibana failed to start, the initial logs for the service can be found in

sudo journalctl --u kibana

ElasticSearch Package Logs

less /var/logs/kibana/kibana.log

If all FW connections are in proper order and kibana is running without any errors, open https://<kibana-dns>:5601. Kibana UI should be presented. Else check the logs for connection issues.

Note: We are establishing the connection with serviceToken between elastic-search cluster and kibana (this is done using Configuring Kibana with Cluster Node Connection). Hence avoid adding username and password in kibana.yml again as it's not required.

2.5 Kibana SSL Certificates for Browser Traffic

Collect/Purchase the SSL certificates for https traffic from browser and update the following parameters in the kibana.yml

server.ssl.enabled: true
server.ssl.key: /etc/kibana/public_certs_kibana/<cert-name>.key # key file of system from SSL 
server.ssl.certificate: /etc/kibana/public_certs_kibana/<kibana-pvt-key>.pem #ca cert file

3.Logstash

Ref: Logstas RPM

3.1.Import the Elasticsearch GPG Key

rpm --import https://artifacts.elastic.co/GPG-KEY-elasticsearch

3.2.Installing from the RPM repository

Create a file called logstash.repo in the /etc/yum.repos.d/ directory for RedHat based distributions

[logstash-8.x]
name=Elastic repository for 8.x packages
baseurl=https://artifacts.elastic.co/packages/8.x/yum
gpgcheck=1
gpgkey=https://artifacts.elastic.co/GPG-KEY-elasticsearch
enabled=1
autorefresh=1
type=rpm-md

Please add the proxy=example.com if you have any proxy setup
Install package

sudo yum install logstash

If there is any error in the above installation or trouble in downloading and installing automatically, please refer to this manual installation guide from the docs.

Note: If above steps of rpm auto-installation don’t work, try downloading the required version from website manually

curl -o https://artifacts.elastic.co/downloads/logstash/logstash-8.8.2-x86_64.rpm
sudo rpm -i logstash-8.8.2-x86_64.rpm

3.3.Configuring Logstash with Cluster Node Connection

For TLS/SSL secure connection between the logstash <> cluster, we need to copy the certs folder present in the cluster /etc/elasticsearch to the logstash installation location.

Configure the following parameters

node.name: <logstash-node-name>
path.data: /var/lib/logstash
path.config: /etc/logstash/conf.d/*.conf
path.logs: /var/logs/logstash
xpack.monitoring.enabled: true
xpack.monitoring.elasticsearch.username: logstash_system
xpack.monitoring.elasticsearch.password: <password-set-in-cluster-for-that-user>
xpack.monitoring.elasticsearch.hosts: ["https://<cluster DNS>:9200"]
xpack.monitoring.elasticsearch.ssl.certificate_authority: [ "/etc/logstash/certs_cluster/http_ca.crt" ]
xpack.monitoring.elasticsearch.ssl.verification_mode: certificate
xpack.monitoring.elasticsearch.sniffing: false
xpack.monitoring.collection.interval: 10s
xpack.monitoring.collection.pipeline.details.enabled: true

Write one connection pipeline for the logstash, in conf.d dir, with logstash.conf file name

input {
 beats {
    client_inactivity_timeout => 1200
    port => 5044
  }
}

output {
        stdout { codec => rubydebug }
        elasticsearch {
        cacert => '/etc/logstash/certs_cluster/http-ca.crt'
        hosts => ["https://0.0.0.0:9200", "https://1.1.1.1:9200", "
https://2.2.2.2:9200"]
        user => user_name
        password => 'password'
        ilm_enabled => true
        ilm_rollover_alias => logstash
        }
}

Note: cacert in the above logstash.conf and certificate_authority value in the logstash.yml has to be the same.

And for security reasons, it's recommended to add the CA cert details logstash-JVM keystore so that self-signed certs are accepted. Caution: else system cannot connect to the cluster at all with any amount of efforts.
Here we can use the ca cert file we copied from the elastic search cluster
or we can directly generate one cert using ssl connection.
Generating on go and passing the key to keystore can be like this
Go to the JDK bin location which is located here, /usr/share/logstash/jdk/bin as we need the keystore

echo -n | openssl s_client -connect <elastic-search-cluster>:9200 | sed -ne '/-BEGIN CERTIFICATE-/,/-END CERTIFICATE-/p' > ./ca_logstash.cer

The above one will generate a ca_logstash.cer file with a headless tag of BEGIN/END but only with a key.
Now run the following to copy that headless key to Java Keystore directly.

Note: The default password for keystore is changeit unless we modify it after installation.

<keytool-path-as-per-system>/keytool -import -alias saelk -file ca_logstash.cer -keystore /usr/share/logstash/jdk/lib/security/cacerts

Testing the configuration
Before running the service/process, its important to confirm that logstash.yml is configured properly, hence use

sudo -u logstash /usr/share/logstash/bin/logstash --path.settings /etc/logstash -t

This should result with configuration Ok else you need fix the config till you get this.

Configuration OK
[INFO ][logstash.runner         ] Using config.test_and_exit mode. Config Validation Result: OK. Exiting Logstash

Then test the *.conf file with logstash as process so that server/service will not impact

Note: conf.d/ should have owner as logstash and user and fix any other path ownership issues based on the error/logs.

sudo -u logstash /usr/share/logstash/bin/logstash -f /etc/logstash/conf.d/<file-name>.conf

The above one will start the logstash as a process and you should see ERROR logs with connection issues. Then it's a success.
Start the logstash service and wait for a while.
You can confirm the connection by going to Kibana UI
Left Menu => Stack-Monitoring=> Nodes ( click on “set up basic config if you don't see any”)
Then you will see all elk-nodes, kibana nodes and logstash nodes if connected.

4.Filebeat Agent

To enable monitoring for any server install file beat in the respective server

4.1 Download Filebeat

Lightweight Log Analysis | Elastic
Extract the filebeat zip file
Create an user group for Filebeat using the below command
Using Root user do the following:

sudo useradd -m filebeat -p filebeat
sudo groupadd filebeat
sudo usermod -a -G filebeat filebeat

Change the Owner of extracted folder from root to filebeat user

chown filebeat:filebeat -R filebeat-7.8.1-linux-x86_64

Add the following lines in filebeat.yml file filebeat.inputs:

filebeat.inputs:
- type: log
  enabled: true
  encoding: iso8859-1
  paths:
          - /var/log/icinga2/icinga2.log
  multiline.pattern: '^[0-9]{4}-[0-9]{2}-[0-9]{2} [0-9]{2}:[0-9]{2}:[0-9]{2}'
  multiline.negate: true
  multiline.match: after
  tags: ["icinga-log"]
output.logstash:
        hosts: ["dns_name:5044"]

5.Make filebeat up

/etc/ filebeat-7.8.1-linux-x86_64/filebeat &

To send the logs from filebeat to logstash server raise a firewall request

Note: if systems are in same other network, then there is no need of firewall request adding firewall entry in IP tables is enough

Destination : IP address
Port : 5044 #default port for Logstash
Create pipeline configuration in Logstash
Go to path /etc/logstash/conf.d in logstash server
Create a file using command : vi filename.conf Add the below content according to system applications :

filter {
  if [log][file][path] =~ "/path/to/log/filename.log" {
    grok {
           match => { "message" => "%{TIMESTAMP_ISO8601:date} %{LOGLEVEL} \(%{DATA:worker_thread}\)\[%{SPACE}%{JAVACLASS:class}:%{GREEDYDATA:message_body}" }
      }
  }
}
output {
  if "grook-pattern" in [tags] {
    elasticsearch {
      cacert => '/etc/logstash/elasticsearch-ca.crt'
      hosts => ["https://0.0.0.0:9200", "https://1.1.1.1:9200", "https://2.2.2.2:9200"]
      user => user_name
      password => 'password'
      ilm_enabled => true
      ilm_pattern => "{now/d}-000001"
      ilm_rollover_alias => grook-pattern
    }
  }
}

Make sure grok pattern should match the log pattern in your server and also tags should be same both in filebeat.yml and in filename.conf files
Now you can see the logs for your server in kibana.

GitLab on premise to GitHub cloud Migration

Bhargavi Chiluka — Tue, 06 Feb 2024 13:07:28 +0000

Introduction

With latest public notice from GitLab, the on premise server instance security and maintainance would be stopped from 2024.

Once this is effective in place, we would left with our on premise Code hosting servers with no updates and on demand maintenance would be a special case to consider.

Goal

Understanding this situation, lets see how we can move our on premise Code hosting to our organization enterprise platform of GitHub.

The migration process is planned to achieve

Zero lose to commit history
No Lose of access to teams/groups
No hault or impact to our deploying strategies at present.

The migration process involving

a 160+repositories and
85+ Deployment pipelines
3 Jenkins server instances Is actually a critical and complex migration with no room for error tolerance.

Strategy

Considering the all use cases, I drafted the migration strategy as follows…

Since we are supposed to migrate from ON PREMISE to CLOUD, We cannot utilise any migration tools provided by GitLab nor GitHub…

Hence, it's a tough manual process entirely, with a lot of repeating steps, evolving to be following steps.

Clone the repo to intermediate device that could access private ok premise instance and cloud on internet.
Detach on premise server configuration with cloned repo
Create a new repo under organization GitHub Cloud account with necessary naming conventions and role based access.
Establish remote repo configuration with cloned repo to new repo.
Synchronise the code base with all branches and commit history. (A tough job)
For the deployment process not to fail, create necessary public private key pair and store in repo level deployment key store.
Generate unique SSH config and clone URL per repo per key to avoid key collision issues in ci/cd instances (Jenkins)

1. Cloning the repo

As explained above, the clone device must be a interoperable between two code hosting platforms to have a seamless migration.

The process is cloning is achieved with

git clone <ssh-based git clone URL>

2. Detach the server configuration

This is as simple as deleting a parameter from configuration after full clone

git remote remove <remote-name>

3. Create new repo in cloud with RBAC

Pertaining to organization standard practice, we should either create private or internal repo. Considering requirements of access, I opted for private repo creation.

But with huge count of repositories, it makes tough to create each repo with UI and also there is not any support from git CLI.

Hence, I adopted GitHub Cli Portable and logged in with github creds to achieve the task via terminal scripts. As follows

gh repo create --private @<org>/<repo-name> --source=<cloned-repo-path> --team=@<org>/team-handle

4. Establish the connection with local repo to new repo

The above one with —source actually created a problem of sync due to only main branch consideration. But our target of pushing all branches with all commits isn't solved.

Hence manual remote addition and push to new repo are required and incorporated with

cd <cloned-repo-path>
git remote add origin <new-repo-ssh-url>

5. Syncing all branches and commits

As explained above, clone always picks up the main/default branch and checked out.

Though it cloned all the branches locally, it cannot directly push them to new repo unless they are checked out.

Reason: When something is cloned, git downloads them as objects and stores in respective stream/tree. And won't copy locally unless you check-out. Hence direct push from local can only push the default/main branch.

With above, it's clear that we should check-out all branches to push them to new repo. But how do we know how many branches are there? Listing all branches and looping then over is a brute-force yet a bad idea.

So we force copied remotes objects inferring the local checkout as follows.

git push --mirror

6. Deploy Keys Setup

Since we have 160+ repos being used over 85+ pipelines and the servers are on premise earlier, we had SSH of one of user account added in Jenkins instances to succeed in code pull.

But with GitHub Cloud involvement, it made it impossible to use same strategy, taking access management and access life cycle policy of the org.

Which evolved to be,

Cannot use SSH key addition to existing developer accounts
Bot User Account SSH setup
Personal access token setup to either of above
Fine grained access token setup of either of above.

That left us with last option of repo level, only read allowed deploy keys setup.

As mentioned, these keys are

Repo level
Read only Access controlled
Unique to usage. (One device-one key)

Hence the deploy key addition is handled via portable Github Cli as follows

Create a password less key first

ssh-keygen -t rsa -b 4096 -N "" -f <path>/<repo-name-as-certificate-name>

The above would generate a private public key pair in any mentioned path.

Add the pub key to repo via cli

gh repo --repo <repo_name> deploy-key add <path-to-certificate>/<repo-nane-as-certificate>.pub

7. Generate unique SSH Config to avoid key collision.

With unique deploy key reach repo, git clone command cannot automatically pickup the right key for repo even if we dump all the keys in relative .ssh directory. And would lead to time out issues in cloning.

To solve this problem, we must pass key to use while cloning to git clone command.

This might be feasible for limited number of use cases but not to automation stages like in Jenkins.

To achieve the scalability, we can actually write SSH configuration which can use different key to different host(domain).

But with each repo in same GitHub, how do write different host names for different repos? Again this can be solved with host name resolution in ssh config.

As a whole, we can write config to tell ssh to use key1 to test1.com and later ask ssh to resolve test1.com to github.com with

Host test1.com
    Host github.com
    IdentityFile <path-to-key-file>
    # If ci/cd is behind a firewall, add proxy
    ProxyCommand nc --proxy-type sock5 --proxy <proxy-addres> %h %p

And git clone command to pickup above config will be like,

git clone git@test1.com:<org>/<new-repo-name>

Conclusion

The whole strategy above explained, would look like the following for one single repo, and looped over the repo titles to be migrated.

#step 1
git clone <on premise server URL>/<repo-nane>
#step 2
cd <repo-name>
git remote remove origin
cd ..
#step 3
gh repo create --private @<org>/<new-repo-name> --source=<cloned-repo-path> --team=@<org>/team-handle
#steo 4
cd <repo-name>
git add remote origin @<org>/<new-repo-name>
#step 5
git push --mirror
#step 6
ssh-keygen -t rsa -b 4096 -N "" -f <path>/<new-repo-name>
cd <repo-name>
# if we are in repo dir, gh will auto pickup correct repo from origin config
gh repo deploy-key add <path-to-certificate>/<new-repo-name>.pub

#step 7
# this config needs to stored in relative ssh config of device/user
Host github-<new-repo-name>.com
    Host github.com
    IdentityFile <path-to-key-file>/<new-repo-name-as-private-key-cert>
    # If ci/cd is behind a firewall, add proxy
    ProxyCommand socks5 ....

# updated git clone command with new URL tobe used in pipeline is
git clone git@github-<new-repo-name>.com:<org>/<new-repo-name>