DEV Community: damilola lawal

A Multi-Tier web Application Stack using; Nginx, Tomcat, Memcached, RabbitMQ on AWS EC2 instance

damilola lawal — Mon, 15 Jul 2024 15:02:38 +0000

Welcome, this project will show how to create a simple multi-tier web application application hosted ec2-instance.

These stack consist of the web layer, application layer and database layer.
Each of these services have its functionality which include;

Nginx: Nginx is a high-performance web server and reverse proxy server that efficiently handles HTTP and other web traffic, known for its stability, rich feature set, and low resource consumption.

Tomcat:Apache Tomcat is an open-source Java Servlet container. It is a web server and servlet container that serves as a platform for running Java applications.

MySQL: MySQL is a widely-used open-source relational database management system (RDBMS) that allows for the efficient storage, retrieval, and management of data using SQL (Structured Query Language).

Memcached: Mem-cached is a distributed memory caching system designed to speed up dynamic web applications by reducing database load through caching data and objects in RAM.

RabbitMQ: RabbitMQ is a robust, open-source message broker that facilitates the efficient handling of message queuing for distributed systems and microservices, supporting multiple messaging protocols.

In web development,

STACKS are collection of services working together to create an experience

Example MERN, LAMP etc.

The image below shows the workflow of these project.

User enters the URL of the website (ip-address will be used for this project).
The request goes to Nginx (act as a load balancer which route request to the Apache Tomcat service- a JAVA web application service).
The web application is hosted on the tomcat server.
The user information will be logged into the MYSQL database.
RabbitMQ serves as a broker/queuing agent, enabling seamless communication and message queuing between distributed systemsRead more.
Memcached service is a database caching agent connected to MySQL and previously accessed information from the database is cached to it reducing database hit and increase performance.

PREREQUISITE

Two EC2 instance ( one centos and one ubuntu).
Gitbash or an IDE.

Create two medium EC2 instance

The centos which will host the Tomcat service, memcache, Mysql and RabbitMQ.
The ubuntu will be hosting Nginx.

MySQL (Database SVC)
Memcache (DB Caching SVC)
RabbitMQ (Broker/Queue SVC)
Tomcat (Application SVC)
Nginx (Web SVC)

We are going to start the configuration from the database to the web layer as its custom to do so.

Setting up MYSQL service:
SSH into the Centos EC2 instance & update OS

sudo yum update -y

Set Repository

sudo yum install epel-release -y

Install MariaDB package

yum install git mariadb-server -y

start MariaDB service and enable to start automatically at boot time.

 systemctl start mariadb
 systemctl enable mariadb

Run the **mysql secure installation **script, to enable us secure our database.

mysql_secure_installation

Also, I will be using admin123 as root password but you can use any password but ensure its strong ( containing alphabet, numeric and characters).

Next, set the database name, create user and grant all remote privilege.

 mysql -u root -padmin123

where user is root and password is admin123

mysql> create database accounts;
mysql> grant all privileges on accounts.* TO 'admin'@'%' identified by 'admin123';
mysql> FLUSH PRIVILEGES;
mysql> exit;

In the above code, we grant all privilege ( SELECT, DELETE, INSERT, UPDATE e.t.c ) on database name "accounts" to a user "admin" who can connect from any host ( % ).

Next, we will clone the Repo that has the source code for our application and initialize database.

git clone -b main https://github.com/lasdapharm/vprofile-project.git
cd vprofile-project
mysql -u root -padmin123 accounts < src/main/resources/db_backup.sql
mysql -u root -padmin123 accounts

db_backup.sql script create all necessary schema and table on the database.

mysql> show tables;
mysql> exit;

Restart mariadb,

systemctl restart mariadb

Check the mariadb status

systemctl status mariadb

SET UP MEMCACHE SERVICE

Install, start & enable memcache

sudo dnf install epel-release -y
sudo dnf install memcached -y
sudo systemctl start memcached
sudo systemctl enable memcached
sudo systemctl status memcached
sudo systemctl restart memcached

SET UP RABBITMQ

update OS to latest patches.

 yum update -y

Install dependencies,

sudo yum install wget -y
cd /tmp/
dnf -y install centos-release-rabbitmq-38
dnf --enablerepo=centos-rabbitmq-38 -y install rabbitmq-server
systemctl enable --now rabbitmq-server

Setup access to user "test" and make it "admin"

sudo sh -c 'echo "[{rabbit, [{loopback_users, []}]}]." >  /etc/rabbitmq/rabbitmq.config'
sudo rabbitmqctl add_user test test
sudo rabbitmqctl set_user_tags test administrator

sh -c '...': Invokes the shell to execute the command within the single quotes.

[{rabbit, [{loopback_users, []}]}]. is the content being written to the RabbitMQ configuration file. This configuration tells RabbitMQ not to restrict the guest user to only loopback (localhost) connections.

By default, RabbitMQ restricts the guest user to only connect from localhost (loopback).
Setting loopback_users to an empty list ([]) removes this restriction, allowing the guest user to connect from any IP address.

Restart RabbitMQ

sudo systemctl restart rabbitmq-server

*SET UP TOMCAT SERVICE
*
When we install other services, we use the package managers (yum and dnf) and this way, we automatically get the systemctl command to manage the service ( start, stop, enable service ) and place the configuration, binaries e.t.c in the rigt directory but for tomcat we are going download the package and configure the systemctl ourselves.

Tomcat service need JDK (openjdk11 : to develop our application) as dependencies so we going install that as well.

dnf -y install java-11-openjdk java-11-openjdk-devel

Install maven ( to build our application from source code) and also install git

 dnf install git maven wget -y

Change directory to /tmp

cd /tmp/

Download and install tomcat package

wget https://archive.apache.org/dist/tomcat/tomcat-9/v9.0.75/bin/apache-tomcat-9.0.75.tar.gz

Extract package

tar xzvf apache-tomcat-9.0.75.tar.gz

Add tomcat user to run tomcat services and add user to a home directory.

useradd --home-dir /usr/local/tomcat --shell /sbin/nologin tomcat

The tomcat user created will be used by tomcat process so we don't need login (--shell /sbin/nologin).

Copy all file in apache-tomcat-9.0.75 to /usr/local/tomcat/

cp -r /tmp/apache-tomcat-9.0.75/* /usr/local/tomcat/

When we do this, the copy command will copy the folder and file as root user and all files and folder will automatically be owned by root.
We need to change the ownership of the files and folder to tomcat user and tomcat group.
This changes is required for systemctl otherwise using systemctl to start, stop or enable tomcat service will fail.

chown -R tomcat.tomcat /usr/local/tomcat

Setup systemctl command for tomcat

Create tomcat service file

vi /etc/systemd/system/tomcat.service

Update the tomcat.srvice with this configuration

[Unit]
Description=Tomcat
After=network.target
[Service]
User=tomcat
WorkingDirectory=/usr/local/tomcat
Environment=JRE_HOME=/usr/lib/jvm/jre
Environment=JAVA_HOME=/usr/lib/jvm/jre
Environment=CATALINA_HOME=/usr/local/tomcat
Environment=CATALINE_BASE=/usr/local/tomcat
ExecStart=/usr/local/tomcat/bin/catalina.sh run
ExecStop=/usr/local/tomcat/bin/shutdown.sh
SyslogIdentifier=tomcat-%i
[Install]
WantedBy=multi-user.target

When we say systemctl start, "ExecStart=/usr/local/tomcat/bin/catalina.sh run" _get executed and for systemctl stop _"ExecStop=/usr/local/tomcat/bin/shutdown.sh" get executed.

Reload the configuration

systemctl daemon-reload

Start and Enable service

systemctl start tomcat
systemctl enable tomcat

Build code and Deploy

Update configuration,

cd vprofile-project

vim src/main/resources/application.properties

Update file with backend server details.

#JDBC Configutation for Database Connection
jdbc.driverClassName=com.mysql.jdbc.Driver
jdbc.url=jdbc:mysql://<pubilc_IP_address>:3306/accounts?useUnicode=true&characterEncoding=UTF-8&zeroDateTimeBehavior=convertToNull
jdbc.username=admin # database username
jdbc.password=admin123 # databse password

#Memcached Configuration For Active and StandBy Host
#For Active Host
memcached.active.host=<pubilc_IP_address>
memcached.active.port=11211
#For StandBy Host
memcached.standBy.host=127.0.0.2
memcached.standBy.port=11211

#RabbitMq Configuration
rabbitmq.address=<pubilc_IP_address>
rabbitmq.port=5672
rabbitmq.username=test
rabbitmq.password=test

Build code
Run below command inside the repository
Make sure you are in the vprofile directory & run the mvn install

mvn install

The command will read the source code and build the artifact ( A deployable product )

Artifact is basically an archive ( in java, archive format is var, jar , war )

Deploy artifact, first stop the tomcat service

systemctl stop tomcat

rm -rf /usr/local/tomcat/webapps/ROOT*
cp target/vprofile-v2.war /usr/local/tomcat/webapps/ROOT.war
systemctl start tomcat
chown tomcat.tomcat /usr/local/tomcat/webapps -R
systemctl restart tomcat

what we did was to remove the default app in java (ROOT) and replace it with our artifact (target/vprofile-v2.war )
We restart tomcat and when tomcat find a war file its will extract it to ROOT.

Finally,
SET UP NGINX

For our nginx which route request to our tomcat service.

We are going to host it on our Ubuntu EC2 instance,

So SSH into the instance,

apt update

Install nginx

apt install nginx -y

Create Nginx conf file

vi /etc/nginx/sites-available/vproapp

upstream vproapp {
    server <public_IP_of_Centos_instance>:8080;
}

server {
    listen 80;
    location / {
        proxy_pass http://vproapp;
    }
}

Remove default nginx conf

 rm -rf /etc/nginx/sites-enabled/default

Create link to activate website and make our website default

ln -s /etc/nginx/sites-available/vproapp /etc/nginx/sites-enabled/vproapp

Restart Nginx

 systemctl restart nginx

Security.

Allow inbound rule for our EC2 instance
For the Centos, allow inbound traffic on port 3306 (Mysql), 11211 (memcached), 5672 ( RabbitMQ) from Ubuntu.
Allow inbound traffic from 0.0.0.0 on port 80 on the Ubuntu instance since its public-facing.

Validation

Access the website from web browser by :80

Click on RabbitMQ

Click on All_user and this will fetch all user on databse

Thank you for reading...

Automated User Management Script for Linux Systems

damilola lawal — Wed, 03 Jul 2024 20:46:32 +0000

Nowadays managing user accounts properly is extremely important for SysOps professionals working in dynamic IT setups.
The pro of user provisioning is that it saves time, ensures consistency , security across systems and improve efficiency since much of the process can be automated.

In this article, we will discuss about bash script and how to create a bash script that can be used for creating and managing accounts of users on Linux servers.

In this article we will be writing a script which that reads a text file containing the employee’s usernames and group names, where each line is formatted as "user;groups".

This script will create users and groups as specified, set up home directories with appropriate permissions and ownership, generate random passwords for the users, and log all actions to var/log/user_management.log

Additionally, it will store the generated passwords securely in /var/secure/user_passwords.txt.
we will ensure error handling for scenarios like existing users by the script.

Each User will have a personal group with the same group name as the username, this group name will not be written in the text file
A user can have multiple groups, each group delimited by comma ","
Usernames and user groups are separated by semicolon ";"
For example;

light; sudo,dev,www-data
idimma; sudo
mayowa; dev,www-data

Where light is username and groups are sudo, dev, www-data
Firstly,

#!/bin/bash
# Check if script is run as root
if [ "$EUID" -ne 0 ]; then
  echo "Please run as root"
  exit 1
fi

The #!/bin/bash line at the beginning of a script is known as the shebang. This shebang line tells the operating system which interpreter to use to execute the script. In this case, it specifies the Bash shell as the interpreter.

The line if [ "$EUID" -ne 0 ]; checks if the script is being run as root. The EUID (Effective User ID) is used to determine the privileges of the executing process. The -ne operator means "not equal." Since the root user has an ID of 0, this condition checks if the current user is not root. If true, it prompts to run the script as root and exits.

Next;

# Check if the input file is provided
if [ -z "$1" ]; then
  echo "Usage: bash create_users.sh <name-of-text-file>"
  exit 1
fi

The script checks if a positional argument is provided. If not, it exits with a status of 1 and displays the message: "Usage: bash create_user.sh ," prompting the user to provide the correct argument.

The -z is a string value which means value has a length of zero and "$1" indicate the first parameter after the script name($0) which is assigned a variable name INPUT_FILE. So it checks if the first positional parameter is empty.

Next;

INPUT_FILE="$1"
LOG_FILE="/var/log/user_management.log"
PASSWORD_FILE="/var/secure/user_passwords.txt"

# Ensure log and password files exist and have the correct permissions
touch $LOG_FILE
chmod 644 $LOG_FILE

mkdir -p /var/secure
touch $PASSWORD_FILE
chmod 600 $PASSWORD_FILE

We will create the directory /var/secure and the file user_password.txt in it to securely store user passwords with permissions set to "600" (read and write only for the owner). Additionally, we will create the file management.log in the /var/log directory to log all events.

The first positional parameter($1) is assigned INPUT_FILE as variable

Next;

# Function to generate random password
generate_password() {
  openssl rand -base64 12
}

The OpenSSL rand command will be used to generate a cryptographic password that is base64 encoded (strong and resistant as it contains letters, numbers, and characters). This function will generate the password, which will be called in the script.

Next;

# Process the input file
while IFS=';' read -r username groups; do
  # Create personal group for the user
  if ! getent group "$username" &>/dev/null; then
    groupadd "$username"
    echo "Created group $username" | tee -a $LOG_FILE
  fi

Let's process the input file;

IFS means Internal Field Separator; this is an environment variable which define a list of characters the Bash shell uses as field separators. These include space, tab, newline.

Use the while condition to read the username and group taking note of the separators by the IFS.

We will create a personal group for the user but before then we will check the system group database (/etc/group) using the getent command to ensure the group is not present.

The negate operator (!) is used to reverse the outcome of an expression. In this case, if the $group does not exist, the expression with ! will evaluate to true. This condition triggers the groupadd command to create the group if it doesn't already exist.

# Check if user already exists
  if id "$username" &>/dev/null; then
    echo "User $username already exists. Skipping..." | tee -a $LOG_FILE
    continue
  fi

If user already exist then our shell script will throw an error message indicating that user exist but we want to redirect such output( standard error or output) to the /dev/null and the tee reads the output and save it silently in the $LOG_FILE.

We then use the continue to create user if user does not exist.

# Create user and personal group
  useradd -m -s /bin/bash -g "$username" "$username"
  if [ $? -eq 0 ]; then
    echo "Created user $username with a personal group $username" | tee -a $LOG_FILE
  else
    echo "Failed to create user $username" | tee -a $LOG_FILE
    continue
  fi

Let's add create user and add to the personal group using the popular command useradd and use the -m(to add a home directory for the user, -s (to set the shell to /bin/bash).

If the user is created and added to the personal group successfully this make the exit status ($?) equal zero (0) and it print the success message "Created user $username with a personal group $username" to the $LOG_FILE and if not it execute the else statement.

Next;

 # Generate password and set it for the user
  password=$(generate_password)
  echo "$username:$password" | chpasswd
  if [ $? -eq 0 ]; then
    echo "$username,$password" >> $PASSWORD_FILE
    echo "Set password for $username" | tee -a $LOG_FILE
  else
    echo "Failed to set password for $username" | tee -a $LOG_FILE
  fi

Let's call the function to generate the password and set password for the user using chpasswd (change password) , write the username,password into the $PASSWORD_FILE and log it.

chpasswd is designed to process multiple username and password pairs from standard input (usually via a pipe | or a redirection <)
.

Next,

# Create additional groups if specified
  if [ -n "$groups" ]; then
    IFS=',' read -ra group_array <<< "$groups"
    for group in "${group_array[@]}"; do
      # Create group if it doesn't exist
      if ! getent group "$group" &>/dev/null; then
        groupadd "$group"
        echo "Created group $group" | tee -a $LOG_FILE
      fi
  fi

we will be creating other group which user will be by passing the $groups as an array and iterate over it by "${group_array[@]}" where "@" insinuate every character.

The character -n is also similar to the -z only that it describe value which length is greater than zero

We will check the system group database (/etc/group) using the getent command but will add a negate operator (!) which will negate the expression and makes it True, if the $group does not exist thereby causing it to create the group if it doesn't exist using the groupadd command as previously discussed.

Next;

usermod -aG "$group" "$username"
      if [ $? -eq 0 ]; then
        echo "Added $username to group $group" | tee -a $LOG_FILE
      else
        echo "Failed to add $username to group $group" | tee -a $LOG_FILE
      fi
    done
  fi

we then add the user to group using the usermod -aG command
Finally,

done < "$INPUT_FILE"
echo "User creation process completed." | tee -a $LOG_FILE

we will close our while loop condition and pass the $INPUT_FILE to read lines from it.
Create a directory, touch a file named create.sh (shell script) in the directory and make the file executable;

sudo mkdir -p ~/bash-script
sudo chmod +x ~/bash-script/create_users.sh

open the create and paste this script;

#!/bin/bash
#check if script is run as root
if [ "$EUID" -ne 0 ]; then
  echo "Please run as root"
  exit 1
fi

# Check if the input file is provided
if [ -z "$1" ]; then
  echo "Usage: bash create_users.sh <name-of-text-file>"
  exit 1
fi

INPUT_FILE="$1"
LOG_FILE="/var/log/user_management.log"
PASSWORD_FILE="/var/secure/user_passwords.txt"

# Ensure log and password files exist and have the correct permissions
touch $LOG_FILE
chmod 644 $LOG_FILE

mkdir -p /var/secure
touch $PASSWORD_FILE
chmod 600 $PASSWORD_FILE

# Function to generate random password
generate_password() {
  openssl rand -base64 12
}

# Process the input file
while IFS=';' read -r username groups; do
  # Create personal group for the user
  if ! getent group "$username" &>/dev/null; then
    groupadd "$username"
    echo "Created group $username" | tee -a $LOG_FILE
  fi

  # Check if user already exists
  if id "$username" &>/dev/null; then
    echo "User $username already exists. Skipping..." | tee -a $LOG_FILE
    continue
  fi

  # Create user and personal group
  useradd -m -s /bin/bash -g "$username" "$username"
  if [ $? -eq 0 ]; then
    echo "Created user $username with a personal group $username" | tee -a $LOG_FILE
  else
    echo "Failed to create user $username" | tee -a $LOG_FILE
    continue
  fi

  # Generate password and set it for the user
  password=$(generate_password)
  echo "$username:$password" | chpasswd
  if [ $? -eq 0 ]; then
    echo "$username,$password" >> $PASSWORD_FILE
    echo "Set password for $username" | tee -a $LOG_FILE
  else
    echo "Failed to set password for $username" | tee -a $LOG_FILE
  fi

  # Create additional groups if specified
  if [ -n "$groups" ]; then
    IFS=',' read -ra group_array <<< "$groups"
    for group in "${group_array[@]}"; do
      # Create group if it doesn't exist
      if ! getent group "$group" &>/dev/null; then
        groupadd "$group"
        echo "Created group $group" | tee -a $LOG_FILE
      fi

      usermod -aG "$group" "$username"
      if [ $? -eq 0 ]; then
        echo "Added $username to group $group" | tee -a $LOG_FILE
      else
        echo "Failed to add $username to group $group" | tee -a $LOG_FILE
      fi
    done
  fi
done < "$INPUT_FILE"

echo "User creation process completed." | tee -a $LOG_FILE

Create a file (input_file.txt) containing username;groups

light; sudo,dev,www-data
idimma; sudo
mayowa; dev,www-data

Run the script and pass the input_file.txt as argument.

Thank you for reading, to learn more kindly join the HNG internship programme to get your tech skill upgraded and land you dream job.
Follow this link
https://hng.tech/internship,
https://hng.tech/hire

hsh