DEV Community: nithinalias

Devops Project - CI/CD Jenkins Ansible Kubernetes

nithinalias — Tue, 05 Apr 2022 14:13:50 +0000

Kubernetes on AWS





                                               Dockerhub 
                                              /        \
                                            /            \                          
                                      Push/Image       Pull\Image        
                                        /                    \
      Pull Code       Copy Artifacts  /     Deploy Container   \
Github--------->Jenkin-------------->Ansible--------------->Kubernetes         |
                    |  
                    | Build Code
                   \|/
                  Maven

Setup Kubernetes on Amazon EKS

You can follow same procedure in the official AWS document Getting started with Amazon EKS – eksctl

Pre-requisites:

an EC2 Instance
Install AWSCLI latest verison

Setup kubectl

Download kubectl version 1.21
Grant execution permissions to kubectl executable
Move kubectl onto /usr/local/bin
Test that your kubectl installation was successful

   curl -o kubectl https://amazon-eks.s3.us-west-2.amazonaws.com/1.21.2/2021-07-05/bin/linux/amd64/kubectl
   chmod +x ./kubectl
   mv ./kubectl /usr/local/bin 
   kubectl version --short --client

Setup eksctl

Download and extract the latest release
Move the extracted binary to /usr/local/bin
Test that your eksclt installation was successful

   curl --silent --location "https://github.com/weaveworks/eksctl/releases/latest/download/eksctl_$(uname -s)_amd64.tar.gz" | tar xz -C /tmp
   sudo mv /tmp/eksctl /usr/local/bin
   eksctl version

Create an IAM Role and attache it to EC2 instance

Note: create IAM user with programmatic access if your bootstrap system is outside of AWS

IAM user should have access to

IAM

EC2

CloudFormation

Note: Check eksctl documentaiton for Minimum IAM policies

Create your cluster and nodes

   eksctl create cluster --name cluster-name  \
   --region region-name \
   --node-type instance-type \
   --nodes-min 2 \
   --nodes-max 2 \ 
   --zones <AZ-1>,<AZ-2>

   example:
   eksctl create cluster --name valaxy-cluster \
   --region ap-south-1 \
   --node-type t2.small \

To delete the EKS clsuter

   eksctl delete cluster cluster-name --region ap-south-1

Validate your cluster using by creating by checking nodes and by creating a pod

   kubectl get nodes
   kubectl run tomcat --image=tomcat

Deploying Nginx pods on Kubernetes

Deploying Nginx Container

    kubectl create deployment  demo-nginx --image=nginx --replicas=2 --port=80
    # kubectl deployment regapp --image=valaxy/regapp --replicas=2 --port=8080
    kubectl get all
    kubectl get pod

Expose the deployment as service. This will create an ELB in front of those 2 containers and allow us to publicly access them.

   kubectl expose deployment demo-nginx --port=80 --type=LoadBalancer
   # kubectl expose deployment regapp --port=8080 --type=LoadBalancer
   kubectl get services -o wide

Create Pod and Service using manifest file

Create pod manifest file

sudo su -
vim pod.yml

apiVersion: v1
kind: Pod
metadata:
  name: demo-pod
  labels: 
    app: demo-app

spec:
  containers: 
    - name: demo-nginx
      image: nginx
      ports:
        - name: demo-nginx
          containerPort: 80

Create service manifest file

vim service.yml

apiVersion: v1
kind: Service
metadata:
  name: demo-service
  labels: 
    app: demo-app

spec:
  ports:
  - name: nginx-port
    port: 80
    targetPort: 80
  selector:
    app: demo-app
  type: LoadBalancer

Run pod and service

kubectl apply -f pod.yml
kubectl get pod

kubectl apply -f service.yml
kubectl get service

kubectl get all
kubectl get pods -o wide
kubectl get services -o wide
kubectl describe pod podname
kubectl describe service servicename

Setup Pod and Service

                          selector:                labels:
                            app: demo-app            app: demo-app
External network-------------->Service------------------->Pod

As we know there will be large number of pods,When we send a request from External network to Service it looks for selector app: demo-app and forward the requset to the pods having Labels app: demo-app.That is why we use selector in service manifest file and labels in pod manifest file.

Integrating Kubernetes in CI/CD pipeline

Go to ansible machine

create a user and allow password authentication.

sudo su -
adduser ansadmin
id ansadmin

vim /etc/ssh/sshd_config

PasswordAuthentication yes
#PasswordAuthentication no

systemctl restart ssh

Modify visudo for created user.

vim /etc/sudoers

# User privilege specification
root    ALL=(ALL:ALL) ALL
ansadmin ALL=(ALL:ALL) NOPASSWD: ALL

User Privilege Lines

root ALL=(ALL:ALL) ALL The first field indicates the username that the rule will apply to (root).

root ALL=(ALL:ALL) ALL The first “ALL” indicates that this rule applies to all hosts.

root ALL=(ALL:ALL) ALL This “ALL” indicates that the root user can run commands as all users.

root ALL=(ALL:ALL) ALL This “ALL” indicates that the root user can 
run commands as all groups.

root ALL=(ALL:ALL) ALL The last “ALL” indicates these rules apply to all commands.

Generate ssh-key.

su - ansadmin
ssh-keygen

Go to Kubernetes machine

Create Deployment and Service using manifest file

Create deployment manifest file

sudo su -
vim deployment.yml

apiVersion: v1
kind: Deployment
metadata:                        --->Deployment name and label
  name: demo-webapp 
  labels: 
    app: webapp

spec:
  replicas: 2
  selector:                      --->Create 2 pods from pod template
    matchLabels:
       app: webapp

template:                                                    |
    metadata:                    ---> pod definition         |        
      labels:                                                |       
        app: webapp                                          |
                                                             |           
    spec: containers                                         |---> Template to create a pod                                     |
    - name: webapp                                           |
      image: nithinalias/mytomcat ---> container definition  |
      imagePullPolicy: Always                                |
      ports:                                                 |
      - containerPort: 8080                                  |   

strategy:
  type: RollingUpdate
  rollingUpdate:                 --->make sure only one pod updated at a time
    maxSugre: 1
    maxUnavialable: 1

Create service manifest file

vim service.yml

apiVersion: v1
kind: Service                    ---> Resource Type

metadata:
  name: demo-service             ---> service name and label
  labels: 
    app: webapp

spec:
  ports:
  - name: nginx-port
    port: 8080                   ---> port number exposed at cluster level
    targetPort: 8080             ---> port that container listening

  selector:                      ---> To which deployment it can send traffic
    app: webapp
  type: LoadBalancer             ---> service type

labels in deployment manifest file need to match with selector in service manifest file.containerPort in deployment manifest file need to match with targetPort in service manifest file.

sudo su -
passwd
adduser ansadmin
id ansadmin
docker login

Allow password authentication.

vim /etc/ssh/sshd_config

PermitRootLogin yes
PasswordAuthentication yes
#PasswordAuthentication no

systemctl restart ssh

Modify visudo for created user.

vim /etc/sudoers

# User privilege specification
root    ALL=(ALL:ALL) ALL
ansadmin ALL=(ALL:ALL) NOPASSWD: ALL

Go to ansible machine and add ip address of kubernetes machine and ansible machine(then only jenkin server can access ansible and kubernetes machine) inside ansible host file.

vim /etc/ansible/hosts

[ansible]
192.168.33.25
[kubernetes]
192.168.33.30

copy the ssh public key to kubernetes machine and ansible machine itself(localhost).

su - ansadmin
ssh-copy-id ansadmin@ip-of-ansiblemachine
ssh-copy-id ansadmin@ip-of-kubernetesmachine
ssh-copy-id root@ip-of-kubernetesmachine
ansible all -m ping

Now authorized key will be generated inside /home/ansadmin/.ssh/authorized_keys of kubernetes,ansible machine and /root/.ssh/authorized_keys of kubernetes machine.

create a directory,give ownership of ansadmin,add ansadmin to docker group and docker.sock permissions to all users.

sudo su -
mkdir /opt/docker
chown ansadmin:ansadmin /opt/docker
usermod -aG docker ansadmin
id ansadmin
chmod 777 /var/run/docker.sock
systemctl restart docker
su - ansadmin
docker login

Create dockerfile for tomcat image creation.

vim /opt/docker/dockerfile

FROM tomcat
RUN cp -R /usr/local/tomcat/webapps.dist/* /usr/local/tomcat/webapps
COPY ./*.war /usr/local/tomcat/webapps

Generate ansible-playbook for building tomcat image.

vim /opt/docker/build_tomcatimage.yml

---

- hosts: ansible

  tasks:
  - name: create docker image
    command: docker build -t mytomcat .
    args:
     chdir: /opt/docker

  - name: create tag to push image onto dockerhub
    command: docker tag mytomcat nithinalias/mytomcat

  - name: push docker image
    command: docker push nithinalias/mytomcat

Generate ansible-playbook to run deployment and service manifest file.

vim /opt/docker/run_kube_manifestfile.yml

---

- hosts: kubernetes
  user: root

  tasks:
  - name: run deployment manifestfile on kubernetes
    command: kubectl apply -f deployment.yml

  - name: run service manifestfile on kubernetes
    command: kubectl apply -f service.yml

  - name: update deployment with new pods if image updated in dockerhub
    command: kubectl rollout restart deployment.apps/demo-webapp

# we don't want to mention exact path like /root/deployment.yml or /root/service.yml, since we are performing the task as root user.

Integrate Ansible with Jenkins

we need to add ansadmin ssh connection of ansible machine.

Manage Jenkins - Configure System - SSH servers - Add - Add name(ansible-server),Hostname(ip-of-ansible machine),username
(ansadmin) - Advanced - Enable Use password authentication or use a different key - Add password of user ansadmin(instead you can use ssh-key/path of ssh-key if present) - Test configuration - Apply - save.

Continuous Integration using Jenkins

New Item - Add item name(CI webapp),select Maven project - OK - Add description - Select Git and add Repositories URL,Branch - Build Triggers(Poll SCM=* * * * * means every minute check the repository and if there any update it will trigger build - Build periodically means it will trigger build periodically even if repository not updated) - Build(Add Root POM=pom.xml,Goals and options=clean install) - Add post-build Action - Send build artifacts over SSH - Select SSH server name(ansible-server),Add sourcefile(webapp/target/*.war),Add Remove prefix(webapp/target),Add Remote directory="//opt//docker"(if it is blank webapp.war will be in /home/ansadmin),
Exec command ="ansible-playbook /opt/docker/build_tomcatimage.yml" - Add post-build action - Build other projects - Go to post-build action(top) - Add Projects to build="CD webapp" - Enable Trigger only if build is stable - Apply - Save - Build Now - Console output.You can see all build outcomes in /var/lib/jenkins/workspace/CI webapp/webapp/target.surefire contain the reports of build and webapp.war is called artifacts.This webapp.war will be in ansible machine /opt/docker directory and copy this to tomcat image.The image created inside ansible machine is pushed to dockerhub.

Continuous Deployment using Jenkins

New Item - Add item name(CD webapp) - OK - Add description - Add post-build Action - Send build artifacts over SSH - Select SSH server name(ansible-server),Exec command =
"ansible-playbook /opt/docker/run_kube_manifestfile.yml" - Apply - Save.

Go to Kubernetes machine

kubectl get service

copy the EXTERNAL-IP of your service and paste it in browser.Now you get the webapp.
OR

You can go to LoadBalancer in AWS - Description - Copy DNS name - paste it in browser.Now you get the webapp.

Source: https://github.com/yankils

Devops Project - CI/CD Jenkins Ansible Docker

nithinalias — Wed, 30 Mar 2022 04:10:33 +0000

CI/CD pipeline using Git,Jenkins and Maven

Install jenkins using vagrant file

# -*- mode: ruby -*-
# vi: set ft=ruby :


Vagrant.configure("2") do |config|

  config.vm.box = "ubuntu/focal64"

   config.vm.network "private_network", ip: "192.168.33.10"


  config.vm.provider "virtualbox" do |vb|
  vb.memory = "1536"
  end

   config.vm.provision "shell", inline: <<-SHELL
   apt update
   apt install -y openjdk-11-jre
   wget -q -O - https://pkg.jenkins.io/debian-stable/jenkins.io.key | apt-key add -
   sh -c 'echo deb http://pkg.jenkins.io/debian-stable binary/ > /etc/apt/sources.list.d/jenkins.list'
   apt update
   apt install -y jenkins
   SHELL
   end

Now login to jenkin http://ip-address:8080 and enter the password
that you get from /var/lib/jenkins/secrets/initialAdminPassword.You can change the password from Manage jenkins - Manage users.

Integrate Git with Jenkins

Manage Jenkins - Manage plugins - Available - Select Github - Install without restart - Go back to the top page - Manage Jenkins - Global Tool Configuration - Add git name,Add Path to Git executable(This we get from linux terminal by using command 'whereis git' and you get ouput '/usr/bin/git'.If it won't work you can use 'git'.) - Apply - save

New Item - Add item name(PullcodefromGithub),select Freestyle - OK - Add description - Select Git and add Repositories URL,Branch - Apply - Save - Build Now - Console output.You can see the output
/var/lib/jenkins/workspace.

Integrate Maven with Jenkins

Install maven

Download the Binary tar.gz archive from https://maven.apache.org/download.cgi

sudo su -
cd /opt
wget https://dlcdn.apache.org/maven/maven-3/3.8.5/binaries/apache-maven-3.8.5-bin.tar.gz
tar -xvzf apache-maven-3.8.5-bin.tar.gz
cd /opt/apache-maven-3.8.5/bin
./mvn -v

Outside this directory 'mvn -v' won't work.we need to set up the environment variables(path of maven,java).

find / -name jvm
export PATH=$PATH:/opt/apache-maven-3.8.5:/opt/apache-maven-3.8.5/bin:/usr/lib/jvm/java-11-openjdk-amd64
echo $PATH

To change enviornment variables permanentely.

vim /root/.profile

# ~/.profile: executed by Bourne-compatible login shells.

if [ "$BASH" ]; then
  if [ -f ~/.bashrc ]; then
    . ~/.bashrc
  fi
fi

M2_HOME=/opt/apache-maven-3.8.5
M2=/opt/apache-maven-3.8.5/bin
JAVA_HOME=/usr/lib/jvm/java-11-openjdk-amd64
PATH=$PATH:$HOME/bin:$JAVA_HOME:$M2_HOME:$M2
export PATH
mesg n 2> /dev/null || true

echo $PATH
source .profile
echo $PATH

Manage Jenkins - Manage plugins - Available - Select Maven Integration - Install without restart - Go back to the top page - Manage Jenkins - Global Tool Configuration - Add JDK name,JAVA_HOME(/usr/lib/jvm/java-11-openjdk-amd64),Maven name,MAVEN_HOME(/opt/apache-maven-3.8.5) - Apply - save

New Item - Add item name(Mavenproject),select Maven project - OK - Add description - Select Git and add Repositories URL,Branch - Build(Add Root POM=pom.xml,Goals and options=clean install) - Apply - Save - Build Now - Console output.You can see all build outcomes in /var/lib/jenkins/workspace/Mavenproject/webapp/target/.surefire contain the reports of build and webapp.war is called artifacts.

Integrating Tomcat server in CI/CD pipeline




       Pull Code        Copy Artifacts & Deploy Code
Github---------->Jenkin----------------------------->Tomcat server
                    |  
                    | Build Code
                   \|/
                  Maven

Setup a Tomcat server
create a virtual machine and install JDK,Tomcat server.
Download tar.gz tomcat packages from https://tomcat.apache.org/download-80.cgi

sudo su -
apt update
apt install -y openjdk-11-jre
cd /opt
wget https://dlcdn.apache.org/tomcat/tomcat-8/v8.5.77/bin/apache-tomcat-8.5.77.tar.gz
tar -xvzf apache-tomcat-8.5.77.tar.gz
cd /opt/apache-tomcat-8.5.77/bin
./startup.sh

Now login to tomcat http://ip-address:8080 - Manage App - 403 Access Denied
By default the Manager is only accessible from a browser running on the same machine as Tomcat. If you wish to modify this restriction, you'll need to edit the Manager's context.xml file.

find / -name context.xml
vim /opt/apache-tomcat-8.5.77/webapps/host-manager/META-INF/context.xml
vim /opt/apache-tomcat-8.5.77/webapps/manager/META-INF/context.xml

Disable the line shown below by using  in both context.xml file.

<!--  <Valve className="org.apache.catalina.valves.RemoteAddrValve"
allow="127\.\d+\.\d+\.\d+|::1|0:0:0:0:0:0:0:1" />  -->

Now restart tomcat server.

cd /opt/apache-tomcat-8.5.77/bin
./shutdown.sh
./startup.sh

create link files for tomcat startup.sh and shutdown.sh

ln -s /opt/apache-tomcat-8.5.77/bin/startup.sh /usr/local/bin/tomcatup
ln -s /opt/apache-tomcat-8.5.77/bin/shutdown.sh /usr/local/bin/tomcatdown
tomcatup

Update users information in the tomcat-users.xml file goto tomcat home directory and Add below users to /opt/apache-tomcat-8.5.77/conf/tomcat-users.xml file.

<role rolename="manager-gui"/>
<role rolename="manager-script"/>
<role rolename="manager-jmx"/>
<role rolename="manager-status"/>
<user username="admin" password="admin" roles="manager-gui, manager-script, manager-jmx, manager-status"/>
<user username="deployer" password="deployer" roles="manager-script"/>
<user username="tomcat" password="s3cret" roles="manager-gui"/>

Now restart tomcat server and using credential login to tomcat http://ip-address:8080 - Manage App

tomcatdown
tomcatup

Integrate Tomcat with jenkin

Manage Jenkins - Manage plugins - Available - Select Deploy to container(This plugin allows you to deploy a war to a container after a successful build) - Install without restart - Go back to the top page - Manage Jenkins - Manage Credentials - Jenkins - Global credentials - Add credentials - kind(username with password) - username="deployer",password="deployer"(If one system want to access another system we need roles="manager-script"),ID="tomcat_deployer",Description="tomcat_deployer" - Apply - save

New Item - Add item name(BuildAndDeploy),select Maven project - OK - Add description - Select Git and add Repositories URL,Branch - Build Triggers(Poll SCM=* * * * * means every minute check the repository and if there any update it will trigger build - Build periodically means it will trigger build periodically even if repository not updated) - Build(Add Root POM=pom.xml,Goals and options=clean install) - Add post-build Action - Deploy war/ear to a container - Add WAR/EAR files=*/.war,Add Containers=Tomcat 8.x Remote,select the deployer credentail that we created,Tomcat URL=http://ip-address:8080 - Apply - Save - Build Now - Console output.You can see all build outcomes in /var/lib/jenkins/workspace/BuildAndDeploy/webapp/target.
surefire contain the reports of build and webapp.war is called artifacts.This webapp.war is deployed inside /opt/apache-tomcat-8.5.77/webapps of tomcat server by authencating the credentials.

Using credential login to tomcat http://ip-address:8080 - Manage App - select the path webapp - Now you get the required output.

Integrate Docker in CI/CD Pipeline



       Pull Code        Copy Artifacts & Deploy Code
Github---------->Jenkin-----------------------------> Docker
                    |  
                    | Build Code
                   \|/
                  Maven

Create a docker virtual machine using Vagrantfile.

# -*- mode: ruby -*-
# vi: set ft=ruby :

Vagrant.configure("2") do |config|

  config.vm.box = "ubuntu/focal64"

  config.vm.network "private_network", ip: "192.168.33.20"

  config.vm.provider "virtualbox" do |vb|
  vb.memory = "1024"
  end

  config.vm.provision "shell", inline: <<-SHELL
     apt update -y
     apt install apt-transport-https ca-certificates -y
     curl software-properties-common
     curl -fsSL https://download.docker.com/linux/ubuntu/gpg | apt-key add -
     add-apt-repository "deb [arch=amd64] https://download.docker.com/linux/ubuntu focal stable"
     apt-cache policy docker-ce
     apt install docker-ce -y
  SHELL
end

Pull and run tomcat image.

sudo su -
docker pull tomcat
docker images
docker run -d --name tomcat-container 8081:8080 tomcat
docker ps -a

Login to tomcat http://ip-address:8081.It will show an error like HTTP Status 404 – Not Found.This is because when we browse http://ip-address:8081 it will look into /usr/local/tomcat/webapps
directory.By default this directory will be empty and contents will be present inside /usr/local/tomcat/webapps.dist.so we need to copy this contents to webapp directory.

docker exec -it tomcat-container /bin/bash
cd /usr/local/tomcat/webapps
ls
cd /usr/local/tomcat/webapps.dist
ls
cp -R /usr/local/tomcat/webapps.dist/* /usr/local/tomcat/webapps
exit

Create tomcat using Dockerfile

sudo su -
vim Dockerfile

FROM ubuntu
RUN apt-get -y update
RUN apt-get -y install openjdk-11-jre
RUN mkdir /opt/tomcat
WORKDIR /opt/tomcat
ADD https://dlcdn.apache.org/tomcat/tomcat-9/v9.0.60/bin/apache-tomcat-9.0.60.tar.gz /opt/tomcat
RUN tar -xvzf apache-tomcat-9.0.60.tar.gz
RUN mv apache-tomcat-9.0.60/* /opt/tomcat
EXPOSE 8080
CMD [ "/opt/tomcat/bin/catalina.sh", "run"]


docker build -t mytomcatserver .
docker run -d --name mytomcat-server -p 8083:8080 mytomcatserver

Create customized Dockerfile for tomcat

sudo su -
vim Dockerfile

FROM tomcat
RUN cp -R /usr/local/tomcat/webapps.dist/* /usr/local/tomcat/webapps


docker build -t mytomcat .
docker run -d --name mytomcat-container -p 8085:8080 mytomcat

Integrate docker with Jenkins and Update tomcat dockerfile to automate deployment process
Enable password authentication in docker machine.

sudo su -
vim /etc/ssh/sshd_config

PasswordAuthentication yes
#PasswordAuthentication no

systemctl restart ssh

Create a user and add to group 'docker'.

adduser dockeradmin
cat /etc/group
usermod -aG docker dockeradmin
id dockeradmin

Create tomcat dockerfile

mkdir /opt/docker
vim /opt/docker/Dockerfile

FROM tomcat
RUN cp -R /usr/local/tomcat/webapps.dist/* /usr/local/tomcat/webapps
COPY ./*.war /usr/local/tomcat/webapps

we need to give ownership of Dockerfile to dockeradmin.Then only we can automate the deployment using jenkin server.After building, webapp.war inside jenkin server workspace(/var/lib/jenkins/workspace/BuildAndDeploy/webapp/target) will be move to /opt/docker of docker machine.

chown -R dockeradmin:dockeradmin /opt/docker

Go to Jenkin Server

Manage Jenkins - Manage plugins - Available - Select Publish Over SSH(Send build artifacts over SSH) - Install without restart - Go back to the top page - Manage Jenkins - Configure System - SSH servers - Add - Add name(dockerhost),Hostname(ip-of-dockermachine),username(dockeradmin) - Advanced - Enable Use password authentication or use a different key - Add password of user dockeradmin(instead you can use ssh-key/path of ssh-key if present) - Test configuration - Apply - save.

New Item - Add item name(BuildAndDeploy),select Maven project - OK - Add description - Select Git and add Repositories URL,Branch - Build Triggers(Poll SCM=* * * * * means every minute check the repository and if there any update it will trigger build - Build periodically means it will trigger build periodically even if repository not updated) - Build(Add Root POM=pom.xml,Goals and options=clean install) - Add post-build Action - Send build artifacts over SSH - Select SSH server name(dockerhost),Add sourcefile(webapp/target/*.war),Add Remove prefix(webapp/target),Add Remote directory="//opt//docker"(if it is blank webapp.war will be in /home/dockeradmin),
Exec command ="cd /opt/docker;
docker build -t mytomcat .;
docker container stop tomcat-container;
docker rm tomcat-container;
docker run -d --name tomcat-container -p 8081:8080 mytomcat" - Apply - Save - Build Now - Console output.You can see all build outcomes in /var/lib/jenkins/workspace/BuildAndDeploy/webapp/target.
surefire contain the reports of build and webapp.war is called
artifacts.This webapp.war will be in docker machine /opt/docker directory and copy this to tomcat image for tomcat container creation.

Now you can browse http://ip-adress:8081/webapp/

Integrating ansible in CI/CD pipeline





                                               Dockerhub 
                                              /        \
                                            /            \                          
                                      Push/Image       Pull\Image        
                                        /                    \
      Pull Code       Copy Artifacts  /     Deploy Container   \
Github--------->Jenkin-------------->Ansible--------------->Docker
                    |  
                    | Build Code
                   \|/
                  Maven

Create a ansible machine having docker using vagrant file.

# -*- mode: ruby -*-
# vi: set ft=ruby :

Vagrant.configure("2") do |config|

  config.vm.box = "ubuntu/focal64"

  config.vm.network "private_network", ip: "192.168.33.25"

  config.vm.provider "virtualbox" do |vb|
  vb.memory = "1536"
  end

  config.vm.provision "shell", inline: <<-SHELL
     apt update -y
     apt install ansible -y
     apt install apt-transport-https ca-certificates -y
     curl software-properties-common
     curl -fsSL https://download.docker.com/linux/ubuntu/gpg | apt-key add -
     add-apt-repository "deb [arch=amd64] https://download.docker.com/linux/ubuntu focal stable"
     apt-cache policy docker-ce
     apt install docker-ce -y
  SHELL
end

create a user and allow password authentication.

sudo su -
adduser ansadmin
id ansadmin

vim /etc/ssh/sshd_config

PasswordAuthentication yes
#PasswordAuthentication no

systemctl restart ssh

Modify visudo for created user.

vim /etc/sudoers

# User privilege specification
root    ALL=(ALL:ALL) ALL
ansadmin ALL=(ALL:ALL) NOPASSWD: ALL

User Privilege Lines

root ALL=(ALL:ALL) ALL The first field indicates the username that the rule will apply to (root).

root ALL=(ALL:ALL) ALL The first “ALL” indicates that this rule applies to all hosts.

root ALL=(ALL:ALL) ALL This “ALL” indicates that the root user can run commands as all users.

root ALL=(ALL:ALL) ALL This “ALL” indicates that the root user can 
run commands as all groups.

root ALL=(ALL:ALL) ALL The last “ALL” indicates these rules apply to all commands.

Generate ssh-key.

su - ansadmin
ssh-keygen

Go to docker machine and create user,add to group docker,docker.sock permissions to all users

sudo su -
adduser ansadmin
usermod -aG docker ansadmin
id ansadmin
chmod 777 /var/run/docker.sock
systemctl restart docker
docker login

Allow password authentication.

vim /etc/ssh/sshd_config

PasswordAuthentication yes
#PasswordAuthentication no

systemctl restart ssh

Modify visudo for created user.

vim /etc/sudoers

# User privilege specification
root    ALL=(ALL:ALL) ALL
ansadmin ALL=(ALL:ALL) NOPASSWD: ALL

Go to ansible machine and add ip address of docker machine and ansible machine(then only jenkin server can access ansible and docker machine) inside ansible host file.

vim /etc/ansible/hosts

[docker]
192.168.33.20
[ansible]
192.168.33.25

copy the ssh public key to docker machine and ansible machine itself(localhost).

su - ansadmin
ssh-copy-id ansadmin@ip-of-ansiblemachine
ssh-copy-id ansadmin@ip-of-dockermachine
ansible all -m ping

Now authorized key will be generated inside /home/ansadmin/.ssh/authorized_keys of docker and ansible machine.

create a directory,give ownership of ansadmin,add ansadmin to docker group and docker.sock permissions to all users.

sudo su -
mkdir /opt/docker
chown ansadmin:ansadmin /opt/docker
usermod -aG docker ansadmin
id ansadmin
chmod 777 /var/run/docker.sock
systemctl restart docker
su - ansadmin
docker login

Create dockerfile for tomcat image creation.

vim /opt/docker/dockerfile

FROM tomcat
RUN cp -R /usr/local/tomcat/webapps.dist/* /usr/local/tomcat/webapps
COPY ./*.war /usr/local/tomcat/webapps

Generate ansible-playbook for building tomcat image.

vim /opt/docker/build_tomcatimage.yml

---

- hosts: ansible

  tasks:
  - name: create docker image
    command: docker build -t mytomcat .
    args:
     chdir: /opt/docker

  - name: create tag to push image onto dockerhub
    command: docker tag mytomcat nithinalias/mytomcat

  - name: push docker image
    command: docker push nithinalias/mytomcat

Generate ansible-playbook for deploy tomcat.

vim /opt/docker/deploy_tomcat.yml

---
- hosts: docker

  tasks:
  - name: stop existing container
    command: docker stop tomcat-container
    ignore_errors: yes

  - name: remove the container
    command: docker rm tomcat-container
    ignore_errors: yes

  - name: remove image
    command: docker rmi nitinalias/mytomcat
    ignore_errors: yes

  - name: create container
    command: docker run -d --name tomcat-container -p 8082:8080 nithinalias/mytomcat


# If there is no container it will show error.To avoid this we use ignore errors: yes
~

Integrate Ansible with Jenkins

Already we installed Publish Over SSH plugin and added dockeradmin
ssh connection of docker machine.Now we need to add ansadmin
ssh connection of ansible machine.

New Item - Add item name(BuildAndDeployUsingAnsible),select Maven project - OK - Add description - Select Git and add Repositories URL,Branch - Build Triggers(Poll SCM=* * * * * means every minute check the repository and if there any update it will trigger build - Build periodically means it will trigger build periodically even if repository not updated) - Build(Add Root POM=pom.xml,Goals and options=clean install) - Add post-build Action - Send build artifacts over SSH - Select SSH server name(ansible-server),Add sourcefile(webapp/target/*.war),Add Remove prefix(webapp/target),Add Remote directory="//opt//docker"(if it is blank webapp.war will be in /home/ansadmin),
Exec command ="ansible-playbook /opt/docker/build_tomcatimage.yml;
sleep 10;
ansible-playbook /opt/docker/deploy_tomcat.yml" - Apply - Save - Build Now - Console output.You can see all build outcomes in /var/lib/jenkins/workspace/BuildAndDeployUsingAnsible/
webapp/target.surefire contain the reports of build and webapp.war is called artifacts.This webapp.war will be in ansible machine /opt/docker directory and copy this to tomcat image.The image created inside ansible machine is pushed to dockerhub.Pull this image from dockerhub to create tomcat-container inside docker
machine.

Now you can browse http://ip-adress:8082/webapp/

Note: To check ansible-playbook and run ansible-playbook on particular host you can use the command shown below.

ansible-playbook /opt/docker/deploy_tomcat.yml --check
ansible-playbook /opt/docker/deploy_tomcat.yml --limit ip-adress/groupname

Source: https://github.com/yankils

Introduction to Kubernetes

nithinalias — Mon, 21 Mar 2022 06:00:06 +0000

Introduction to MINIKUBE

MINIKUBE

- Open source tool
- To run K8s locally on your system
- Runs a single node K8s cluster inside a VM on your system.
 ___________________                    _____________________
|                   |                  |                     |
|    Minikube       |                  |      MASTER         |
|       /|\         |                  |      ______         |
|        |          |                  |      ______         |
|        |          |______________|\  |                     |
|       VM          |              | \ |      WORKER         |
|       /|\         |______________| / |      ______         |
|        |          |              |/  |      ______         |
|        |          |                  |                     |
|        |          |                  |DOCKER PRE-INSTALLED |
|     HOST OS       |                  |                     |
|___________________|                  |_____________________|
  HOST MACHINE                                 NODE

Install minikube

apt-get update
apt-get install curl
apt-get install apt-transport-https
apt install virtualbox virtualbox-ext-pack

Install kubectl
https://kubernetes.io/docs/tasks/tools/
Install Docker
https://docs.docker.com/engine/install/ubuntu/
Install minikube
https://minikube.sigs.k8s.io/docs/start/

minikube start
kubectl get po -A
minikube dashboard

K8s Commands

kubectl get nodes
kubectl get pods -o wide
kubectl cluster-info
kubectl config view
kubectl config get-clusters
kubectl config delete-clusters
kubectl config get-contexts
kubectl config current-context
kubectl config set-context context1 --cluster=cluster1 --user=user1 --namespace=namespace1
kubectl config use-context context1
kubectl config current-context
kubectl config get-contexts
kubectl config delete-context context1
kubectl get namespace
kubectl config set-context --current --namespace=namespace1
kubectl create namespace namespace2
kubectl get namespace
kubectl delete namespace namespace2
kubectl get pods --namespace=default
kubectl api-versions

Introduction to KUBECTL

KUBECTL

- Kubernetes command-line tool
- Allows you to run commands against K8s clusters.
- use kubectl to deploy applications, inspect and manage cluster 
  resources, and view logs
- KUBECTL is the most powerful of three clients(UI,API,KUBECTL).
- create pods,create services,destroy pods....

                                                       UI
                                 ______________________/_    
                                |       ____________  /  |
                                |      |            |/   |
   MASTER PROCESSES--------------------| API SERVER |----- API
                                |      |____________|\   |           
                                |                     \  |
                                |                    CLI (KUBECTL)
                                |                        |
                                | /  POD       SERVICE   | 
   WORKER PROCESSES------------------POD       SECRET    |
                                | \  POD       CONFIGMAP |
                                |                        | 
                                |________________________|
                                           NODE
Install Kubectl
https://kubernetes.io/docs/tasks/tools/

To Create a Kubernetes Cluster Using Kubeadm

You can use the two links shown below for Kubernetes Cluster creation


https://github.com/kodekloudhub/certified-kubernetes-administrator-course

https://kubernetes.io/docs/setup/production-environment/tools/kubeadm/install-kubeadm/

https://www.cloudsigma.com/how-to-install-and-use-kubernetes-on-ubuntu-20-04/

https://alibaba-cloud.medium.com/how-to-install-and-deploy-kubernetes-on-ubuntu-16-04-6769fd1646db

https://www.digitalocean.com/community/tutorials/how-to-create-a-kubernetes-cluster-using-kubeadm-on-ubuntu-20-04

 ___________________                    _____________________
|                   |                  |                     |
|Container Platform |                  |  Container Platform |
|___________________|                  |_____________________|
| Kubeadm,Kubeclet, |                  |  Kubeadm,Kubeclet,  |
|     Kubectl       |                  |      Kubectl        | 
|___________________|                  |_____________________|
|Designate to become|                  |                     |
|      Master       |         _________|  Join the Cluster   |
|___________________|         |        |_____________________|
|  CN Installation  |         |                 SLAVE
|___________________|         |                                 
|                   |         |
|   Cluster Ready   |         | 
|                   |/|_______| 
|___________________|\|                  
      MASTER

Kubeadm Commands

kubeadm init
kubeadm join
kubeadm reset

Deploy a webapplication using kubernetes
In Master create directory and make deployment file

mkdir demo
cd demo
vim webappdeploymentfile.yml

apiVersion: apps/v1
kind: Deployment
metadata:
  name: webapp1
  labels:
    app: webapp-sql
    tier: frontend
spec:
  replicas: 1
  selector:
    matchLabels:
      app: webapp-sql
      tier: frontend
  template:
    metadata:
      labels:
        app: webapp-sql
        tier: frontend
    spec:
      containers:
      - name: webapp1
        image: hshar/webapp
        ports:
        - containerPort: 8081

vim mysqldeploymentfile.yml

apiVersion: apps/v1
kind: Deployment
metadata:
  name: sqldb
  labels:
    app: webapp-sql
    tier: backend
spec:
  replicas: 1
  selector:
    matchLabels:
      app: webapp-sql
      tier: backend
  template:
    metadata:
      labels:
        app: webapp-sql
        tier: backend
    spec:
      containers:
      - name: mysql
        image: hshar/mysql:5.5
        ports:
        - containerPort: 3306

Run the deployment commands

kubectl apply -f webappdeploymentfile.yml
kubectl apply -f mysqldeploymentfile.yml
kubectl get deployment

Make service file

vim webappservice.yml

apiVersion: v1
kind: Service
metadata:
  name: webapp-sql
spec:
  selector:
    app: webapp-sql
    tier: frontend
  ports:
  - port: 80
  type: NodePort

vim mysqlservice.yml

apiVersion: v1
kind: Service
metadata:
  name: webapp-sql1
spec:
  selector:
    app: webapp-sql
    tier: backend
  ports:
  - port: 3306
  clusterIP: None

Run the service commands

kubectl apply -f webappservice.yml
kubectl apply -f mysqlservice.yml
kubectl get services

Enter into webapp container and modify it

kubectl get pods -o wide
kubectl exec -it <webapppodname> bash
kubectl exec -it <mysqlpodname> bash
kubectl get services

Now open the webapplication in the browser http://ip-adress:port

How to create single pod

use the below commands in master node.

kubectl get nodes
kubectl run pod1 --image nginx
kubectl get pods -o wide
kubectl describe pod pod1
kubectl describe pod pod1 | less

How to create multiple pods and update them using yaml file
Use the deployment yaml file given below

vim deploymentfile.yml

apiVersion: apps/v1
kind: Deployment
metadata:
  name: mynginx
spec:
  replicas: 3
  selector:
    matchLabels:
      type: webserver
  template:
    metadata:
      name: mypod
      labels:
        type: webserver
    spec:
      containers:
        - name: c1
          image: nginx:1.7.9

kubectl get all
kubectl describe deployment mynginx
kubectl set image deploy mynginx c1=nginx:1.9.1
kubectl get all

When we start updating nginx:1.7.9 to nginx:1.9.1,replicaset of nginx:1.7.9 scaledown and replicaset of nginx:1.9.1 scaleup.

Ingress

vim Ingress.yml

apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  name: example-ingress
  annotations:
    nginx.ingress.kubernetes.io/rewrite-target: /$1
spec:
  rules:
    - host: hello-world.info
      http:
        paths:
          - path: /
            pathType: Prefix
            backend:
              service:
                name: web
                port:
                  number: 8080


kubectl apply -f Ingress.yml
kubectl get ingress

sudo vim /etc/hosts

10.0.2.15  hello-world.info
#10.0.2.15 is the ADDRESS that we got from 'kubectl get ingress' command

Now we can check the traffic using the command 'curl hello-world.info'.

Statefulset

vim statefulset.yml

apiVersion: v1
kind: Service
metadata:
  name: nginx
  labels:
    app: nginx
spec:
  ports:
  - port: 80
    name: web
  clusterIP: None
  selector:
    app: nginx
---
apiVersion: apps/v1
kind: StatefulSet
metadata:
  name: web
spec:
  serviceName: "nginx"
  replicas: 2
  selector:
     matchLabels:
       app: nginx
  template:
    metadata:
      labels:
        app: nginx
    spec:
      containers:
      - name: nginx
        image: k8s.gcr.io/nginx-slim:0.8
        ports:
        - containerPort: 80
          name: web
        volumeMounts:
        - name: www
          mountPath: /usr/share/nginx/html
  volumeClaimTemplates:
  - metadata:
      name: www
    spec:
      accessModes: [ "ReadWriteOnce" ]
      resources:
        requests:
          storage: 1Gi

kubectl apply -f statefulset.yml
kubectl get all
kubectl get pods -w -l app=nginx
kubectl delete pod -l app=nginx
kubectl get pod -l app=nginx

Now we can see pods are terminating and if you use 'kubectl get pod -l app=nginx' command after sometime, it began container creating.Deleting or Scaling a statefulset down will not delete the volume associated with the statefulset.

Monitoring Kubernetes cluster
Installing and configuring helm

curl https://baltocdn.com/helm/signing.asc | sudo apt-key add -
sudo apt-get install apt-transport-https --yes
echo "deb https://baltocdn.com/helm/stable/debian/ all main" | sudo tee /etc/apt/sources.list.d/helm-stable-debian.list
sudo apt-get update
sudo apt-get install helm

kubectl get ns
kubectl create namespace prometheus
kubectl get ns
helm repo add prometheus-community https://prometheus-community.github.io/helm-charts
helm repo update
helm install kube-prometheus-stack prometheus-community/kube-prometheus-stack --namespace prometheus
kubectl get pods -n prometheus
kubectl port-forward -n prometheus prometheus-kube-prometheus-stack-prometheus-0 9090

Now you browse prometheus by http://ip-addresss:9090.Here you can check memory,pod,namespace,alert,graph,status....

kubectl port-forward -n prometheus kube-prometheus-stack-grafana-6c74f5565b-vfbxq 3000

Now you browse grafana by http://ip-addresss:3000.For getting username and password use the command below.

kubectl get secret --namespace prometheus prometheus-grafana -o yaml

echo "admin-password" | base64 --decode
echo "admin-user" | base64 --decode

Now you can login grafana and monitor kubernetes clusters.

Docker Swarm services

nithinalias — Wed, 16 Mar 2022 10:57:16 +0000

Docker Swarm Commands

docker swarm init --advertise-addr <ip-addr>
docker service ls
docker service ps <name>
docker service create <name> <image-name>
docker service rm <name>
docker service scale <name>=5
docker swarm leave --force
docker node ls
docker node ps
docker node rm <id>

Create three virtual machines using vagrant(managenode,node1,node2).

# -*- mode: ruby -*-
# vi: set ft=ruby :

# All Vagrant configuration is done below. The "2" in Vagrant.configure
# This defines the version of vagrant
Vagrant.configure("2") do |config|

MACHINE = ["managenode","node1","node2"]
N = 2

(0..N).each do |i|

  config.vm.define "#{MACHINE[i]}" do |node|
  node.vm.hostname = MACHINE[i]
  node.vm.box = "ubuntu/focal64"
  node.vm.network :private_network, ip: "192.168.33.#{10+i}"

  node.vm.provider "virtualbox" do |vb|
  vb.memory = "1500"
  end

  node.vm.provision "shell", inline: <<-SHELL
     apt update -y
     apt install apt-transport-https ca-certificates -y
     curl software-properties-common
     curl -fsSL https://download.docker.com/linux/ubuntu/gpg | apt-key add -
     add-apt-repository "deb [arch=amd64] https://download.docker.com/linux/ubuntu focal stable"
     apt-cache policy docker-ce
     apt install docker-ce -y
     SHELL


end
end
end

sudo docker swarm init --advertise-addr <ip-addr of server>

Now you will receive docker swarm join token address.Copy and paste it node1 and node2

docker swarm join --token <join-token> <ipaddr:port number>

Now you can see the created nodes from managenode using the command below.

docker node ls

create an image

Create a dockerfile for Apache webserver and Copy the Index.html

Make a directory
mkdir /test
cd /test
Create a sample web page with name index.html
vim /test/index.html

<!DOCTYPE html>
<html>
<body>

<h1>Hello World</h1>

<p>My Personal website</p>

</body>
</html>

Now create a file dockerfile
vim /test/dockerfile

FROM ubuntu
ENV DEBIAN_FRONTEND=noninteractive
RUN apt-get update
RUN apt-get install apache2 -y
RUN apt-get install apache2-utils -y
RUN apt-get clean
COPY index.html /var/www/html/
EXPOSE 80
CMD ["apache2ctl","-D","FOREGROUND"]

Next step is to build the docker file by using the docker build command.

sudo docker build -t myapachewebserver .

Create Service

docker service create --name "Apachewebserver" -p 8080:80 myapachewebserver

To list services

docker service ls

To list task of services

docker service ps Apachewebserver

Now you can browse the webserver

http://ipaddress-of-managenode:8080
http://ipaddress-of-node1:8080
http://ipaddress-of-node2:8080

Here if any of the node terminated, the service would disturb in that node.If node1 is down then we wouldnot get the webserver through http://ipaddress-of-node1:8080.

To remove service

docker service rm Apachewebserver

If any of the node terminated, the service wouldnot disturbed if you use "--mode global"

docker service create --name "Apachewebserver" -p 8080:80 --mode global myapachewebserver

If you want to make replica you can use command shown below

docker service rm Apachewebserver

docker service create --name "Apachewebserver" -p 8080:80 --replica 2 myapachewebserver

Here managenode will select the best performing node from the node1 and node2 along with managenode.

Now you can browse the webserver

http://ipaddress-of-managenode:8080
http://ipaddress-of-node1:8080
http://ipaddress-of-node2:8080

you can check the services by using the command shown below

docker node ps
docker service ps Apachewebserver
docker service ls

you can check whether the containers present or not by using the command shown below

docker ps

So here we could see the Apachewebserver container only in managenode and any one of the nodes(node1 and node2).

Scaling

You can scale the service by using the command shown below.

docker service scale Apachewebserver=5
docker service ps Apachewebserver

you could see 5 copies of Apachewebserver.Managenode will disrtibute the service according to the best performing nodes like managenode,2 node1 and 2 node2.

If you want to drain the service you can use the command below.

docker node update --availability drain managenode
docker service ps Apachewebserver
docker node ls

Now your managenode availability got drained and an additional node will be created.Now managenode has no power to scale up and down the services.Use the below command to retain the power of managenode.

docker node update --availability active managenode

Docker Swarm Service using overlay network demo

First remove the service

docker service rm Apachewebserver

create an apache service

docker service create --name Apache2 --mode global -d -p 8003:80 httpd

docker service ls
docker service ps Apache2

Now you can browse the webserver

http://ipaddress-of-managenode:8003
http://ipaddress-of-node1:8003
http://ipaddress-of-node2:8003

Create a overlay network

docker network create -d overlay myoverlay1

Lets create a web application using php and mysql.we can pull a sample application from hshar/webapp and create a service out of it.

docker service create --name webapp1 -d --network myoverlay1 -p 8001:80 hshar/webapp

docker service ls
docker service ps Apache2

create a service for mysql

docker service create --name mysql -d --network myoverlay1 -p 3306:3306 hshar/mysql:5.5

docker service ls
docker service ps mysql

Managenode will distribute the webapp1 and myql in different nodes
as part of load balancing.

docker ps
docker exec -it <containerID> bash
nano /var/www/html/index.php

<html>
<head>
<title>Docker Sample App</title>

<?php
if($_SERVER['REQUEST_METHOD'] == "POST")
{
$servername = "mysql";
$username = "root";
$password = "edureka";
$dbname = "docker";
$name=$_POST["name"];
$phone=$_POST["phone"];

// Create connection
$conn = new mysqli($servername, $username, $password, $dbname);
// Check connection
if ($conn->connect_error) {
    die("Connection failed: " . $conn->connect_error);
}

$sql = "INSERT INTO emp (name, phone)
VALUES ('".$name."', '".$phone."')";

if ($conn->query($sql) === TRUE) {
    echo "New record created successfully";
} else {
    echo "Error: " . $sql . "<br>" . $conn->error;
}

$conn->close();
}
?>
</head>
<body>
        <form action="index.php" method="POST">
                <input type="text" name="name">
                <input type="text" name="phone">
                <input type="submit" name="submit">
        </form>
</body>
</html>

exit

docker ps
docker exec -it <containerID> bash

echo $MYSQL_ROOT_PASSWORD
mysql -u root -pedureka
CREATE DATABASE docker;
USE docker;
CREATE TABLE emp(name VARCHAR(15),phone VARCHAR(12));
exit

exit

Now you can browse the webserver

http://ipaddress-of-managenode:8001
http://ipaddress-of-node1:8001
http://ipaddress-of-node2:8001

Add the name and phone.Now you can see the name and phone in table "emp" of database "docker" in mysql.

docker exec -it <containerID> bash

mysql -u root -pedureka
USE docker;
show tables;
select * from emp;
exit

exit

To leave node from docker swarm

docker swarm leave

After a node leaves the swarm, you can run the docker node rm command on a manager node to remove the node from the node list.

docker node rm <nodename>

Docker Compose deploy MERN Stack

nithinalias — Tue, 15 Mar 2022 12:30:23 +0000

Clone the MERN stack directory from github

https://github.com/sidpalas/devops-directive

You can make MERN stack using the blog link shown below.

https://medium.com/swlh/how-to-create-your-first-mern-mongodb-express-js-react-js-and-node-js-stack-7e8b20463e66

Edit the database file inside the MERN stack directory as shown below.

cd mern-docker-compose/
vim server/db/index.js


const mongoose = require('mongoose')

mongoose
    .connect('mongodb://mongo:27017/cinema', { useNewUrlParser: true, useUnifiedTopology: true })
    .catch(e => {
        console.error('Connection error', e.message)
    })

const db = mongoose.connection

module.exports = db

If you are deploying MERN stack in Virtual Machine, you need to change localhost to ipaddress of virtual machine (baseURL: 'http://localhost:3000/api' to baseURL:'http://ipaddress:3000/api').

cd mern-docker-compose/
vim client/src/api/index.js


import axios from 'axios'

const api = axios.create({
    baseURL: 'http://localhost:3000/api',
})

export const insertMovie = payload => api.post(`/movie`, payload)
export const getAllMovies = () => api.get(`/movies`)
export const updateMovieById = (id, payload) => api.put(`/movie/${id}`, payload)
export const deleteMovieById = id => api.delete(`/movie/${id}`)
export const getMovieById = id => api.get(`/movie/${id}`)

const apis = {
    insertMovie,
    getAllMovies,
    updateMovieById,
    deleteMovieById,
    getMovieById,
}

export default apis

Then build and run docker-compose file

make build
make run

Docker compose Install Wordpress using Ansible

nithinalias — Sun, 13 Mar 2022 12:27:39 +0000

Create ansible role

roles
├── files
│   └── docker-compose.yml
│   
├── vars
│    └── main.yml
├── dockerwordpressplaybook.yml
└── readme.md

Add docker-compose.yml in files directory

version: '3'

services:
  # Database
  db:
    container_name: mysql_container
    image: mysql:5.7
    volumes:
      - db_data:/var/lib/mysql
    restart: always
    environment:
      MYSQL_ROOT_PASSWORD: password
      MYSQL_DATABASE: wordpress
      MYSQL_USER: wordpress
      MYSQL_PASSWORD: wordpress
    networks:
      - wpsite
  # phpmyadmin
  phpmyadmin:
    depends_on:
      - db
    container_name: phpmyadmin_container
    image: phpmyadmin/phpmyadmin
    restart: always
    ports:
      - '8080:80'
    environment:
      PMA_HOST: db
      MYSQL_ROOT_PASSWORD: password
    networks:
      - wpsite
  # Wordpress
  wordpress:
    depends_on:
      - db
    container_name: wordpress_container
    image: wordpress:latest
    ports:
      - '8000:80'
    restart: always
    volumes: ['./:/var/www/html']
    environment:
      WORDPRESS_DB_HOST: db:3306
      WORDPRESS_DB_USER: wordpress
      WORDPRESS_DB_PASSWORD: wordpress
    networks:
      - wpsite
networks:
  wpsite:
volumes:
  db_data:

Add main.yml in vars directory

---
admin_user: dev
docker_required_packages:
  - "apt-transport-https"
  - "ca-certificates"
  - "curl"
  - "gnupg-agent"
  - "software-properties-common"
  - "python3-pip"
  - "python3-setuptools"
docker_gpg_url: https://download.docker.com/linux/ubuntu/gpg
docker_repo: deb https://download.docker.com/linux/ubuntu focal stable
docker_packges:
  - "docker-ce"
  - "docker-ce-cli"
  - "containerd.io"
docker_compose_url: https://github.com/docker/compose/releases/download/1.28.2/docker-compose-Linux-x86_64

Add dockerwordpressplaybook.yml

---
- hosts: docker
  vars_files:
    - vars/main.yml

  tasks:
  - name: Install aptitude using apt
    apt: name=aptitude state=latest update_cache=yes force_apt_get=yes

  - name: Install required system packages for Docker
    apt: name={{ docker_required_packages }} state=latest update_cache=yes

  - name: Add Docker GPG key
    apt_key:
      url: "{{ docker_gpg_url }}"
      state: present

  - name: Add Docker repository
    apt_repository:
      repo: "{{ docker_repo }}"
      state: present

  - name: Install Docker
    apt: name={{ docker_packges }} state=latest update_cache=yes

  - name: Install Python Docker module
    pip:
      name: docker

  - name: Add adminstrator to docker group
    user:
      name: "{{ admin_user }}"
      groups: docker
      append: yes
      createhome: yes

  - name: Install Docker Compose
    get_url:
      url: "{{ docker_compose_url }}"
      dest: /usr/local/bin/docker-compose
      mode: u+x,g+x,o+x

  - name: Set up docker-compose file
    template:
      src: "files/docker-compose.yml"
      dest: "/home/{{ admin_user }}/wordpress/"

  - name: Set permission for docker daemon
    file:
      path: /var/run/docker.sock
      mode: '0770'

  - name: docker compose up
    shell:
      chdir: "/home/{{ admin_user }}/wordpress/"
      cmd: docker-compose up -d

Use the below commands to see images,containers,volumes

docker images
docker ps -a
docker volume list

To shutdown the container use the command shown below

docker-compose down

Since we use below command in docker-compose.yml file you get the access to edit the wordpress files.If we remove the containers volume will not removed.

volumes: ['./:/var/www/html']

For example if we try to add media in wordpress, the default size would be 2 MB.we can change the size by editing .htaccess file.

# BEGIN WordPress
php_value memory_limit 100M
php_value upload_max_filesize 64M
php_value post_max_size 64M
php_value max_execution_time 300
# END WordPress

To remove containers along with volumes use the command below

docker-compose down --volumes

Ansible to install docker and create docker images

nithinalias — Sun, 13 Mar 2022 08:24:36 +0000

create an ansible file to install docker and create docker images

---
- hosts: docker
  become: true
  vars:
    container_count: 1
    default_container_name: docker
    default_container_image: ubuntu
    default_container_command: sleep 1d

  tasks:
    - name: Install aptitude
      apt:
        name: aptitude
        state: latest
        update_cache: true

    - name: Install required system packages
      apt:
        pkg:
          - apt-transport-https
          - ca-certificates
          - curl
          - software-properties-common
          - python3-pip
          - virtualenv
          - python3-setuptools
        state: latest
        update_cache: true

    - name: Add Docker GPG apt Key
      apt_key:
        url: https://download.docker.com/linux/ubuntu/gpg
        state: present

    - name: Add Docker Repdoository
      apt_repository:
        repo: deb https://download.docker.com/linux/ubuntu focal stable
        state: present

    - name: Update apt and install docker-ce
      apt:
        name: docker-ce
        state: latest
        update_cache: true
      notify: Restart docker

    - name: Install Docker Module for Python
      pip:
        name: docker

    - name: Pull default Docker image
      docker_image:
        name: "{{ default_container_image }}"
        source: pull

    - name: Create default containers
      docker_container:
        name: "{{ default_container_name }}{{ item }}"
        image: "{{ default_container_image }}"
        command: "{{ default_container_command }}"
        state: present
      with_sequence: count={{ container_count }}

  handlers:
    - name: Restart docker
      service:
        name: docker
        state: restarted

Docker Commands and Dockerfile Creation

nithinalias — Thu, 10 Mar 2022 18:04:19 +0000

Docker Commands

docker --version
docker --help
docker images
docker pull ubuntu
docker run -it -d ubuntu
docker ps
docker ps -a
docker exec -it containerID bash
exit
docker stop containerID
docker login
docker commit containerID Zulaikha/ubuntu
docker tag Imagename Zulaikha/ubuntu
docker push Zulaikha/ubuntu
docker rm containerID
docker rmi ImageID
docker container logs containerID
docker container rm containerID
docker container kill containerID
docker container run containerID
docker container start containerID
docker export --output="latest.tar" containerID
docker import /home/username/Downloads/demo.tgz

Dockerfile

Create a dockerfile for Apache webserver

Make a directory to build our Docker file for which you can use vim editor.

mkdir /test
cd /test
sudo vim dockerfile

FROM ubuntu
ENV DEBIAN_FRONTEND=noninteractive
RUN apt-get update
RUN apt-get install apache2 -y
RUN apt-get install apache2-utils -y
RUN apt-get clean
EXPOSE 80
CMD ["apache2ctl","-D","FOREGROUND"]

Description of the above commands

Ubuntu is our base image in which we are launching our server.
In the second line, is to set a non-interactive environment.
In the third line, the apt-get update command is used to update all the packages on Ubuntu.
In the fourth line, we are installing apache2 on our image.
In the fifth line, we are installing all the necessary utility Apache packages.
In the sixth line, the apt-get clean command cleans all the unnecessary files from the system.
In the seventh line, the EXPOSE command is used to expose the port 80 of Apache in the container.
The last command is used to run apache2 in the background.

Next step is to build the docker file by using the docker build command.

sudo docker build -t myapachewebserver .

Command:
-t: this option is to tag the image

The web server file has built, the next step is to create a container from the image for that we use the docker run command.

docker run -d -p 8080:80 myapachewebserver
                 OR
docker run -it -p 8080:80 myapachewebserver
Commands:
-d: This option is used to run the container in detached mode i.e the container can run in the background.

-p: This option is used to map our port number with 5000 port numbers on our localhost.
-it: This argument is used to allocate a bash shell and take standard input.

If you go to your web browser and write http://host_ip:8080 your Apache server is up and running on that port.

Create a dockerfile for Apache webserver and Copy the Index.html

Method 1

Make a directory
mkdir /test
cd /test
Create a sample web page with name index.html
vim /test/index.html

<!DOCTYPE html>
<html>
<body>

<h1>Hello World</h1>

<p>My Personal website</p>

</body>
</html>

Now create a file dockerfile
vim /test/dockerfile

FROM ubuntu
ENV DEBIAN_FRONTEND=noninteractive
RUN apt-get update
RUN apt-get install apache2 -y
RUN apt-get install apache2-utils -y
RUN apt-get clean
COPY index.html /var/www/html/
EXPOSE 80
CMD ["apache2ctl","-D","FOREGROUND"]

Next step is to build the docker file by using the docker build command.

sudo docker build -t myapachewebserver .

Command:
-t: this option is to tag the image

The web server file has built, the next step is to create a container from the image for that we use the docker run command.

docker run -d -p 8080:80 myapachewebserver
                 OR
docker run -it -p 8080:80 myapachewebserver
Commands:
-d: This option is used to run the container in detached mode i.e the container can run in the background.

-p: This option is used to map our port number with 5000 port numbers on our localhost.
-it: This argument is used to allocate a bash shell and take standard input.

If you go to your web browser and write http://host_ip:8080 your
Apache server is up and running on that port.

Method 2

Another way to create apache web server image is using manual commands.
Step 1: Get the latest ubuntu Docker image by using docker pull command. Docker pull command is used to download or pull latest image from Docker Hub repositories.

docker pull ubuntu:latest

Step 2: To check and list all docker images

docker images

Step 3: To run docker image we use following command
-it : This argument is used to allocate a bash shell and take standard input.
-- name : This argument is used to tag a name to the running container.

docker run -it --name webserver ubuntu:latest

Step 4: Now install Apache webserver and it’s all dependencies

[root@Docker_Id]#apt-get update -y 
[root@Docker_Id]#apt-get install apache2 -y 
[root@Docker_Id]#apt-get install nano -y

Step 5: Now create a create a webpage at location /var/www/html/index.html

[root@Docker_Id]# nano /var/www/html/index.html

<!DOCTYPE html>
<html>
<body>

<h1>Hello World</h1>

<p>My Personal website</p>

</body>
</html>

Step 6: Exit from the running container using exit command. Exit command will stop the container. Exit command of docker same as power off or shut down of our computer.

[root@Docker_Id]# exit

Step 7: Now we have container in which apache webserver is installed and our webpage is configured. We can make a new customized docker image from the stopped docker image using docker commit command. Docker commit command will build our own image.

#docker commit <container_id or name of container will launching> <Name of new image>:<version name>

docker commit webserver webserver:v1
docker images

Step 8: Launching a webserver container using our customized image.
-p: This argument is used to port forwarding. Which means anybody from outside who comes for 8080 its request is forwarded to port 80. Port 80 is default port number where apache webserver runs.
/usr/sbin/apache2 –D FOREGROUND: This argument is command which will run when container is launched this command will start the web server

docker run -p 8080:80 webserver:v1 /usr/sbin/apache2 -D FOREGROUND

Step 9: To See the Result on web browser

http://:8080/

Create a dockerfile for Nginx webserver and Copy the Index.html

we need to change the docker file as below.Rest of the steps are same as above.

FROM ubuntu
RUN apt-get update
RUN apt-get install nginx -y
COPY index.html /var/www/html/
EXPOSE 80
CMD ["nginx","-g","daemon off;"]

AWS Automation

nithinalias — Tue, 08 Mar 2022 17:11:01 +0000

Introduction CloudFormation

Template snippets

https://docs.aws.amazon.com/AWSCloudFormation/latest/UserGuide/CHAP_TemplateQuickRef.html

Management Tool - cloudFormation - create new stack - Upload a template to Amazon S3 - choose file(file is shown below) - your region should be exact as ImageId - Next - add stack,keyname - Next - Next - create - click on stack name - you could see all in detail - Go to EC2 you could see your Instance

Resources:
  MyEC2Instance:
    Type: 'AWS::EC2::Instance'
    Properties:
      InstanceType: t2.micro
      ImageId: ami-0de53d8956e8dcf80
      KeyName: !Ref KeyName
      SecurityGroups:
        - !Ref InstanceSecurityGroup
      Tags:
        - Key: Name
          Value: My CF Instance
  InstanceSecurityGroup:
    Type: 'AWS::EC2::SecurityGroup'
    Properties:
      GroupName: MyDMZSecurityGroup
      GroupDescription: Enable SSH access via port 22
      SecurityGroupIngress:
        IpProtocol: tcp
        FromPort: 22
        ToPort: 22
        CidrIp: 0.0.0.0/0

Outputs: 
  InstanceID:
    Description: The Instance ID
    Value: !Ref MyEC2Instance

Management Tool - cloudFormation - select stack - Action - Delete stack - yes Delete - Go to S3 - click on the bucket containing template - click on the template file - make public - click on the link you could see the the template - After cloudformation delete finishes you can delete this bucket.

Introduction Beanstalk

Compute - Elastic Beanstalk - Get started - add application name,platform = PHP,Enable upload your code - upload(choose index.zip file contain a __MACOSX folder and index file) - upload - create application - Now a dashboard creates - configuration contain software,Instance,Load Balancer etc.... - you will get a Url, copy and paste in webbrowser to get that application.

index file content shown below

<html>
<head><title>Hello Cloud Gurus</title></head>
<body><h1 align="center">Hello Cloud Gurus! This is webpage was provisioned using Elastic Beanstalk!</h1></body>
</html>

Action - Delete Application - Delete

AWS OpsWorks

AWS VPC

nithinalias — Mon, 07 Mar 2022 16:34:06 +0000

VPC Overview

Build a Custom VPC

Network and Content Delivery - VPC - Your VPCs - create VPC - add name,IPv4 CIDR block = 10.0.0.0/16,Enable Amazon provided IPV6 CIDR block,Tenancy = default - yes,create - Now automatically new Route Table,Network ACLs,Security Groups creates.

Go to Subnets - create Subnet - add name tag,select the VPC you created,select availability zone,IPv4 CIDR block = 10.0.1.0/24 -
yes create - Go to Subnets - create Subnet - add name tag,select the VPC you created,select another availability zone,IPv4 CIDR block = 10.0.2.0/24 - yes create

Go to Internet Gateway - create Internet Gateway - add name tag - yes create - Attach to VPC - yes attach

Go to Route Tables - select the Route table - Subnet Association - There will be no subnet association - Create Route Table - add name tag,select VPC - yes create - Routes(To enable internet) - Edit - Add another route - Destination(0.0.0.0/0),Target = select the internet gateway - save - Edit(To add ipv6) - Add another route - Destination(::/0),Target = select the internet gateway - save - subnet Association - Edit - select subnet(IPv4 CIDR block = 10.0.1.0/24) - save(10.0.1.0/24 = public network,10.0.2.0/24 = private network) - Go to subnets - select 10.0.1.0/24 subnet - subnet Actions - Modify auto-assign IP settings - Enable auto-assign public ipv4 address - save - Launch an EC2 Instance - configure Instance - Network(select the created VPC),select subnets(10.0.1.0/24) - Security Group(Add ssh,http,https rule) - Launch(Public Instance) - Launch another EC2 Instance - configure Instance - Network(select the created VPC),select subnets(10.0.2.0/24) - Launch(Private Instance)

Network and Security - Security Groups - create security Group - add name,select created VPC - Inbound(Add rule - SSH,MYSQL/AURORA,HTTP,HTTPS,ALL ICMP - Source = 10.0.1.0/24 for all rules) - create - Add this Security Group to created Private Instance - Action - Networking - Change Security Group - select and Assign Security Groups

Now login to Public Instance that we created - check Internet is available or not(Internet will be available) - Ping private Instance IP - you will get pinging output - Now copy the login-key of Private Instance to Public Instance - Then login to private Instance from Public Instance - Here Internet will not available

Network Address Translation(NAT)

EC2 - Launch an Instance - choose AMI - community AMIs - amazon-ami-vpc-nat-hvm-.... - select - Configure Instance - Network(select the created VPC),select subnets(10.0.1.0/24) - Security Group(Add ssh,http,https rule) - Launch - Select the NAT instance - choose Actions - Networking - Change source/destination check - choose Stop - save

VPC - Route tables - select main VPC Route table - Routes - Edit - Add another route - Destination(0.0.0.0/0),Target = select the NAT Instance gateway - save

Now log in private instance from public instance and check internet.Now you will get internet.Now Terminate the nat instance and then we loose internet connection.

VPC - NAT gateway - create Nat gateway - select subnet(Public subnet 10.0.1.0/24 ) - Create new EIP(Elastic ip) - create a NAT Gateway - Go to Route Table - select main VPC Route table - Routes - Remove old route - Add another route -Destination(0.0.0.0/0),Target = select the NAT gateway - save

Now log in private instance from public instance and check internet.Now you will get internet.

Access Control Lists (ACLs)

VPC - Network ACLs - create Network ACLs - add name,select the created VPC - yes,create

In created Public Instance do the below codes

yum update -y
yum install httpd -y
service httpd start
chkconfig httpd on
echo "<html><h1>Hello Cloud Gurus!</h1></html>" > /var/www/html/index.html

Now we get the website in web-browser http://ipaddress of Public Instance

VPC - Network ACLs - select the newly created ACL - Inbound Rule - Edit - Add another rule - Rule = 100,Type = HTTP(80),source = 0.0.0.0/0 - Rule = 200,Type = HTTPS(443),source = 0.0.0.0/0 - Rule = 300,Type = SSH(22),source = 0.0.0.0/0 - save - Outbound Rule - Edit - Add another rule - Rule = 100,Type = HTTP(80),source = 0.0.0.0/0 - Rule = 200,Type = HTTPS(443),source = 0.0.0.0/0 - Rule = 300,Type = Custom TCP Rule,Port Range = 1024 - 65535,source = 0.0.0.0/0 - save - subnet Association - select public subnet(10.0.1.0/24) - save

Now if we add Inbound Rule - Edit - Add another rule - Rule = 101,Type = HTTP(80),source = myipaddress/subnet,Allow/Deny = Deny - Now we get the website in web-browser http://ipaddress of Public Instance

Now if we add Inbound Rule - Edit - Add another rule - Rule = 99,Type = HTTP(80),source = myipaddress/subnet,Allow/Deny = Deny - Now we donot get the website in web-browser http://ipaddress of Public Instance

Here Rule = 99 which comes first in Inbound Rule and that rule get preference first.

VPC End Points

IAM - Role - create role - EC2 - EC2 - Next - AmazonS3FullAccess - Next - Add role-name - create role - Add the role to Private Instance - Action - Instance settings - Attach/Replace IAM Role - select the role - Apply

VPC - Network ACLs - select newly created default Network ACL - subnet association - Edit - Select Public subnet (10.0.1.0/24) - save

Now login to private Instance from Public Instance

sudo su 
aws s3 ls

Now you get S3 bucket list.

Go to VPC - Route Tables - select newly created main Route Table - Routes - Edit - Remove(0.0.0.0/0) - save

Now if you use the command shown below you cannot see S3 bucket list.

aws s3 ls

Go to Endpoint - create Endpoint - select s3 gateway(com.amazonaws.us-east-1.s3) - select created VPC - select main private route table - create Endpoints

Now using "aws s3 ls" you get S3 bucket list.

Go to VPC - Route Tables - select newly created main Route Table - Routes - Now you can see VPC Endpoints(This is not behind NAT gateway this is VPC Endpoints)

Custom VPCs and Application Load Balancers

EC2 - Load Balancer - create Load Balancer - create Application Load Balancer - select the created VPC - Here you need atleast 2 Availabilty Zone of Public Instance

VPC Flow Logs

Go to VPC - Select the created VPC - Action - Create Flow Log - click on Set Up Permission link - view Policy Document - Allow - Go back select the role now created - create Flow Log - But this will show an error because we need to create Destination Log Group - Go to Cloudwatch - Logs - create Log group - add a name - create Log Group - Go to Flow Log - add log group - create flow log

NAT's Vs Bastions

VPC Clean Up

EC2 - Select Public and Private Instance - Actions - Instance state - Terminate - Yes Terminate

VPC - NAT Gateways - select nat gateway - Action - Delete NAT Gateway - EndPoints - select Endpoints - Action - Delete Enpoints - Internet Gateway - select Internet Gateway - Detach from VPC - yes,Delete - your VPC - select VPC - Action - Delete

AWS Network and Route53

nithinalias — Mon, 07 Mar 2022 07:55:53 +0000

DNS

Register Your Domain Name

Networking and Content Delivery - Route53 - DNS management - Get started now - Registered domains - Register Domain - Add domain name - check(check the price and all) - Add to cart - continue - Add details and Register domain name

Create 4 EC2 instance in different region.Add this bootstrap script in configure Instance - Advance Details
Webserver -1 in Northern Virginia
Webserver -1 in Northern Virginia
Webserver -2 in Sydney
Webserver -3 in south America sauo paulo

#!/bin/bash
yum update -y
yum install httpd -y
cd /var/www/html
echo "webserver -1 - Northern Virginia " > index.html
service httpd start
chkconfig httpd on

Simple Routing Policy

Route53 - click on Hosted Zones - click on created DNS name - create a Record set - add TTL second = 60 and value as 4 webservers ip address like shown below,Routing policy = simple - create(wait for sometime to service come into effect) - Now type the domain name in webbrowser - you will get the website and region from you get website might be random.

Public ip of Northern Virginia1
Public ip of Northern Virginia2
Public ip of Sydney
Public ip of south America sauo paulo

Weighted Routing Policy

Route53 - click on Hosted Zones - click on created DNS name - Delete simple Routing policy created earlier - create a Record set - add TTL second = 60 and value as Northern Virginia 1 webserver ip address,Routing policy = weighted,weight = 25,setID = webserver1 - create - Do the same for other 3 also(total =25%*4=100%)- wait for sometime to service come into effect - Now type the domain name in webbrowser - you will get the website according to weighted.

Latency Routing Policy

Route53 - click on Hosted Zones - click on created DNS name - Delete Weighted Routing policy created earlier - create a Record set - add TTL second = 60 and value as Northern Virginia 1 webserver ip address and Northern Virginia 2 webserver ip address,Routing policy = Latency,Region automatically will come,setID = webserver1&2 - create(Do this for other two webservers and total there will be three types) - wait for sometime to service come into effect - Now type the domain name in webbrowser - you will get the website with low latency region(use VPN client to change your current network location and test the latency.If you connect to Australia in VPN and type your domain in webbrowser you get the response from sydney).

Failover Routing Policy

Route53 - click on Hosted Zones - click on created DNS name - Delete Latency Routing policy created earlier - Go to Health checks - create health check - Add name,what to monitor = endpoint,specify enpoint by = IP address,protocol = HTTP, IP address = public ip of Northern Virginia 1,Host name = domain name,port = 80,path = /index.html - Next - create alarm = no - create health check - wait sometime to service come into effect - Route53 - click on Hosted Zones - click on created DNS name - create a Record set - add TTL second = 60 and value as Northern Virginia 1 webserver ip address,Routing policy = Failover,Failover Record Type = Primary,setID = primary,Enable Associated with health check and add the created health check - create(Do this for Sydney and set Failover Record Type = Secondary,setID = Secondary ) - wait for sometime to service come into effect - Now type the domain name in webbrowser - you will get the website of Northern Virginia1.If we stop the Northern Virginia1 EC2 Instance(Go to health checks status will be unhealthy) you will get the website of Sydney(secondary)

Geolocation Routing Policy

Route53 - click on Hosted Zones - click on created DNS name - Delete Failover Routing policy created earlier - create a Record set - add TTL second = 60 and value as Northern Virginia 1 webserver ip address,Routing policy = Geolocation,Location = North America,setID = North America set - create(Do this for sydney add TTL second = 60 and value as Sydney webserver ip address,Routing policy = Geolocation,Location = Oceana,setID = Sydney) - wait for sometime to service come into effect - Use VPN client to change your current network location to United states and Now type the domain name in webbrowser you get website of Northern Virginia1.If you connect to Australia in VPN and type your domain in webbrowser you get the website from sydney.

Multivalue Routing Policy

Route53 - click on Hosted Zones - click on created DNS name - Delete Geolocation Routing policy created earlier - create a Record set - add TTL second = 60 and value as Northern Virginia 1 webserver ip address,Routing policy = Multivalue,setID = web01USEAST-1 - create(Do this Northern Virginia 2 - add TTL second = 60 and value as Northern Virginia 2 webserver ip address,Routing policy = Multivalue,setID = web02USEAST-1) - wait for sometime to service come into effect - Now type the domain name in webbrowser - you will get the website random and if one EC2 Instance stopped you get website second one.

AWS Security

nithinalias — Sat, 05 Mar 2022 17:31:55 +0000

Compliance on AWS

DDOS

AWS marketplace security products

IAM Custom Policies

IAM - Policies - create policy - visual editor - choose a service - S3 - In access level list,read - click on Resources - Enable All resources - Review Policy - Add name - create policy

IAM - Roles - create role - select EC2 - Next Permission - select the created policy - Next - Add role name - create role

Go to S3(create 2 buckets in different region) - create bucket - Add name,region - create

Go to EC2 - Launch Instance - Then access Instance from putty or terminal - switch to root(sudo su) - aws s3 ls - unable to locate credentials

Go to IAM - users - add user - add a username - enable Programatic access - Next - Enable Group Administrator - Next - create user - Now you will get access keyid,secret access key - Go to EC2 Instance - Actions Instance settings - Attach Replace IAM Role - select the created role - Apply - Now type the command 'aws s3 ls' inside the Instance - Now it will show the S3 bucket we created earlier

create a testfile and copy to S3 bucket

echo "Hello World" > test.txt
ls
test.txt
aws s3 cp /home/ec2-user/test.txt s3://bucketname

It will show upload failed since we didnt give write access.
Go to IAM - policies - select the policy that we created - Edit policy - Enable write in Access level - Review policy - save changes - Now you can upload file using above command

aws s3 cp /home/ec2-user/test.txt s3://bucketname
aws s3 ls s3://bucketname

Same you can do in second bucket in other region

MFA AND REPORTING WITH IAM

Go to IAM - Activate MFA on your root account - manage MFA - Enable virtual MFA device - Next step - Next step - scan the barcode using Google authenticator in mobile - add the generated authentication codes - Activate virtual MFA

This for root account.Now we can do for user account.

IAM - users - Add user - Add username,Enable programmatic Access - Next - create group - select AdminstrationAccess,give groupname - create group - select the created group - create user(Now you will get access keyid,secret access key) - Download.csv - Go to users - select the created user - security credentials - click on Assigned MFA device - Enable virtual MFA device - Next step - Next step - scan the barcode using Google authenticator in mobile - add the generated authentication codes - Activate virtual MFA

Go to Instance and remove the role that we created early - select Instance - Actions - Instance settings - Attach Replace IAM Role - No role - Apply - yes,detach - Login using putty or terminal - switch to root(sudo su) - aws s3 ls - unable to locate credentials

configure user using access keyid,secret access key

aws configure
Add access keyid
Add secret access key
Add region name
ENTER
ENTER
aws s3 ls

Now you can see the bucket created.Now use the command below for MFA

aws iam create-virtual-mfa-device --virtual-mfa-device-name EC2-User --outfile /home/ec2-user/QRCode.png --bootstrap-method QRCodePNG

Now using 'ls' command you can see QRCode.png and copy the file to s3 bucket

aws s3 cp /home/ec2-user/QRCode.png s3://bucketname

Go to s3 in amazon service - select the QRCode.png - Actions - make public - make public - click on QRCode.png - click on Link - Now you get the QRCode - scan the barcode using Google authenticator in mobile

aws iam enable-mfa-device --user-name EC2-User --serial-number arn:aws:iam::"USERNUMBERHERE":mfa/EC2-User --authentication-code-1 "CODE1HERE" --authentication-code-2 "CODE2HERE"

"USERNUMBERHERE" = we get from IAM - users - select user - user ARN(copy the numbers)

"CODE1HERE" and "CODE2HERE" = we get from google authenticator

Go to IAM - users - select user - security credential - you can see Assigned MFA device

Go to IAM - credential report - Download Report - Give details of
users credential

Security Token Service

Security and Logging

AWS WAF

AWS Hypervisors

Dedicated Instances vs Dedicated Hosts

EC2 - Instances - Dedicated Hosts - Allocate a Host - Add Instance type,availability zone - Allocate host
OR
EC2 - Launch Instance - configure Instance(Tenancy = Dedicated Instance or Dedicated Host) - Launch

AWS system manager EC2 Run Command

Go to IAM - Roles - create role - EC2 - EC2 Role for Simple Systems Manager - Next - Next - Add role name - EC2 - Launch Instance - configure Instance(Add created role) - Launch

Go to System Manager - Run command - select Aws configure Cloudwatch - select created Instance in Target Instance - Run

AWS system manager Parameter store

EC2 - system manager shared resources(bottom of the page) - parameter store - Get started now - Add name,Enable secure string - create parameter - we can pass this to cloud formation,lamda etc

AWS config with s3

Management tools - config - Rules - Add rule - select s3 - you can see rules related to s3 like s3-bucket-public-read-prohibited,s3-bucket-public-write-prohibited etc..

Presigned URLs

Go to IAM - Roles - create role - EC2 - Next - select Amazons3FullAccess - Next - Add rolename - create role - Launch Instance - Add created role in configure Instance - Launch - Login to created Instance

make new s3 bucket and copy a testfile to it using the command shown below

aws s3 ls
aws s3 mb s3://bucketname
echo "Hello World" > test.txt
aws s3 cp test.txt s3://bucketname
aws s3 ls s3://bucketname

If we go to S3(management console) and access the test.txt by clicking on link we cannot open it.
To access the file for 300 seconds we need to use the command shown below

aws s3 presign s3://bucketname/test.txt --expires-in 300

This generate a https link.copy and paste this link in browser to access the file for 300 seconds.

Inspector vs Trusted Advicer

Ec2 - Launch an Instance
Security,Identity and Compliance - Inspector - Get started - choose or create role - view details - IAM role = create a new IAM role - Add role name - Allow - Tag your Ec2 Instance - Manage Tags - select Instance and add key and tag - Go back - Install AWS agent - open the link 'To install the Amazon Inspector Agent on a Linux based EC2 Instance' - Login to EC2 instance and run the commands to install AWS agent

sudo su 
wget https://inspector-agent.amazonaws.com/linux/latest/install
curl -O https://inspector-agent.amazonaws.com/linux/latest/install

Now go to Inspector window - Next - add name,key,value created earlier - Next - Add name,Rule packages = common vulnerabilities and exposures1.1,Duration = 1hr - Next - create - select the created Inspector - Run - After 1 hr it will give the result - Assesment runs - Download Report - Full Report - Generate Report

Go to Assessment Templates - create - add name = master template,target name, rule packages(add all the rules),duration=24hr - create and run - After 24 hrs we can download the report

Go to Management Tools - Trusted Advisor - Give details of Cost optimization,performance,security,Fault Tolerance - we need business or enterprise subscription to unlock these.

Shared Responsibility

Other Security Aspects

CloudTrail - Turning It On and Validating Logs

IAM - Groups - create New Group - Add Group name - Next - AWS CloudTrailFullAccess -

Go to S3 - select a created bucket - Management - Add Lifecycle rule - Add Rule name - Next - Current Version - Next - click current version - Add expire days - Next - save